RomanNum3ral
/
Cybersecurity-Projects
mirror of https://github.com/CarterPerez-dev/Cybersecurity-Projects.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #304 from CarterPerez-dev/project/security-news-scraper 

Project/security news scraper
This commit is contained in:
Carter Perez 2026-07-09 17:23:32 -04:00 committed by GitHub
parent bcd9aef757 874114c351
commit 318e5967be
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
121 changed files with 15664 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

32
PROJECTS/intermediate/security-news-scraper/.github/workflows/release.yml vendored Normal file
View File

@ -0,0 +1,32 @@
# ©AngelaMos | 2026
# release.yml
name: release
on:
push:
tags:
- "v*"
permissions:
contents: write
jobs:
goreleaser:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- uses: actions/setup-go@v5
with:
go-version-file: go.mod
- uses: goreleaser/goreleaser-action@v6
with:
distribution: goreleaser
version: "~> v2"
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

27
PROJECTS/intermediate/security-news-scraper/.gitignore vendored Normal file
View File

@ -0,0 +1,27 @@
# ©AngelaMos | 2026
# .gitignore
# private dev + research material (never public)
docs/
# Go build artifacts
/nadezhda
/bin/
*.test
*.out
*.prof
# local runtime data (SQLite store + WAL sidecars)
*.db
*.db-wal
*.db-shm
*.sqlite
/data/
# large local KEV catalog snapshot (manual download; tests use kev-sample.json)
/testdata/kev/kev-full.json
# env / secrets / local overrides
.env
*.local.yaml
config.local.yaml

45
PROJECTS/intermediate/security-news-scraper/.goreleaser.yaml Normal file
View File

@ -0,0 +1,45 @@
# ©AngelaMos | 2026
# .goreleaser.yaml
version: 2
project_name: nadezhda
before:
hooks:
- go mod tidy
builds:
- id: nadezhda
main: ./cmd/nadezhda
binary: nadezhda
env:
- CGO_ENABLED=0
goos: [linux, darwin]
goarch: [amd64, arm64]
ldflags:
- -s -w -X github.com/CarterPerez-dev/nadezhda/internal/version.Version={{ .Version }}
archives:
- id: nadezhda
name_template: "{{ .ProjectName }}_{{ .Version }}_{{ .Os }}_{{ .Arch }}"
formats: [tar.gz]
files:
- README.md
- LICENSE
checksum:
name_template: "checksums.txt"
changelog:
sort: asc
filters:
exclude:
- "^docs:"
- "^test:"
- "^chore:"
release:
github:
owner: CarterPerez-dev
name: nadezhda

661
PROJECTS/intermediate/security-news-scraper/LICENSE Normal file
View File

@ -0,0 +1,661 @@
GNU AFFERO GENERAL PUBLIC LICENSE
Version 3, 19 November 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU Affero General Public License is a free, copyleft license for
software and other kinds of works, specifically designed to ensure
cooperation with the community in the case of network server software.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
our General Public Licenses are intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
Developers that use our General Public Licenses protect your rights
with two steps: (1) assert copyright on the software, and (2) offer
you this License which gives you legal permission to copy, distribute
and/or modify the software.
A secondary benefit of defending all users' freedom is that
improvements made in alternate versions of the program, if they
receive widespread use, become available for other developers to
incorporate. Many developers of free software are heartened and
encouraged by the resulting cooperation. However, in the case of
software used on network servers, this result may fail to come about.
The GNU General Public License permits making a modified version and
letting the public access it on a server without ever releasing its
source code to the public.
The GNU Affero General Public License is designed specifically to
ensure that, in such cases, the modified source code becomes available
to the community. It requires the operator of a network server to
provide the source code of the modified version running there to the
users of that server. Therefore, public use of a modified version, on
a publicly accessible server, gives the public access to the source
code of the modified version.
An older license, called the Affero General Public License and
published by Affero, was designed to accomplish similar goals. This is
a different license, not a version of the Affero GPL, but Affero has
released a new version of the Affero GPL which permits relicensing under
this license.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU Affero General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Remote Network Interaction; Use with the GNU General Public License.
Notwithstanding any other provision of this License, if you modify the
Program, your modified version must prominently offer all users
interacting with it remotely through a computer network (if your version
supports such interaction) an opportunity to receive the Corresponding
Source of your version by providing access to the Corresponding Source
from a network server at no charge, through some standard or customary
means of facilitating copying of software. This Corresponding Source
shall include the Corresponding Source for any work covered by version 3
of the GNU General Public License that is incorporated pursuant to the
following paragraph.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the work with which it is combined will remain governed by version
3 of the GNU General Public License.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU Affero General Public License from time to time. Such new versions
will be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU Affero General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU Affero General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU Affero General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as published
by the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If your software can interact with users remotely through a computer
network, you should also make sure that it provides a way for users to
get its source. For example, if your program is a web application, its
interface could display a "Source" link that leads users to an archive
of the code. There are many ways you could offer source, and different
solutions will be better for different programs; see section 13 for the
specific requirements.
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU AGPL, see
<https://www.gnu.org/licenses/>.

201
PROJECTS/intermediate/security-news-scraper/README.md Normal file
View File

@ -0,0 +1,201 @@
<!-- ©AngelaMos | 2026 -->
<!-- README.md -->
```json
███╗ ██╗ █████╗ ██████╗ ███████╗███████╗██╗ ██╗██████╗ █████╗
████╗ ██║██╔══██╗██╔══██╗██╔════╝╚══███╔╝██║ ██║██╔══██╗██╔══██╗
██╔██╗ ██║███████║██║ ██║█████╗ ███╔╝ ███████║██║ ██║███████║
██║╚██╗██║██╔══██║██║ ██║██╔══╝ ███╔╝ ██╔══██║██║ ██║██╔══██║
██║ ╚████║██║ ██║██████╔╝███████╗███████╗██║ ██║██████╔╝██║ ██║
╚═╝ ╚═══╝╚═╝ ╚═╝╚═════╝ ╚══════╝╚══════╝╚═╝ ╚═╝╚═════╝ ╚═╝ ╚═╝
```
[![Cybersecurity Projects](https://img.shields.io/badge/Cybersecurity--Projects-Project%20%2339-red?style=flat&logo=github)](https://github.com/CarterPerez-dev/Cybersecurity-Projects/tree/main/PROJECTS/intermediate/security-news-scraper)
[![Go](https://img.shields.io/badge/Go-1.25-00ADD8?style=flat&logo=go&logoColor=white)](https://go.dev)
[![Keyless](https://img.shields.io/badge/enrichment-keyless-4B7BEC?style=flat)](https://www.cisa.gov/known-exploited-vulnerabilities-catalog)
[![Single static binary](https://img.shields.io/badge/binary-single%20static-6d4aff?style=flat)](https://pkg.go.dev/modernc.org/sqlite)
[![License: AGPLv3](https://img.shields.io/badge/License-AGPL_v3-purple.svg)](https://www.gnu.org/licenses/agpl-3.0)
> A keyless security-news and CVE intelligence engine in Go. Point it at a set of RSS and Atom feeds and it fetches them politely, clusters the same story as it breaks across outlets, extracts every CVE it mentions, enriches each one with authoritative exploit intelligence, ranks the whole set by what actually matters, and hands you a browsable dossier in your terminal. It ships as a single static binary over a local SQLite store, needs no API key to do any of it, and treats the news as the product with the CVE data as the intelligence that sharpens it.
<p align="center">
<img src="assets/tui.png" alt="The nadezhda terminal UI: a ranked, color-coded dossier of security news, with per-story scores, source dots, and CVE severity" width="100%">
</p>
## Why aggregate security news
There is more security news published every day than any person can read, and almost all of it is noise relative to the one story that matters to you right now. The hard problem is not gathering headlines. It is deciding which three of the four hundred you should act on before lunch.
The signal is usually some combination of the same few factors. A vulnerability that many outlets picked up in the same few hours is trending for a reason. A CVE that CISA just added to its Known Exploited Vulnerabilities catalog is being used against real targets today. A flaw with a high EPSS score is one the rest of the world is about to start exploiting. Nadezhda computes that combination and sorts by it, so a CVSS 6.5 that is on the KEV catalog can outrank a CVSS 9.8 that nobody has touched.
The cost of missing that signal is not hypothetical. Log4Shell went from a GitHub issue to every security outlet on earth within hours and landed on the KEV catalog almost immediately; a tool that ranks by cross-outlet velocity plus KEV plus EPSS floats it to the top of the first scrape. Equifax fell to a known Apache Struts flaw that had a patch available for months, and the intelligence to prioritize it sat in public feeds the whole time, so the failure was one of triage, not of information. MOVEit was mass-exploited in a tight window where the advisories, the KEV listing, and the coverage all arrived together. Velocity was the tell.
## What it is
This is not a stub. It ingests real feeds, enriches real CVEs from keyless authoritative sources, clusters and ranks real coverage, and surfaces it four ways. Every capability below is exercised by unit tests, offline fixture-driven tests, and a live run against the real feeds.
**Ingestion**
- Seven seeded RSS/Atom feeds (Krebs on Security, The Hacker News, BleepingComputer, SecurityWeek, Dark Reading, The Register, CISA), fetched concurrently through a per-host rate limiter with an honest User-Agent
- Conditional GET: an unchanged feed answers `304 Not Modified` and costs almost nothing, and a retry honors `Retry-After` and never retries a timeout or a cancellation
- Fail-soft by construction: one broken feed is reported at the end and never aborts the run, and every article deduplicates at the database on both its canonical URL and a content hash
**Enrichment, keyless by default**
- CVE core from the CVE Program's cvelistV5 records: a CVSS score resolved by a fixed version precedence (v4.0, then v3.1, then v3.0, then v2.0) read across both the vendor and the CISA-ADP containers, plus CWE and description
- CISA KEV membership, with the `knownRansomwareCampaignUse` string mapped explicitly to a real ransomware signal rather than treated as a boolean
- FIRST EPSS probability and percentile, parsed from the quoted strings the API actually returns so the score never silently reads as zero
- A TTL cache with a separate, shorter negative TTL, so a not-yet-published CVE is not re-fetched every run; NVD 2.0 is supported as an optional key-gated booster, never a requirement
**Clustering and ranking**
- Connected-components (union-find) clustering: two items join when they share a CVE, or when they come from different outlets and their title token sets clear a Jaccard threshold inside a time window
- A pure, deterministic, news-first weighted score: recency, cross-outlet velocity, source trust, and a keyword watchlist carry 70 percent of the weight, with KEV, CVSS, and EPSS as the supporting 30, so the same corpus always sorts the same way and is pinned by golden-order tests
**Surfaces**
- A colorful bubbletea terminal UI: a ranked list of dossiers with severity-reactive rows and KEV chips, a scrollable detail view with every outlet link and the full CVE card, and `o` to open a story in your browser
- A Markdown or JSON digest of the top-ranked clusters, for feeding a newsletter, a report, or another tool
- A single-CVE lookup that shows the enriched record and every stored story that mentions it
**Content ideation, optional AI, off by default**
- Turn ranked clusters into summaries and content angles through one `Provider` interface: a local Qwen model via Ollama (keyless default), or Claude, OpenAI, or Gemini behind a single pasted key
- A re-runnable in-binary setup wizard writes keys to a `0600` credentials file behind a name allowlist, read from the environment and never logged
**Watch daemon, optional**
- A long-lived scheduler that re-ingests on a timer and posts genuinely new, high-signal stories to Slack, Discord, or any JSON webhook
- New is measured on fetch time, not publish time, so a backfilled advisory fetched today still alerts exactly once; the daemon shuts down cleanly on Ctrl-C or SIGTERM and never crashes on a transient feed or network error
## Quick Start
```bash
curl -fsSL https://angelamos.com/nadezhda/install.sh | bash
```
One command, zero further steps: it grabs a prebuilt binary for your platform (no Go toolchain needed), drops it on your `PATH`, and leaves `nadezhda` runnable by name. Then pull the news and browse it:
```bash
nadezhda scrape # pull every enabled feed, cluster, and enrich CVEs
nadezhda tui # browse the ranked dossier (? for keys, o opens a story)
nadezhda digest --top 20 # render a ranked Markdown digest (or --format json)
nadezhda cve CVE-2021-44228 # one enriched CVE and the stories that mention it
```
The default flow is `scrape` then `tui`; scrape already runs enrichment best-effort, so there is nothing else to wire up. A scrape prints a per-source table and a one-line summary:
```
SOURCE STATUS PARSED NEW DUP CVE ERR
krebs ok 10 3 7 1 0
bleepingcomputer ok 18 6 12 4 0
cisa 304 - - - - -
...
12 new, 45 duplicate, 8 CVE refs across 7 sources (0 failed)
132 clusters (14 multi-source, largest 5)
enriched 80/80 CVEs (14 KEV, 0 not found)
```
The optional layers are one command each:
```bash
nadezhda ai # set up AI ideation: paste one key, or point at a local Ollama
nadezhda watch --interval 1h # run as a daemon, alerting new high-signal stories to a webhook
```
Prefer the Go toolchain? A real `go install github.com/CarterPerez-dev/nadezhda/cmd/nadezhda@latest` works too, and `just build` builds from a checkout. Building from source needs Go 1.25+, fetched automatically if you are on an older Go.
> [!TIP]
> This project uses [`just`](https://github.com/casey/just) as a command runner. Type `just` to see every recipe.
>
> Install: `curl -sSf https://just.systems/install.sh | bash -s -- --to ~/.local/bin`
## Learn
This project ships a full teaching track. Read it in order, or jump to what you need.
| Doc | What it covers |
|-----|----------------|
| [`learn/00-OVERVIEW.md`](learn/00-OVERVIEW.md) | What the tool is, prerequisites, the project layout, and a quick tour |
| [`learn/01-CONCEPTS.md`](learn/01-CONCEPTS.md) | The CVE intelligence stack, clustering as a velocity signal, and the ranking model, grounded in real incidents |
| [`learn/02-ARCHITECTURE.md`](learn/02-ARCHITECTURE.md) | The pipeline, the package map, the data model, and the design decisions |
| [`learn/03-IMPLEMENTATION.md`](learn/03-IMPLEMENTATION.md) | A code walkthrough from a feed item to a ranked, enriched cluster |
| [`learn/04-CHALLENGES.md`](learn/04-CHALLENGES.md) | Extension ideas, from adding a source to replacing the clustering with LSH |
## Architecture
One Go module, no service boundary, no message queue, and no external database. The `scrape` command runs the whole left-to-right flow once; the `watch` daemon runs it on a timer. Everything in `internal` exists to feed one function, `ingestAndCluster`, and everything downstream exists to rank and present what it produced.
```
sources.yaml (embedded default, or a user --config file)
│
▼
┌──────────────────────────────────────────────┐
│ fetch N workers, per-host rate limit, │ internal/fetch
│ conditional GET (ETag / 304) │
└───────────────────────┬──────────────────────┘
│ raw feed bytes
▼
┌──────────────────────────────────────────────┐
│ parse → normalize → ingest │ parse / normalize / ingest
│ RSS/Atom, canonical URL, hash, fail-soft │
└───────────────────────┬──────────────────────┘
CVE regex │ union-find
┌──────────┴──────────┐
▼ ▼
┌──────────────────────────────────────────────┐
│ enrich CVE list / KEV / EPSS, cached, │ enrich / cve
│ keyless │
└───────────────────────┬──────────────────────┘
▼
┌──────────────────────────────────────────────┐
│ rank deterministic, news-first score │ internal/rank
└───────────────────────┬──────────────────────┘
┌────────────┼────────────┬───────────┐
▼ ▼ ▼ ▼
digest tui ideate watch
(export) (browse) (AI) (daemon)
```
**Design decisions:** the store is `modernc.org/sqlite`, a pure-Go SQLite with no CGO, so the four-platform release binaries fall out of one build and the core needs no database server. Enrichment folds into `scrape` because the keyless CVE source is unthrottled, and it runs best-effort under a timeout so the news never blocks or fails on a slow CVE lookup. The `watch` daemon is a pure scheduler that imports nothing from the store or the pipeline; the concrete work is injected as a closure, so the loop, the graceful shutdown, and the fail-soft behavior are all unit-tested with a fake clock and no network. Alerts use a fetch-time watermark, not publish time, so an advisory published last week but fetched today is correctly new to you.
## Build and Test
```bash
just build # -> ./nadezhda (or: go build -o nadezhda ./cmd/nadezhda)
just test # go test ./... (18 packages)
just ollama-up # stand up the local Qwen runtime in Docker (optional AI)
```
Ranking is a pure function pinned to golden-order tests: fixed inputs produce one exact ordering. The keyless CVE client is pinned to a known-answer fixture, Log4Shell (CVE-2021-44228): the CVSS 10.0 score is read out of the CISA-ADP container because the vendor container holds only a placeholder, the weakness resolves to CWE-502, it is KEV-listed, and its EPSS score sits near the top of the scale. The `knownRansomwareCampaignUse` string mapping and the quoted-string EPSS parsing each have a regression test, because both are the quiet kind of bug that decodes wrong with no error. The watch scheduler is tested with a fake ticker a test drives by hand, so its behavior is verified with no clock, no network, and no database.
## Project Structure
```
security-news-scraper/
├── cmd/nadezhda/ # the CLI: one file per command + the shared pipeline seam
│ ├── scrape.go # the whole tool in miniature: runScrape
│ ├── pipeline.go # ingestAndCluster, the seam shared by scrape and watch
│ ├── tui.go # digest.go ideate.go watch.go cve.go list.go sources.go ai.go
│ └── main.go # root.go, the cobra command tree
├── internal/
│ ├── fetch/ # concurrent, rate-limited, conditional HTTP (+ an unused robots gate)
│ ├── parse/ # RSS/Atom via gofeed, HTML fallback via goquery
│ ├── normalize/ # canonical URLs, content hashing, time parsing
│ ├── ingest/ # the fan-out orchestrator, fail-soft per source
│ ├── cluster/ # union-find clustering by title similarity and shared CVE
│ ├── cve/ # keyless clients: cvelistV5, CISA KEV, FIRST EPSS (+ optional NVD)
│ ├── enrich/ # enrichment orchestration and the TTL cache
│ ├── rank/ # the pure, deterministic weighted score
│ ├── store/ # SQLite: connection, migrations, and every typed query
│ ├── ai/ # the opt-in ideation layer, four providers behind one interface
│ ├── setup/ # the in-binary credential wizard
│ ├── watch/ # the daemon scheduler, decoupled from the work it runs
│ ├── tui/ # the bubbletea terminal browser
│ ├── export/ # the Markdown and JSON digest renderers
│ ├── source/ # the source registry and the embedded sources.yaml
│ ├── config/ # every tunable, with defaults and validation
│ └── version/ # the ldflags-injected version string
├── testdata/ # captured feed and API fixtures, driving the offline tests
├── install.sh # the one-shot curl-able installer
├── learn/ # the teaching track (public)
└── justfile # every recipe
```
## License
[AGPL 3.0](LICENSE).

BIN
PROJECTS/intermediate/security-news-scraper/assets/tui.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 326 KiB

22
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/ai.go Normal file
View File

@ -0,0 +1,22 @@
// ©AngelaMos | 2026
// ai.go
package main
import (
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/setup"
)
var aiCmd = &cobra.Command{
Use: "ai",
Short: "Set up an AI provider for ideation (interactive, re-runnable)",
RunE: func(cmd *cobra.Command, args []string) error {
return setup.Run(cmd.InOrStdin(), cmd.OutOrStdout())
},
}
func init() {
rootCmd.AddCommand(aiCmd)
}

105
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/cve.go Normal file
View File

@ -0,0 +1,105 @@
// ©AngelaMos | 2026
// cve.go
package main
import (
"database/sql"
"errors"
"fmt"
"strings"
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const naValue = "n/a"
var cveCmd = &cobra.Command{
Use: "cve CVE-YYYY-NNNN",
Short: "Show an enriched CVE and the articles mentioning it",
Args: cobra.ExactArgs(1),
RunE: runCVE,
}
func init() {
rootCmd.AddCommand(cveCmd)
}
func runCVE(cmd *cobra.Command, args []string) error {
cfg, err := loadConfig()
if err != nil {
return err
}
st, err := store.Open(cfg.DBPath)
if err != nil {
return err
}
defer st.Close()
id := strings.ToUpper(args[0])
c, err := st.GetCVE(id)
if err != nil {
if errors.Is(err, sql.ErrNoRows) {
return fmt.Errorf("%s not found; run 'nadezhda scrape' to discover it", id)
}
return err
}
out := cmd.OutOrStdout()
fmt.Fprintf(out, "%s\n", c.ID)
if c.EnrichedAt == 0 {
fmt.Fprintln(out, " not enriched yet; run 'nadezhda enrich'")
} else {
fmt.Fprintf(out, " CVSS: %s\n", cvssLine(c))
fmt.Fprintf(out, " CWE: %s\n", orNA(c.CWE))
fmt.Fprintf(out, " KEV: %s\n", kevLine(c))
fmt.Fprintf(out, " EPSS: %s\n", epssLine(c))
if c.Description != "" {
fmt.Fprintf(out, "\n %s\n", c.Description)
}
}
articles, err := st.ArticlesForCVE(id)
if err != nil {
return err
}
fmt.Fprintf(out, "\nMentioned in %d article(s):\n", len(articles))
for _, a := range articles {
fmt.Fprintf(out, " [%s] %s\n %s\n", a.SourceName, a.Title, a.CanonicalURL)
}
return nil
}
func cvssLine(c store.CVE) string {
if c.CVSSScore == nil {
return naValue
}
return fmt.Sprintf("%.1f %s (%s) %s", *c.CVSSScore, orNA(c.CVSSSeverity), orNA(c.CVSSVersion), c.CVSSVector)
}
func kevLine(c store.CVE) string {
if !c.IsKEV {
return "no"
}
ransom := "no"
if c.KEVRansomware {
ransom = "yes"
}
return fmt.Sprintf("yes (added %s, ransomware: %s)", orNA(c.KEVDateAdded), ransom)
}
func epssLine(c store.CVE) string {
if c.EPSS == nil || c.EPSSPercentile == nil {
return naValue
}
return fmt.Sprintf("%.5f (percentile %.5f)", *c.EPSS, *c.EPSSPercentile)
}
func orNA(s string) string {
if s == "" {
return naValue
}
return s
}

101
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/digest.go Normal file
View File

@ -0,0 +1,101 @@
// ©AngelaMos | 2026
// digest.go
package main
import (
"fmt"
"os"
"time"
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/export"
"github.com/CarterPerez-dev/nadezhda/internal/rank"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const (
defaultDigestTop = 20
formatMarkdown = "md"
formatJSON = "json"
outFilePerm = 0o644
)
var (
digestTop int
digestSince string
digestFormat string
digestOut string
)
var digestCmd = &cobra.Command{
Use: "digest",
Short: "Render a ranked digest of story clusters to Markdown or JSON",
RunE: runDigest,
}
func init() {
digestCmd.Flags().IntVar(&digestTop, "top", defaultDigestTop, "show the top N ranked clusters")
digestCmd.Flags().StringVar(&digestSince, "since", "", "only clusters active within this window (e.g. 24h, 168h)")
digestCmd.Flags().StringVar(&digestFormat, "format", formatMarkdown, "output format: md or json")
digestCmd.Flags().StringVar(&digestOut, "out", "", "write to this file instead of stdout")
rootCmd.AddCommand(digestCmd)
}
func runDigest(cmd *cobra.Command, args []string) error {
cfg, err := loadConfig()
if err != nil {
return err
}
if digestFormat != formatMarkdown && digestFormat != formatJSON {
return fmt.Errorf("invalid --format %q: want md or json", digestFormat)
}
var since int64
now := time.Now()
if digestSince != "" {
d, err := time.ParseDuration(digestSince)
if err != nil {
return fmt.Errorf("invalid --since %q: %w", digestSince, err)
}
since = now.Add(-d).Unix()
}
st, err := store.Open(cfg.DBPath)
if err != nil {
return err
}
defer st.Close()
clusters, err := st.DigestClusters(since)
if err != nil {
return err
}
scored := rank.Rank(clusters, cfg.Rank, cfg.Watchlist, now)
shown := len(scored)
if digestTop > 0 && digestTop < shown {
shown = digestTop
}
var rendered string
if digestFormat == formatJSON {
rendered, err = export.JSON(scored, digestTop)
if err != nil {
return err
}
} else {
rendered = export.Markdown(scored, digestTop)
}
if digestOut != "" {
if err := os.WriteFile(digestOut, []byte(rendered), outFilePerm); err != nil {
return fmt.Errorf("write digest to %s: %w", digestOut, err)
}
fmt.Fprintf(cmd.OutOrStdout(), "wrote %d clusters to %s\n", shown, digestOut)
return nil
}
fmt.Fprint(cmd.OutOrStdout(), rendered)
return nil
}

76
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/enrich.go Normal file
View File

@ -0,0 +1,76 @@
// ©AngelaMos | 2026
// enrich.go
package main
import (
"fmt"
"net/http"
"os"
"time"
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/config"
"github.com/CarterPerez-dev/nadezhda/internal/cve"
"github.com/CarterPerez-dev/nadezhda/internal/enrich"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const nvdAPIKeyEnv = "NVD_API_KEY"
var enrichCmd = &cobra.Command{
Use: "enrich",
Short: "Refresh CVE intelligence (CVE List / NVD, CISA KEV, EPSS) for extracted CVEs",
Long: "Refresh CVE intelligence for extracted CVEs. Keyless by default (CVE Program, CISA KEV, EPSS); uses NVD only when NVD_API_KEY is set. scrape already runs this automatically, so this command is for manually refreshing.",
RunE: runEnrich,
}
func init() {
rootCmd.AddCommand(enrichCmd)
}
func runEnrich(cmd *cobra.Command, args []string) error {
cfg, err := loadConfig()
if err != nil {
return err
}
st, err := store.Open(cfg.DBPath)
if err != nil {
return err
}
defer st.Close()
stats, err := enrich.Run(cmd.Context(), st, buildEnrichClients(cfg), time.Now(), cfg.Enrich.CacheTTLHours, cfg.Enrich.NegativeTTLHours)
if err != nil {
return err
}
fmt.Fprintf(cmd.OutOrStdout(),
"enriched %d/%d CVEs (%d not found, %d KEV, %d errors)\n",
stats.Enriched, stats.Total, stats.NotFound, stats.KEVHits, stats.Errors)
return nil
}
func buildEnrichClients(cfg config.Config) enrich.Clients {
httpClient := &http.Client{Timeout: time.Duration(cfg.Fetch.TimeoutSeconds) * time.Second}
return enrich.Clients{
Core: buildCoreSource(httpClient, nvdAPIKey(cfg)),
KEV: cve.NewKEVClient(httpClient, cve.KEVEndpoint),
EPSS: cve.NewEPSSClient(httpClient, cve.EPSSEndpoint),
}
}
func buildCoreSource(httpClient *http.Client, apiKey string) cve.CVESource {
if apiKey != "" {
return cve.NewNVDClient(httpClient, cve.NVDEndpoint, apiKey)
}
return cve.NewCVEListClient(httpClient, cve.CVEListEndpoint)
}
func nvdAPIKey(cfg config.Config) string {
if k := os.Getenv(nvdAPIKeyEnv); k != "" {
return k
}
return cfg.Enrich.NVDAPIKey
}

181
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/ideate.go Normal file
View File

@ -0,0 +1,181 @@
// ©AngelaMos | 2026
// ideate.go
package main
import (
"encoding/json"
"errors"
"fmt"
"io"
"time"
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/ai"
"github.com/CarterPerez-dev/nadezhda/internal/rank"
"github.com/CarterPerez-dev/nadezhda/internal/setup"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const defaultIdeateTop = 10
var (
ideateTop int
ideateSince string
ideateProvider string
ideateForce bool
)
var ideateCmd = &cobra.Command{
Use: "ideate",
Short: "Generate content angles from ranked clusters via an AI provider (opt-in)",
RunE: runIdeate,
}
func init() {
ideateCmd.Flags().IntVar(&ideateTop, "top", defaultIdeateTop, "ideate the top N ranked clusters")
ideateCmd.Flags().StringVar(&ideateSince, "since", "", "only clusters active within this window (e.g. 24h, 168h)")
ideateCmd.Flags().StringVar(&ideateProvider, "provider", "", "override the configured provider: qwen|openai|anthropic|gemini")
ideateCmd.Flags().BoolVar(&ideateForce, "force", false, "re-ideate clusters that already have a note for this provider")
rootCmd.AddCommand(ideateCmd)
}
func runIdeate(cmd *cobra.Command, args []string) error {
cfg, err := loadConfig()
if err != nil {
return err
}
aiCfg := cfg.AI
if ideateProvider != "" {
aiCfg.Provider = ideateProvider
aiCfg.Enabled = true
}
if !aiCfg.Enabled {
if !isInteractive(cmd) {
return fmt.Errorf("AI is not set up — run `nadezhda ai` to configure a provider")
}
fmt.Fprintln(cmd.OutOrStdout(), "AI is not set up yet — let's fix that.")
if err := setup.Run(cmd.InOrStdin(), cmd.OutOrStdout()); err != nil {
return err
}
cfg, err = loadConfig()
if err != nil {
return err
}
aiCfg = cfg.AI
if !aiCfg.Enabled {
return fmt.Errorf("AI still not configured after setup")
}
if aiCfg.Provider == ai.ProviderQwen && !setup.OllamaReachable(aiCfg.Qwen.BaseURL) {
fmt.Fprintln(cmd.OutOrStdout(), "Ollama isn't reachable yet — finish the steps above, then run: nadezhda ideate")
return nil
}
}
provider, err := ai.Factory(aiCfg)
if err != nil {
return err
}
now := time.Now()
var since int64
if ideateSince != "" {
d, err := time.ParseDuration(ideateSince)
if err != nil {
return fmt.Errorf("invalid --since %q: %w", ideateSince, err)
}
since = now.Add(-d).Unix()
}
st, err := store.Open(cfg.DBPath)
if err != nil {
return err
}
defer st.Close()
clusters, err := st.DigestClusters(since)
if err != nil {
return err
}
scored := rank.Rank(clusters, cfg.Rank, cfg.Watchlist, now)
if ideateTop > 0 && ideateTop < len(scored) {
scored = scored[:ideateTop]
}
out := cmd.OutOrStdout()
ctx := cmd.Context()
var generated, skipped, refused, failed int
for _, s := range scored {
cid := s.Cluster.ClusterID
if !ideateForce {
exists, err := st.AINoteExists(cid, provider.Name())
if err != nil {
return err
}
if exists {
skipped++
fmt.Fprintf(out, "skip cluster %d (already ideated by %s; use --force)\n", cid, provider.Name())
continue
}
}
res, err := provider.Generate(ctx, ai.RequestFromCluster(s.Cluster))
if err != nil {
if errors.Is(err, ai.ErrRefused) {
refused++
fmt.Fprintf(out, "refused cluster %d (provider declined); skipping\n", cid)
} else {
failed++
fmt.Fprintf(out, "warn cluster %d: %v\n", cid, err)
}
continue
}
angles, err := json.Marshal(res.Angles)
if err != nil {
return err
}
note := store.AINote{
ClusterID: cid,
Provider: provider.Name(),
Summary: res.Summary,
Why: res.Why,
AnglesJSON: string(angles),
Format: res.Format,
CreatedAt: time.Now().Unix(),
}
if err := st.InsertAINote(note); err != nil {
return err
}
generated++
printIdeation(out, s, res)
}
fmt.Fprintf(out, "\nideated %d, skipped %d, refused %d, failed %d (provider: %s)\n", generated, skipped, refused, failed, provider.Name())
if generated == 0 && failed > 0 {
return fmt.Errorf("all %d ideation attempts failed", failed)
}
return nil
}
func printIdeation(out io.Writer, s rank.Scored, res ai.IdeationResult) {
fmt.Fprintf(out, "\n=== cluster %d score %.2f [%s] ===\n", s.Cluster.ClusterID, s.Score, res.Format)
fmt.Fprintf(out, "%s\n\n", clusterHeadline(s.Cluster))
fmt.Fprintf(out, "summary: %s\n\n", res.Summary)
fmt.Fprintf(out, "why: %s\n\n", res.Why)
fmt.Fprintln(out, "angles:")
for i, a := range res.Angles {
fmt.Fprintf(out, " %d. %s\n", i+1, a)
}
}
func clusterHeadline(c store.DigestCluster) string {
for _, a := range c.Articles {
if a.Title != "" {
return a.Title
}
}
return "(untitled cluster)"
}

133
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/ideate_test.go Normal file
View File

@ -0,0 +1,133 @@
// ©AngelaMos | 2026
// ideate_test.go
package main
import (
"bytes"
"context"
"encoding/json"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/setup"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
func seedCluster(t *testing.T, st *store.Store) {
t.Helper()
stmts := []string{
`INSERT INTO sources (id, name, url, type, weight, enabled) VALUES (1, 'krebs', 'https://krebsonsecurity.com', 'rss', 1.0, 1)`,
`INSERT INTO articles (id, source_id, canonical_url, content_hash, title, published_at) VALUES (1, 1, 'https://krebsonsecurity.com/a', 'h1', 'Critical RCE exploited in the wild', 1000)`,
`INSERT INTO clusters (id, cluster_key, first_seen, last_seen, size) VALUES (1, 'k1', 900, 1000, 1)`,
`INSERT INTO cluster_members (cluster_id, article_id) VALUES (1, 1)`,
}
for _, s := range stmts {
if _, err := st.DB().Exec(s); err != nil {
t.Fatalf("seed %q: %v", s, err)
}
}
}
func TestIdeateCommandEndToEnd(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
t.Setenv(setup.EnvProvider, "")
t.Setenv(setup.EnvQwenURL, "")
dir := t.TempDir()
dbPath := filepath.Join(dir, "test.db")
st, err := store.Open(dbPath)
if err != nil {
t.Fatalf("open store: %v", err)
}
seedCluster(t, st)
st.Close()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, _ := json.Marshal(map[string]any{
"choices": []map[string]any{{"message": map[string]any{
"content": `{"summary":"a critical rce","why":"widely exploited","angles":["hook one","hook two","hook three"],"format":"newsletter"}`,
}}},
})
_, _ = w.Write(body)
}))
defer srv.Close()
cfgPath := filepath.Join(dir, "config.yaml")
cfgYAML := "ai:\n enabled: true\n provider: qwen\n qwen:\n base_url: " + srv.URL + "\n model: qwen2.5:7b\n"
if err := os.WriteFile(cfgPath, []byte(cfgYAML), 0o644); err != nil {
t.Fatal(err)
}
prevCfg, prevDB, prevTop, prevProv, prevForce, prevSince := flagConfig, flagDB, ideateTop, ideateProvider, ideateForce, ideateSince
defer func() {
flagConfig, flagDB, ideateTop, ideateProvider, ideateForce, ideateSince = prevCfg, prevDB, prevTop, prevProv, prevForce, prevSince
}()
flagConfig, flagDB, ideateTop, ideateProvider, ideateForce, ideateSince = cfgPath, dbPath, 1, "", false, ""
cmd := &cobra.Command{}
cmd.SetContext(context.Background())
var buf bytes.Buffer
cmd.SetOut(&buf)
if err := runIdeate(cmd, nil); err != nil {
t.Fatalf("runIdeate: %v\noutput:\n%s", err, buf.String())
}
out := buf.String()
for _, want := range []string{"ideated 1", "a critical rce", "hook one", "newsletter"} {
if !strings.Contains(out, want) {
t.Errorf("ideate output missing %q\n---\n%s", want, out)
}
}
st2, err := store.Open(dbPath)
if err != nil {
t.Fatal(err)
}
ok, err := st2.AINoteExists(1, "qwen")
st2.Close()
if err != nil || !ok {
t.Fatalf("note not persisted: ok=%v err=%v", ok, err)
}
buf.Reset()
if err := runIdeate(cmd, nil); err != nil {
t.Fatalf("second runIdeate: %v", err)
}
if !strings.Contains(buf.String(), "skip cluster 1") {
t.Errorf("second run should skip existing note:\n%s", buf.String())
}
buf.Reset()
ideateForce = true
if err := runIdeate(cmd, nil); err != nil {
t.Fatalf("force runIdeate: %v", err)
}
if !strings.Contains(buf.String(), "ideated 1") {
t.Errorf("--force should regenerate, not skip:\n%s", buf.String())
}
}
func TestIdeateDisabledErrors(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
t.Setenv(setup.EnvProvider, "")
t.Setenv(setup.EnvQwenURL, "")
prevCfg, prevProv := flagConfig, ideateProvider
defer func() { flagConfig, ideateProvider = prevCfg, prevProv }()
flagConfig, ideateProvider = "", ""
cmd := &cobra.Command{}
cmd.SetContext(context.Background())
cmd.SetOut(&bytes.Buffer{})
cmd.SetIn(strings.NewReader(""))
err := runIdeate(cmd, nil)
if err == nil || !strings.Contains(err.Error(), "not set up") {
t.Fatalf("expected a 'not set up' error, got %v", err)
}
}

92
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/list.go Normal file
View File

@ -0,0 +1,92 @@
// ©AngelaMos | 2026
// list.go
package main
import (
"fmt"
"time"
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const (
defaultListLimit = 50
dateLayout = "2006-01-02"
noDate = "----------"
)
var (
listSource string
listSince string
listMinCVSS float64
listKEV bool
listKeyword string
listLimit int
)
var listCmd = &cobra.Command{
Use: "list",
Short: "List stored articles with filters",
RunE: runList,
}
func init() {
listCmd.Flags().StringVar(&listSource, "source", "", "filter by source name")
listCmd.Flags().StringVar(&listSince, "since", "", "only articles published within this window (e.g. 24h, 168h)")
listCmd.Flags().Float64Var(&listMinCVSS, "min-cvss", 0, "only articles referencing a CVE with CVSS >= this")
listCmd.Flags().BoolVar(&listKEV, "kev", false, "only articles referencing a KEV-listed CVE")
listCmd.Flags().StringVar(&listKeyword, "keyword", "", "filter by keyword in title or summary")
listCmd.Flags().IntVar(&listLimit, "limit", defaultListLimit, "maximum rows to show")
rootCmd.AddCommand(listCmd)
}
func runList(cmd *cobra.Command, args []string) error {
cfg, err := loadConfig()
if err != nil {
return err
}
filter := store.ListFilter{
Source: listSource,
MinCVSS: listMinCVSS,
KEV: listKEV,
Keyword: listKeyword,
Limit: listLimit,
}
if listSince != "" {
d, err := time.ParseDuration(listSince)
if err != nil {
return fmt.Errorf("invalid --since %q: %w", listSince, err)
}
filter.Since = time.Now().Add(-d).Unix()
}
st, err := store.Open(cfg.DBPath)
if err != nil {
return err
}
defer st.Close()
articles, err := st.ListArticles(filter)
if err != nil {
return err
}
out := cmd.OutOrStdout()
fmt.Fprintf(out, "%-10s %-16s %s\n", "DATE", "SOURCE", "TITLE")
for _, a := range articles {
fmt.Fprintf(out, "%-10s %-16s %s\n", formatDate(a.PublishedAt), a.SourceName, a.Title)
}
fmt.Fprintf(out, "\n%d article(s)\n", len(articles))
return nil
}
func formatDate(unix int64) string {
if unix == 0 {
return noDate
}
return time.Unix(unix, 0).UTC().Format(dateLayout)
}

19
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/main.go Normal file
View File

@ -0,0 +1,19 @@
// ©AngelaMos | 2026
// main.go
package main
import (
"context"
"os"
"os/signal"
"syscall"
)
func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if err := rootCmd.ExecuteContext(ctx); err != nil {
os.Exit(1)
}
}

30
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/pipeline.go Normal file
View File

@ -0,0 +1,30 @@
// ©AngelaMos | 2026
// pipeline.go
package main
import (
"context"
"time"
"github.com/CarterPerez-dev/nadezhda/internal/cluster"
"github.com/CarterPerez-dev/nadezhda/internal/config"
"github.com/CarterPerez-dev/nadezhda/internal/fetch"
"github.com/CarterPerez-dev/nadezhda/internal/ingest"
"github.com/CarterPerez-dev/nadezhda/internal/source"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
func ingestAndCluster(ctx context.Context, fc *fetch.Client, st *store.Store, cfg config.Config, targets []source.Source, start time.Time) (ingest.Summary, cluster.Stats, error) {
summary, err := ingest.Run(ctx, fc, st, cfg, targets, start)
if err != nil {
return ingest.Summary{}, cluster.Stats{}, err
}
sinceUnix := start.Unix() - int64(cfg.Cluster.LookbackHours)*secondsPerHour
windowSeconds := int64(cfg.Cluster.WindowHours) * secondsPerHour
stats, err := cluster.Rebuild(st, cfg.Cluster.TitleJaccard, windowSeconds, sinceUnix)
if err != nil {
return summary, cluster.Stats{}, err
}
return summary, stats, nil
}

60
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/root.go Normal file
View File

@ -0,0 +1,60 @@
// ©AngelaMos | 2026
// root.go
package main
import (
"os"
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/config"
"github.com/CarterPerez-dev/nadezhda/internal/setup"
)
var (
flagConfig string
flagDB string
)
var rootCmd = &cobra.Command{
Use: "nadezhda",
Short: "Security news and CVE aggregation engine",
Long: "Nadezhda aggregates cybersecurity news, enriches CVEs with NVD, CISA KEV, and EPSS intelligence, clusters stories across outlets, and ranks what matters.",
SilenceUsage: true,
}
func init() {
rootCmd.PersistentFlags().StringVar(&flagConfig, "config", "", "path to config.yaml (optional)")
rootCmd.PersistentFlags().StringVar(&flagDB, "db", "", "override database path")
}
func loadConfig() (config.Config, error) {
if err := setup.Load(); err != nil {
return config.Config{}, err
}
cfg, err := config.Load(flagConfig)
if err != nil {
return config.Config{}, err
}
if flagDB != "" {
cfg.DBPath = flagDB
}
if p := os.Getenv(setup.EnvProvider); p != "" {
cfg.AI.Enabled = true
cfg.AI.Provider = p
}
if u := os.Getenv(setup.EnvQwenURL); u != "" {
cfg.AI.Qwen.BaseURL = u
}
return cfg, nil
}
func isInteractive(cmd *cobra.Command) bool {
f, ok := cmd.InOrStdin().(*os.File)
if !ok {
return false
}
fi, err := f.Stat()
return err == nil && fi.Mode()&os.ModeCharDevice != 0
}

162
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/scrape.go Normal file
View File

@ -0,0 +1,162 @@
// ©AngelaMos | 2026
// scrape.go
package main
import (
"context"
"fmt"
"time"
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/config"
"github.com/CarterPerez-dev/nadezhda/internal/enrich"
"github.com/CarterPerez-dev/nadezhda/internal/fetch"
"github.com/CarterPerez-dev/nadezhda/internal/ingest"
"github.com/CarterPerez-dev/nadezhda/internal/source"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const secondsPerHour = 3600
const enrichBudget = 5 * time.Minute
const (
statusNotModified = "304"
statusError = "error"
statusOK = "ok"
dash = "-"
)
var (
scrapeSource string
scrapeNoEnrich bool
)
var scrapeCmd = &cobra.Command{
Use: "scrape",
Short: "Ingest all enabled sources once",
RunE: runScrape,
}
func init() {
scrapeCmd.Flags().StringVar(&scrapeSource, "source", "", "ingest only this source by name")
scrapeCmd.Flags().BoolVar(&scrapeNoEnrich, "no-enrich", false, "skip the keyless CVE enrichment pass")
rootCmd.AddCommand(scrapeCmd)
}
func runScrape(cmd *cobra.Command, args []string) error {
cfg, err := loadConfig()
if err != nil {
return err
}
srcs, err := source.Load(cfg.SourcesPath)
if err != nil {
return err
}
targets, err := selectTargets(srcs, scrapeSource)
if err != nil {
return err
}
st, err := store.Open(cfg.DBPath)
if err != nil {
return err
}
defer st.Close()
fc := fetch.New(fetch.Options{
UserAgent: cfg.Fetch.UserAgent,
PerHostRate: cfg.Fetch.PerHostRate,
PerHostBurst: cfg.Fetch.PerHostBurst,
Timeout: time.Duration(cfg.Fetch.TimeoutSeconds) * time.Second,
MaxRetries: cfg.Fetch.MaxRetries,
})
ctx := cmd.Context()
now := time.Now()
summary, cstats, err := ingestAndCluster(ctx, fc, st, cfg, targets, now)
if err != nil {
return err
}
printSummary(cmd, summary)
fmt.Fprintf(cmd.OutOrStdout(), "%d clusters (%d multi-source, largest %d)\n",
cstats.Total, cstats.MultiSource, cstats.LargestSize)
if !scrapeNoEnrich {
enrichAfterScrape(ctx, cmd, cfg, st)
}
return nil
}
func enrichAfterScrape(ctx context.Context, cmd *cobra.Command, cfg config.Config, st *store.Store) {
out := cmd.OutOrStdout()
ectx, cancel := context.WithTimeout(ctx, enrichBudget)
defer cancel()
stats, err := enrich.Run(ectx, st, buildEnrichClients(cfg), time.Now(), cfg.Enrich.CacheTTLHours, cfg.Enrich.NegativeTTLHours)
if err != nil {
if done := stats.Enriched + stats.NotFound; done > 0 {
fmt.Fprintf(out, "enriched %d/%d CVEs before stopping: %v\n", done, stats.Total, err)
} else {
fmt.Fprintf(out, "enrich skipped: %v (news is unaffected)\n", err)
}
return
}
if stats.Total == 0 {
return
}
fmt.Fprintf(out, "enriched %d/%d CVEs (%d KEV, %d not found)\n",
stats.Enriched, stats.Total, stats.KEVHits, stats.NotFound)
}
func selectTargets(srcs []source.Source, only string) ([]source.Source, error) {
if only != "" {
for _, s := range srcs {
if s.Name == only {
return []source.Source{s}, nil
}
}
return nil, fmt.Errorf("scrape: unknown source %q", only)
}
return source.Enabled(srcs), nil
}
func printSummary(cmd *cobra.Command, summary ingest.Summary) {
out := cmd.OutOrStdout()
fmt.Fprintf(out, "%-18s %-8s %-8s %-5s %-5s %-5s %-5s\n", "SOURCE", "STATUS", "PARSED", "NEW", "DUP", "CVE", "ERR")
totalCVEs := 0
for _, r := range summary.Results {
totalCVEs += r.CVEs
fmt.Fprintf(out, "%-18s %-8s %-8s %-5s %-5s %-5s %-5s\n",
r.Name, status(r), count(r, r.Parsed), count(r, r.New), count(r, r.Duplicates), count(r, r.CVEs), count(r, r.ItemErrors))
}
newArticles, duplicates, failed := summary.Totals()
fmt.Fprintf(out, "\n%d new, %d duplicate, %d CVE refs across %d sources (%d failed)\n",
newArticles, duplicates, totalCVEs, len(summary.Results), failed)
for _, r := range summary.Results {
if r.Err != nil {
fmt.Fprintf(out, " %s: %v\n", r.Name, r.Err)
}
}
}
func status(r ingest.SourceResult) string {
switch {
case r.Err != nil:
return statusError
case r.NotModified:
return statusNotModified
default:
return statusOK
}
}
func count(r ingest.SourceResult, n int) string {
if r.Err != nil || r.NotModified {
return dash
}
return fmt.Sprintf("%d", n)
}

60
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/sources.go Normal file
View File

@ -0,0 +1,60 @@
// ©AngelaMos | 2026
// sources.go
package main
import (
"fmt"
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/source"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
var sourcesCmd = &cobra.Command{
Use: "sources",
Short: "List configured sources and persist them to the store",
RunE: runSources,
}
func init() {
rootCmd.AddCommand(sourcesCmd)
}
func runSources(cmd *cobra.Command, args []string) error {
cfg, err := loadConfig()
if err != nil {
return err
}
srcs, err := source.Load(cfg.SourcesPath)
if err != nil {
return err
}
st, err := store.Open(cfg.DBPath)
if err != nil {
return err
}
defer st.Close()
for _, s := range srcs {
if _, err := st.UpsertSource(store.SourceInput{
Name: s.Name, Title: s.Title, URL: s.URL, Type: string(s.Type),
Weight: s.Weight, Tags: s.Tags, Enabled: s.Enabled,
}); err != nil {
return err
}
}
fmt.Printf("%-18s %-8s %-7s %s\n", "NAME", "ENABLED", "WEIGHT", "URL")
for _, s := range srcs {
enabled := "no"
if s.Enabled {
enabled = "yes"
}
fmt.Printf("%-18s %-8s %-7.2f %s\n", s.Name, enabled, s.Weight, s.URL)
}
fmt.Printf("\n%d sources (%d enabled) persisted to %s\n",
len(srcs), len(source.Enabled(srcs)), cfg.DBPath)
return nil
}

117
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/tui.go Normal file
View File

@ -0,0 +1,117 @@
// ©AngelaMos | 2026
// tui.go
package main
import (
"encoding/json"
"fmt"
"time"
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/ai"
"github.com/CarterPerez-dev/nadezhda/internal/rank"
"github.com/CarterPerez-dev/nadezhda/internal/store"
"github.com/CarterPerez-dev/nadezhda/internal/tui"
)
var tuiSince string
var tuiCmd = &cobra.Command{
Use: "tui",
Short: "Browse aggregated news in an interactive terminal UI",
RunE: runTUI,
}
func init() {
tuiCmd.Flags().StringVar(&tuiSince, "since", "", "only clusters active within this window (e.g. 24h, 168h)")
rootCmd.AddCommand(tuiCmd)
}
func runTUI(cmd *cobra.Command, args []string) error {
cfg, err := loadConfig()
if err != nil {
return err
}
now := time.Now()
var since int64
if tuiSince != "" {
d, err := time.ParseDuration(tuiSince)
if err != nil {
return fmt.Errorf("invalid --since %q: %w", tuiSince, err)
}
since = now.Add(-d).Unix()
}
st, err := store.Open(cfg.DBPath)
if err != nil {
return err
}
defer st.Close()
loader := func() (tui.Data, error) {
clusters, err := st.DigestClusters(since)
if err != nil {
return tui.Data{}, err
}
scored := rank.Rank(clusters, cfg.Rank, cfg.Watchlist, now)
detail := make(map[string]store.CVE)
for _, s := range scored {
for _, v := range s.Cluster.CVEs {
if _, ok := detail[v.ID]; ok {
continue
}
full, err := st.GetCVE(v.ID)
if err != nil {
continue
}
detail[v.ID] = full
}
}
notes := map[int64]ai.IdeationResult{}
if persisted, err := st.LatestAINotes(); err == nil {
for cid, n := range persisted {
var angles []string
_ = json.Unmarshal([]byte(n.AnglesJSON), &angles)
notes[cid] = ai.IdeationResult{Summary: n.Summary, Why: n.Why, Angles: angles, Format: n.Format}
}
}
return tui.Data{Scored: scored, CVEDetail: detail, Notes: notes}, nil
}
var ideator tui.Ideator
if cfg.AI.Enabled {
provider, err := ai.Factory(cfg.AI)
if err != nil {
return err
}
ctx := cmd.Context()
ideator = func(c store.DigestCluster) (ai.IdeationResult, error) {
res, err := provider.Generate(ctx, ai.RequestFromCluster(c))
if err != nil {
return ai.IdeationResult{}, err
}
angles, err := json.Marshal(res.Angles)
if err != nil {
return ai.IdeationResult{}, err
}
note := store.AINote{
ClusterID: c.ClusterID,
Provider: provider.Name(),
Summary: res.Summary,
Why: res.Why,
AnglesJSON: string(angles),
Format: res.Format,
CreatedAt: time.Now().Unix(),
}
if err := st.InsertAINote(note); err != nil {
return ai.IdeationResult{}, fmt.Errorf("save note: %w", err)
}
return res, nil
}
}
return tui.Run(loader, ideator)
}

25
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/version.go Normal file
View File

@ -0,0 +1,25 @@
// ©AngelaMos | 2026
// version.go
package main
import (
"fmt"
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/version"
)
var versionCmd = &cobra.Command{
Use: "version",
Short: "Print version",
RunE: func(cmd *cobra.Command, args []string) error {
fmt.Printf("%s %s\n", version.Name, version.Version)
return nil
},
}
func init() {
rootCmd.AddCommand(versionCmd)
}

258
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/watch.go Normal file
View File

@ -0,0 +1,258 @@
// ©AngelaMos | 2026
// watch.go
package main
import (
"context"
"fmt"
"io"
"net/http"
"net/url"
"time"
"github.com/spf13/cobra"
"github.com/CarterPerez-dev/nadezhda/internal/config"
"github.com/CarterPerez-dev/nadezhda/internal/enrich"
"github.com/CarterPerez-dev/nadezhda/internal/fetch"
"github.com/CarterPerez-dev/nadezhda/internal/rank"
"github.com/CarterPerez-dev/nadezhda/internal/source"
"github.com/CarterPerez-dev/nadezhda/internal/store"
"github.com/CarterPerez-dev/nadezhda/internal/watch"
)
const untitledCluster = "(untitled cluster)"
var (
watchInterval string
watchOnce bool
watchNoEnrich bool
)
var watchCmd = &cobra.Command{
Use: "watch",
Short: "Run as a daemon, re-ingesting on an interval with an optional webhook notify",
Long: "Run nadezhda as a long-lived daemon that re-ingests every enabled source on an interval " +
"(default from config, override with --interval). Each cycle scrapes, clusters, and enriches " +
"exactly like the scrape command. When watch.webhook_url is set, genuinely new high-signal " +
"stories are POSTed to that webhook (Slack, Discord, or any JSON endpoint).",
RunE: runWatch,
}
func init() {
watchCmd.Flags().StringVar(&watchInterval, "interval", "", "re-ingest interval, e.g. 30m or 1h (overrides watch.interval in config)")
watchCmd.Flags().BoolVar(&watchOnce, "once", false, "run a single cycle and exit instead of looping")
watchCmd.Flags().BoolVar(&watchNoEnrich, "no-enrich", false, "skip the keyless CVE enrichment pass each cycle")
rootCmd.AddCommand(watchCmd)
}
func runWatch(cmd *cobra.Command, args []string) error {
cfg, err := loadConfig()
if err != nil {
return err
}
interval, err := resolveWatchInterval(cfg)
if err != nil {
return err
}
if cfg.Watch.WebhookURL != "" {
if err := validateWebhookURL(cfg.Watch.WebhookURL); err != nil {
return err
}
}
srcs, err := source.Load(cfg.SourcesPath)
if err != nil {
return err
}
targets := source.Enabled(srcs)
if len(targets) == 0 {
return fmt.Errorf("watch: no enabled sources to poll")
}
st, err := store.Open(cfg.DBPath)
if err != nil {
return err
}
defer st.Close()
fc := fetch.New(fetch.Options{
UserAgent: cfg.Fetch.UserAgent,
PerHostRate: cfg.Fetch.PerHostRate,
PerHostBurst: cfg.Fetch.PerHostBurst,
Timeout: time.Duration(cfg.Fetch.TimeoutSeconds) * time.Second,
MaxRetries: cfg.Fetch.MaxRetries,
})
out := cmd.ErrOrStderr()
if watchNoEnrich && cfg.Watch.NotifyOnKEV {
fmt.Fprintln(out, "watch: --no-enrich with notify_on_kev set: KEV status is never computed, so KEV alerts will not fire")
}
cycle := func(ctx context.Context) (watch.Report, error) {
start := time.Now()
summary, cstats, err := ingestAndCluster(ctx, fc, st, cfg, targets, start)
if err != nil {
return watch.Report{}, err
}
newArticles, duplicates, failed := summary.Totals()
var enriched, kevHits int
if !watchNoEnrich {
enriched, kevHits = watchEnrich(ctx, out, st, cfg)
}
notable, err := buildNotable(st, cfg, start)
if err != nil {
return watch.Report{}, err
}
return watch.Report{
Start: start,
Duration: time.Since(start),
NewArticles: newArticles,
Duplicates: duplicates,
Clusters: cstats.Total,
Enriched: enriched,
KEVHits: kevHits,
Failed: failed,
Notable: notable,
}, nil
}
opts := watch.Options{
Interval: interval,
RunAtStart: true,
Cycle: cycle,
Out: out,
}
if cfg.Watch.WebhookURL != "" {
opts.Notifier = watch.WebhookNotifier{
URL: cfg.Watch.WebhookURL,
Client: &http.Client{Timeout: time.Duration(cfg.Fetch.TimeoutSeconds) * time.Second},
}
}
if watchOnce {
return watch.Once(cmd.Context(), opts)
}
return watch.Run(cmd.Context(), opts)
}
func resolveWatchInterval(cfg config.Config) (time.Duration, error) {
raw := cfg.Watch.Interval
if watchInterval != "" {
raw = watchInterval
}
d, err := time.ParseDuration(raw)
if err != nil {
return 0, fmt.Errorf("watch: invalid interval %q: %w", raw, err)
}
if d < config.MinWatchInterval {
return 0, fmt.Errorf("watch: interval %s is below the minimum %s", d, config.MinWatchInterval)
}
return d, nil
}
func validateWebhookURL(raw string) error {
u, err := url.Parse(raw)
if err != nil {
return fmt.Errorf("watch: invalid webhook_url %q: %w", raw, err)
}
if u.Scheme != "http" && u.Scheme != "https" {
return fmt.Errorf("watch: webhook_url must be http or https, got %q", raw)
}
if u.Host == "" {
return fmt.Errorf("watch: webhook_url must include a host, got %q", raw)
}
return nil
}
func watchEnrich(ctx context.Context, out io.Writer, st *store.Store, cfg config.Config) (enriched, kevHits int) {
ectx, cancel := context.WithTimeout(ctx, enrichBudget)
defer cancel()
stats, err := enrich.Run(ectx, st, buildEnrichClients(cfg), time.Now(), cfg.Enrich.CacheTTLHours, cfg.Enrich.NegativeTTLHours)
if err != nil && ctx.Err() == nil {
fmt.Fprintf(out, "watch: enrich degraded, KEV/CVE data may be stale: %v\n", err)
}
return stats.Enriched, stats.KEVHits
}
func buildNotable(st *store.Store, cfg config.Config, cycleStart time.Time) ([]watch.NotableItem, error) {
fresh, err := st.NewlyFetchedClusters(cycleStart.Unix())
if err != nil {
return nil, err
}
scored := rank.Rank(fresh, cfg.Rank, cfg.Watchlist, time.Now())
items := make([]watch.NotableItem, 0, cfg.Watch.NotifyMaxItems)
for _, sc := range scored {
if len(items) >= cfg.Watch.NotifyMaxItems {
break
}
if isNotable(sc, cfg.Watch) {
items = append(items, toNotable(sc))
}
}
return items, nil
}
func isNotable(sc rank.Scored, w config.Watch) bool {
if sc.Score >= w.NotifyMinScore {
return true
}
if w.NotifyOnKEV {
for _, v := range sc.Cluster.CVEs {
if v.IsKEV {
return true
}
}
}
return false
}
func toNotable(sc rank.Scored) watch.NotableItem {
c := sc.Cluster
title, link := representativeArticle(c.Articles)
var maxCVSS float64
var isKEV bool
cves := make([]string, 0, len(c.CVEs))
for _, v := range c.CVEs {
cves = append(cves, v.ID)
if v.CVSSScore != nil && *v.CVSSScore > maxCVSS {
maxCVSS = *v.CVSSScore
}
if v.IsKEV {
isKEV = true
}
}
return watch.NotableItem{
Title: title,
URL: link,
Score: sc.Score,
MaxCVSS: maxCVSS,
IsKEV: isKEV,
CVEs: cves,
Sources: distinctSources(c.Articles),
}
}
func representativeArticle(articles []store.DigestArticle) (title, link string) {
if len(articles) == 0 {
return untitledCluster, ""
}
best := articles[0]
for _, a := range articles[1:] {
if a.SourceWeight > best.SourceWeight {
best = a
}
}
return best.Title, best.CanonicalURL
}
func distinctSources(articles []store.DigestArticle) int {
set := make(map[string]struct{}, len(articles))
for _, a := range articles {
set[a.SourceName] = struct{}{}
}
return len(set)
}

149
PROJECTS/intermediate/security-news-scraper/cmd/nadezhda/watch_test.go Normal file
View File

@ -0,0 +1,149 @@
// ©AngelaMos | 2026
// watch_test.go
package main
import (
"fmt"
"path/filepath"
"strconv"
"testing"
"time"
"github.com/CarterPerez-dev/nadezhda/internal/config"
"github.com/CarterPerez-dev/nadezhda/internal/rank"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
func TestIsNotable(t *testing.T) {
kevCluster := store.DigestCluster{CVEs: []store.DigestCVE{{ID: "CVE-2026-1", IsKEV: true}}}
plainCluster := store.DigestCluster{}
tests := []struct {
name string
scored rank.Scored
watch config.Watch
want bool
}{
{"score above threshold", rank.Scored{Cluster: plainCluster, Score: 0.9}, config.Watch{NotifyMinScore: 0.5}, true},
{"score below threshold, not kev", rank.Scored{Cluster: plainCluster, Score: 0.2}, config.Watch{NotifyMinScore: 0.5}, false},
{"below threshold but kev with notify_on_kev", rank.Scored{Cluster: kevCluster, Score: 0.1}, config.Watch{NotifyMinScore: 0.5, NotifyOnKEV: true}, true},
{"kev but notify_on_kev off", rank.Scored{Cluster: kevCluster, Score: 0.1}, config.Watch{NotifyMinScore: 0.5, NotifyOnKEV: false}, false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := isNotable(tt.scored, tt.watch); got != tt.want {
t.Errorf("isNotable = %v, want %v", got, tt.want)
}
})
}
}
func TestToNotablePicksHighestAuthoritySource(t *testing.T) {
cvss := 9.8
sc := rank.Scored{
Score: 0.77,
Cluster: store.DigestCluster{
Articles: []store.DigestArticle{
{Title: "low authority", CanonicalURL: "https://low", SourceName: "low", SourceWeight: 0.5},
{Title: "high authority", CanonicalURL: "https://high", SourceName: "high", SourceWeight: 1.0},
},
CVEs: []store.DigestCVE{{ID: "CVE-2026-1", CVSSScore: &cvss, IsKEV: true}},
},
}
n := toNotable(sc)
if n.Title != "high authority" || n.URL != "https://high" {
t.Errorf("representative should be the highest-source-weight article, got %q / %q", n.Title, n.URL)
}
if !n.IsKEV || n.MaxCVSS != 9.8 {
t.Errorf("cve signals wrong: kev=%v cvss=%v", n.IsKEV, n.MaxCVSS)
}
if n.Sources != 2 {
t.Errorf("distinct sources = %d, want 2", n.Sources)
}
if len(n.CVEs) != 1 || n.CVEs[0] != "CVE-2026-1" {
t.Errorf("cves wrong: %v", n.CVEs)
}
if n.Score != 0.77 {
t.Errorf("score = %v, want 0.77", n.Score)
}
}
func TestResolveWatchInterval(t *testing.T) {
cfg := config.Default()
watchInterval = ""
defer func() { watchInterval = "" }()
if d, err := resolveWatchInterval(cfg); err != nil || d != time.Hour {
t.Errorf("default interval = %v, %v; want 1h, nil", d, err)
}
watchInterval = "30m"
if d, err := resolveWatchInterval(cfg); err != nil || d != 30*time.Minute {
t.Errorf("flag override = %v, %v; want 30m, nil", d, err)
}
watchInterval = "5s"
if _, err := resolveWatchInterval(cfg); err == nil {
t.Error("expected an error for a sub-minimum interval")
}
watchInterval = "nonsense"
if _, err := resolveWatchInterval(cfg); err == nil {
t.Error("expected an error for a garbage interval")
}
}
func TestValidateWebhookURL(t *testing.T) {
good := []string{"https://hooks.slack.com/services/x", "http://127.0.0.1:39871/hook"}
for _, u := range good {
if err := validateWebhookURL(u); err != nil {
t.Errorf("validateWebhookURL(%q) = %v, want nil", u, err)
}
}
bad := []string{"example.com/hook", "ftp://x/y", "not a url", ""}
for _, u := range bad {
if err := validateWebhookURL(u); err == nil {
t.Errorf("validateWebhookURL(%q) = nil, want error", u)
}
}
}
func TestBuildNotableCapsToNotifyMax(t *testing.T) {
st, err := store.Open(filepath.Join(t.TempDir(), "watch.db"))
if err != nil {
t.Fatal(err)
}
defer st.Close()
src, err := st.UpsertSource(store.SourceInput{Name: "s", URL: "https://s/f", Type: "rss", Weight: 1, Enabled: true})
if err != nil {
t.Fatal(err)
}
var rows []store.ClusterRow
for i := 0; i < 5; i++ {
id, err := st.InsertArticle(store.Article{
SourceID: src, CanonicalURL: fmt.Sprintf("https://s/%d", i),
ContentHash: "c" + strconv.Itoa(i), TitleHash: "t" + strconv.Itoa(i),
Title: "story " + strconv.Itoa(i), PublishedAt: 10000, FetchedAt: 10000,
})
if err != nil {
t.Fatal(err)
}
rows = append(rows, store.ClusterRow{Key: strconv.Itoa(i), Members: []int64{id}, FirstSeen: 10000, LastSeen: 10000})
}
if err := st.ReplaceClusters(rows); err != nil {
t.Fatal(err)
}
cfg := config.Default()
cfg.Watch.NotifyMinScore = 0
cfg.Watch.NotifyMaxItems = 2
items, err := buildNotable(st, cfg, time.Unix(9999, 0))
if err != nil {
t.Fatal(err)
}
if len(items) != 2 {
t.Fatalf("buildNotable returned %d items, want the notify_max_items cap of 2", len(items))
}
}

56
PROJECTS/intermediate/security-news-scraper/go.mod Normal file
View File

@ -0,0 +1,56 @@
module github.com/CarterPerez-dev/nadezhda
go 1.25.0
require (
github.com/PuerkitoBio/goquery v1.12.0
github.com/charmbracelet/bubbles v1.0.0
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/mmcdole/gofeed v1.3.0
github.com/spf13/cobra v1.10.2
github.com/temoto/robotstxt v1.1.2
golang.org/x/sync v0.21.0
golang.org/x/term v0.41.0
golang.org/x/time v0.15.0
gopkg.in/yaml.v3 v3.0.1
modernc.org/sqlite v1.53.0
)
require (
github.com/andybalholm/cascadia v1.3.3 // indirect
github.com/aymanbagabas/go-osc52/v2 v2.0.1 // indirect
github.com/charmbracelet/colorprofile v0.4.1 // indirect
github.com/charmbracelet/x/ansi v0.11.6 // indirect
github.com/charmbracelet/x/cellbuf v0.0.15 // indirect
github.com/charmbracelet/x/term v0.2.2 // indirect
github.com/clipperhouse/displaywidth v0.9.0 // indirect
github.com/clipperhouse/stringish v0.1.1 // indirect
github.com/clipperhouse/uax29/v2 v2.5.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/mattn/go-localereader v0.0.1 // indirect
github.com/mattn/go-runewidth v0.0.19 // indirect
github.com/mmcdole/goxpp v1.1.1-0.20240225020742-a0c311522b23 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.2 // indirect
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 // indirect
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/muesli/termenv v0.16.0 // indirect
github.com/ncruces/go-strftime v1.0.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/spf13/pflag v1.0.9 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
golang.org/x/net v0.52.0 // indirect
golang.org/x/sys v0.44.0 // indirect
golang.org/x/text v0.35.0 // indirect
modernc.org/libc v1.73.4 // indirect
modernc.org/mathutil v1.7.1 // indirect
modernc.org/memory v1.11.0 // indirect
)

207
PROJECTS/intermediate/security-news-scraper/go.sum Normal file
View File

@ -0,0 +1,207 @@
github.com/PuerkitoBio/goquery v1.12.0 h1:pAcL4g3WRXekcB9AU/y1mbKez2dbY2AajVhtkO8RIBo=
github.com/PuerkitoBio/goquery v1.12.0/go.mod h1:802ej+gV2y7bbIhOIoPY5sT183ZW0YFofScC4q/hIpQ=
github.com/andybalholm/cascadia v1.3.3 h1:AG2YHrzJIm4BZ19iwJ/DAua6Btl3IwJX+VI4kktS1LM=
github.com/andybalholm/cascadia v1.3.3/go.mod h1:xNd9bqTn98Ln4DwST8/nG+H0yuB8Hmgu1YHNnWw0GeA=
github.com/aymanbagabas/go-osc52/v2 v2.0.1 h1:HwpRHbFMcZLEVr42D4p7XBqjyuxQH5SMiErDT4WkJ2k=
github.com/aymanbagabas/go-osc52/v2 v2.0.1/go.mod h1:uYgXzlJ7ZpABp8OJ+exZzJJhRNQ2ASbcXHWsFqH8hp8=
github.com/charmbracelet/bubbles v1.0.0 h1:12J8/ak/uCZEMQ6KU7pcfwceyjLlWsDLAxB5fXonfvc=
github.com/charmbracelet/bubbles v1.0.0/go.mod h1:9d/Zd5GdnauMI5ivUIVisuEm3ave1XwXtD1ckyV6r3E=
github.com/charmbracelet/bubbletea v1.3.10 h1:otUDHWMMzQSB0Pkc87rm691KZ3SWa4KUlvF9nRvCICw=
github.com/charmbracelet/bubbletea v1.3.10/go.mod h1:ORQfo0fk8U+po9VaNvnV95UPWA1BitP1E0N6xJPlHr4=
github.com/charmbracelet/colorprofile v0.4.1 h1:a1lO03qTrSIRaK8c3JRxJDZOvhvIeSco3ej+ngLk1kk=
github.com/charmbracelet/colorprofile v0.4.1/go.mod h1:U1d9Dljmdf9DLegaJ0nGZNJvoXAhayhmidOdcBwAvKk=
github.com/charmbracelet/lipgloss v1.1.0 h1:vYXsiLHVkK7fp74RkV7b2kq9+zDLoEU4MZoFqR/noCY=
github.com/charmbracelet/lipgloss v1.1.0/go.mod h1:/6Q8FR2o+kj8rz4Dq0zQc3vYf7X+B0binUUBwA0aL30=
github.com/charmbracelet/x/ansi v0.11.6 h1:GhV21SiDz/45W9AnV2R61xZMRri5NlLnl6CVF7ihZW8=
github.com/charmbracelet/x/ansi v0.11.6/go.mod h1:2JNYLgQUsyqaiLovhU2Rv/pb8r6ydXKS3NIttu3VGZQ=
github.com/charmbracelet/x/cellbuf v0.0.15 h1:ur3pZy0o6z/R7EylET877CBxaiE1Sp1GMxoFPAIztPI=
github.com/charmbracelet/x/cellbuf v0.0.15/go.mod h1:J1YVbR7MUuEGIFPCaaZ96KDl5NoS0DAWkskup+mOY+Q=
github.com/charmbracelet/x/term v0.2.2 h1:xVRT/S2ZcKdhhOuSP4t5cLi5o+JxklsoEObBSgfgZRk=
github.com/charmbracelet/x/term v0.2.2/go.mod h1:kF8CY5RddLWrsgVwpw4kAa6TESp6EB5y3uxGLeCqzAI=
github.com/clipperhouse/displaywidth v0.9.0 h1:Qb4KOhYwRiN3viMv1v/3cTBlz3AcAZX3+y9OLhMtAtA=
github.com/clipperhouse/displaywidth v0.9.0/go.mod h1:aCAAqTlh4GIVkhQnJpbL0T/WfcrJXHcj8C0yjYcjOZA=
github.com/clipperhouse/stringish v0.1.1 h1:+NSqMOr3GR6k1FdRhhnXrLfztGzuG+VuFDfatpWHKCs=
github.com/clipperhouse/stringish v0.1.1/go.mod h1:v/WhFtE1q0ovMta2+m+UbpZ+2/HEXNWYXQgCt4hdOzA=
github.com/clipperhouse/uax29/v2 v2.5.0 h1:x7T0T4eTHDONxFJsL94uKNKPHrclyFI0lm7+w94cO8U=
github.com/clipperhouse/uax29/v2 v2.5.0/go.mod h1:Wn1g7MK6OoeDT0vL+Q0SQLDz/KpfsVRgg6W7ihQeh4g=
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f h1:Y/CXytFA4m6baUTXGLOoWe4PQhGxaX0KpnayAqC48p4=
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f/go.mod h1:vw97MGsxSvLiUE2X8qFplwetxpGLQrlU1Q9AUEIzCaM=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e h1:ijClszYn+mADRFY17kjQEVQ1XRhq2/JR1M3sGqeJoxs=
github.com/google/pprof v0.0.0-20250317173921-a4b03ec1a45e/go.mod h1:boTsfXsheKC2y+lKOCMpSfarhxDeIzfZG1jqGcPl3cA=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/hashicorp/golang-lru/v2 v2.0.7 h1:a+bsQ5rvGLjzHuww6tVxozPZFVghXaHOwFs4luLUK2k=
github.com/hashicorp/golang-lru/v2 v2.0.7/go.mod h1:QeFd9opnmA6QUJc5vARoKUSoFhyfM2/ZepoAG6RGpeM=
github.com/inconshreveable/mousetrap v1.1.0 h1:wN+x4NVGpMsO7ErUn/mUI3vEoE6Jt13X2s0bqwp9tc8=
github.com/inconshreveable/mousetrap v1.1.0/go.mod h1:vpF70FUmC8bwa3OWnCshd2FqLfsEA9PFc4w1p2J65bw=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/lucasb-eyer/go-colorful v1.3.0 h1:2/yBRLdWBZKrf7gB40FoiKfAWYQ0lqNcbuQwVHXptag=
github.com/lucasb-eyer/go-colorful v1.3.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
github.com/mattn/go-localereader v0.0.1 h1:ygSAOl7ZXTx4RdPYinUpg6W99U8jWvWi9Ye2JC/oIi4=
github.com/mattn/go-localereader v0.0.1/go.mod h1:8fBrzywKY7BI3czFoHkuzRoWE9C+EiG4R1k4Cjx5p88=
github.com/mattn/go-runewidth v0.0.19 h1:v++JhqYnZuu5jSKrk9RbgF5v4CGUjqRfBm05byFGLdw=
github.com/mattn/go-runewidth v0.0.19/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
github.com/mmcdole/gofeed v1.3.0 h1:5yn+HeqlcvjMeAI4gu6T+crm7d0anY85+M+v6fIFNG4=
github.com/mmcdole/gofeed v1.3.0/go.mod h1:9TGv2LcJhdXePDzxiuMnukhV2/zb6VtnZt1mS+SjkLE=
github.com/mmcdole/goxpp v1.1.1-0.20240225020742-a0c311522b23 h1:Zr92CAlFhy2gL+V1F+EyIuzbQNbSgP4xhTODZtrXUtk=
github.com/mmcdole/goxpp v1.1.1-0.20240225020742-a0c311522b23/go.mod h1:v+25+lT2ViuQ7mVxcncQ8ch1URund48oH+jhjiwEgS8=
github.com/modern-go/concurrent v0.0.0-20180228061459-e0a39a4cb421/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9Gz0M=
github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6 h1:ZK8zHtRHOkbHy6Mmr5D264iyp3TiX5OmNcI5cIARiQI=
github.com/muesli/ansi v0.0.0-20230316100256-276c6243b2f6/go.mod h1:CJlz5H+gyd6CUWT45Oy4q24RdLyn7Md9Vj2/ldJBSIo=
github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELUXHmA=
github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
github.com/ncruces/go-strftime v1.0.0 h1:HMFp8mLCTPp341M/ZnA4qaf7ZlsbTc+miZjCLOFAw7w=
github.com/ncruces/go-strftime v1.0.0/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
github.com/rivo/uniseg v0.4.7 h1:WUdvkW8uEhrYfLC4ZzdpI2ztxP1I582+49Oc5Mq64VQ=
github.com/rivo/uniseg v0.4.7/go.mod h1:FN3SvrM+Zdj16jyLfmOkMNblXMcoc8DfTHruCPUcx88=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU=
github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4=
github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY=
github.com/spf13/pflag v1.0.9/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
github.com/temoto/robotstxt v1.1.2 h1:W2pOjSJ6SWvldyEuiFXNxz3xZ8aiWX5LbfDiOFd7Fxg=
github.com/temoto/robotstxt v1.1.2/go.mod h1:+1AmkuG3IYkh1kv0d2qEB9Le88ehNO0zwOr3ujewlOo=
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.13.0/go.mod h1:y6Z2r+Rw4iayiXXAIxJIDAJ1zMW4yaTpebo8fPOliYc=
golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
golang.org/x/crypto v0.23.0/go.mod h1:CKFgDieR+mRhux2Lsu27y0fO304Db0wZe70UKqHu0v8=
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/exp v0.0.0-20231006140011-7918f672742d h1:jtJma62tbqLibJ5sFQz8bKtEM8rJBtfilJ2qTU199MI=
golang.org/x/exp v0.0.0-20231006140011-7918f672742d/go.mod h1:ldy0pHrwJyGW56pPQzzkH36rKxoZW1tw7ZJpeKx+hdo=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.12.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.15.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
golang.org/x/mod v0.36.0 h1:JJjpVx6myfUsUdAzZuOSTTmRE0PfZeNWzzvKrP7amb4=
golang.org/x/mod v0.36.0/go.mod h1:moc6ELqsWcOw5Ef3xVprK5ul/MvtVvkIXLziUOICjUQ=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.10.0/go.mod h1:0qNGK6F8kojg2nk9dLZ2mShWaEBan6FAoqfSigmmuDg=
golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.52.0 h1:He/TN1l0e4mmR3QqHMT2Xab3Aj3L9qjbhRm78/6jrW0=
golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.3.0/go.mod h1:FU7BRWz2tNW+3quACPkgCx/L+uEAv1htQ0V83Z9Rj+Y=
golang.org/x/sync v0.6.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sync v0.21.0 h1:HLII4xRRTtCRkxYp4HNFF0Js/Og6q2i++KXbg0gHCwM=
golang.org/x/sync v0.21.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.12.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.20.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.44.0 h1:ildZl3J4uzeKP07r2F++Op7E9B29JRUy+a27EibtBTQ=
golang.org/x/sys v0.44.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw=
golang.org/x/telemetry v0.0.0-20240228155512-f48c80bd79b2/go.mod h1:TeRTkGYfJXctD9OcfyVLyj2J3IxLnKwHJR8f4D8a3YE=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
golang.org/x/term v0.12.0/go.mod h1:owVbMEjm3cBLCHdkQu9b1opXd4ETQWc3BhuQGKgXgvU=
golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
golang.org/x/term v0.20.0/go.mod h1:8UkIAJTvZgivsXaD6/pH6U9ecQzZ45awqEOzuCvwpFY=
golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
golang.org/x/term v0.41.0 h1:QCgPso/Q3RTJx2Th4bDLqML4W6iJiaXFq2/ftQF13YU=
golang.org/x/term v0.41.0/go.mod h1:3pfBgksrReYfZ5lvYM0kSO0LIkAl4Yl2bXOkKP7Ec2A=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
golang.org/x/text v0.13.0/go.mod h1:TvPlkZtksWOMsz7fbANvkp4WM8x/WCo/om8BMLbz+aE=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.15.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/text v0.35.0 h1:JOVx6vVDFokkpaq1AEptVzLTpDe9KGpj5tR4/X+ybL8=
golang.org/x/text v0.35.0/go.mod h1:khi/HExzZJ2pGnjenulevKNX1W67CUy0AsXcNubPGCA=
golang.org/x/time v0.15.0 h1:bbrp8t3bGUeFOx08pvsMYRTCVSMk89u4tKbNOZbp88U=
golang.org/x/time v0.15.0/go.mod h1:Y4YMaQmXwGQZoFaVFk4YpCt4FLQMYKZe9oeV/f4MSno=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.13.0/go.mod h1:HvlwmtVNQAhOuCjW7xxvovg8wbNq7LwfXh/k7wXUl58=
golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
golang.org/x/tools v0.45.0 h1:18qN3FAooORvApf5XjCXgsuayZOEtXf6JK18I3+ONa8=
golang.org/x/tools v0.45.0/go.mod h1:LuUGqqaXcXMEFEruIVJVm5mgDD8vww/z/SR1gQ4uE/0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
modernc.org/cc/v4 v4.28.4 h1:Hd/4Es+MBj+/7hSdZaisNyu6bv3V0Dp2MdllyfqaH+c=
modernc.org/cc/v4 v4.28.4/go.mod h1:OnovgIhbbMXMu1aISnJ0wvVD1KnW+cAUJkIrAWh+kVI=
modernc.org/ccgo/v4 v4.34.4 h1:OVnSOWQjVKOYkFxoHYB+qQmSHK5gqMqARM+K9DpR/Ws=
modernc.org/ccgo/v4 v4.34.4/go.mod h1:qdKqE8FNIYyysougB1RX9MxCzp5oJOcQXSobANJ4TuE=
modernc.org/fileutil v1.4.0 h1:j6ZzNTftVS054gi281TyLjHPp6CPHr2KCxEXjEbD6SM=
modernc.org/fileutil v1.4.0/go.mod h1:EqdKFDxiByqxLk8ozOxObDSfcVOv/54xDs/DUHdvCUU=
modernc.org/gc/v2 v2.6.5 h1:nyqdV8q46KvTpZlsw66kWqwXRHdjIlJOhG6kxiV/9xI=
modernc.org/gc/v2 v2.6.5/go.mod h1:YgIahr1ypgfe7chRuJi2gD7DBQiKSLMPgBQe9oIiito=
modernc.org/gc/v3 v3.1.3 h1:6QAplYyVO+KdPW3pGnqmJDUxtkec8ooEWvks/hhU3lc=
modernc.org/gc/v3 v3.1.3/go.mod h1:HFK/6AGESC7Ex+EZJhJ2Gni6cTaYpSMmU/cT9RmlfYY=
modernc.org/goabi0 v0.2.0 h1:HvEowk7LxcPd0eq6mVOAEMai46V+i7Jrj13t4AzuNks=
modernc.org/goabi0 v0.2.0/go.mod h1:CEFRnnJhKvWT1c1JTI3Avm+tgOWbkOu5oPA8eH8LnMI=
modernc.org/libc v1.73.4 h1:+ra4Ui8ngyt8HDcO1FTDPWlkAh6yOdaO2yAoh8MddQA=
modernc.org/libc v1.73.4/go.mod h1:DXZ3eO8qMCNn2SnmTNCiC71nJ9Rcq3PsnpU6Vc4rWK8=
modernc.org/mathutil v1.7.1 h1:GCZVGXdaN8gTqB1Mf/usp1Y/hSqgI2vAGGP4jZMCxOU=
modernc.org/mathutil v1.7.1/go.mod h1:4p5IwJITfppl0G4sUEDtCr4DthTaT47/N3aT6MhfgJg=
modernc.org/memory v1.11.0 h1:o4QC8aMQzmcwCK3t3Ux/ZHmwFPzE6hf2Y5LbkRs+hbI=
modernc.org/memory v1.11.0/go.mod h1:/JP4VbVC+K5sU2wZi9bHoq2MAkCnrt2r98UGeSK7Mjw=
modernc.org/opt v0.2.0 h1:tGyef5ApycA7FSEOMraay9SaTk5zmbx7Tu+cJs4QKZg=
modernc.org/opt v0.2.0/go.mod h1:03fq9lsNfvkYSfxrfUhZCWPk1lm4cq4N+Bh//bEtgns=
modernc.org/sortutil v1.2.1 h1:+xyoGf15mM3NMlPDnFqrteY07klSFxLElE2PVuWIJ7w=
modernc.org/sortutil v1.2.1/go.mod h1:7ZI3a3REbai7gzCLcotuw9AC4VZVpYMjDzETGsSMqJE=
modernc.org/sqlite v1.53.0 h1:20WG8N9q4ji/dEqGk4uiI0c6OPjSeLTNYGFCc3+7c1M=
modernc.org/sqlite v1.53.0/go.mod h1:xoEpOIpGrgT48H5iiyt/YXPCZPEzlfmfFwtk8Lklw8s=
modernc.org/strutil v1.2.1 h1:UneZBkQA+DX2Rp35KcM69cSsNES9ly8mQWD71HKlOA0=
modernc.org/strutil v1.2.1/go.mod h1:EHkiggD70koQxjVdSBM3JKM7k6L0FbGE5eymy9i3B9A=
modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=

259
PROJECTS/intermediate/security-news-scraper/install.sh Executable file
View File

@ -0,0 +1,259 @@
#!/usr/bin/env bash
# ©AngelaMos | 2026
# install.sh
#
# One-shot installer for nadezhda. Takes a fresh machine to `nadezhda` runnable
# by its bare name, with zero further steps, whether run from a clone or piped
# from a domain via curl. Prefers a prebuilt release binary (no Go needed);
# falls back to building from source (auto-installs the Go toolchain if absent).
set -euo pipefail
# ============================================================================
# CONFIG
# ============================================================================
REPO_OWNER="CarterPerez-dev"
REPO_NAME="nadezhda"
BINARY="nadezhda"
TAGLINE="Security news + CVE intelligence in your terminal. Keyless by default."
REPO_URL="https://github.com/${REPO_OWNER}/${REPO_NAME}.git"
INSTALL_DIR="${NADEZHDA_INSTALL_DIR:-$HOME/.local/bin}"
DEFAULT_BRANCH="main"
GO_MIN="1.21"
PREBUILT=1
# ============================================================================
# Colors
# ============================================================================
if [ -t 2 ] && [ -z "${NO_COLOR:-}" ]; then
BOLD=$'\033[1m'; DIM=$'\033[2m'; RED=$'\033[31m'; GREEN=$'\033[32m'
YELLOW=$'\033[33m'; CYAN=$'\033[36m'; RESET=$'\033[0m'
else
BOLD=""; DIM=""; RED=""; GREEN=""; YELLOW=""; CYAN=""; RESET=""
fi
info() { printf '%s\n' " ${CYAN}+${RESET} $*" >&2; }
ok() { printf '%s\n' " ${GREEN}+${RESET} $*" >&2; }
warn() { printf '%s\n' " ${YELLOW}!${RESET} $*" >&2; }
die() { printf '%s\n' " ${RED}x $*${RESET}" >&2; exit 1; }
header(){ printf '\n%s\n\n' "${BOLD}${CYAN}--- $* ---${RESET}" >&2; }
have() { command -v "$1" >/dev/null 2>&1; }
trap 'printf "%s\n" "${RED}x install failed${RESET}" >&2' ERR
TMP_DIR=""
cleanup() { [ -n "$TMP_DIR" ] && rm -rf "$TMP_DIR"; return 0; }
trap cleanup EXIT
banner() {
printf '%s' "${CYAN}${BOLD}" >&2
cat >&2 <<'ART'
╭───────────────────────╮
│ n a d e z h d a │
╰───────────────────────╯
ART
printf '%s\n' "${RESET}" >&2
printf '%s\n' " ${DIM}${TAGLINE}${RESET}" >&2
}
# ============================================================================
# Privilege + package-manager fan
# ============================================================================
SUDO=""
if [ "$(id -u)" -ne 0 ]; then
if have sudo; then SUDO="sudo"; fi
fi
pkg_install() {
if have apt-get; then $SUDO apt-get update -y || warn "apt update had errors; continuing"
$SUDO apt-get install -y --no-install-recommends "$@"
elif have dnf; then $SUDO dnf install -y "$@"
elif have pacman; then $SUDO pacman -S --needed --noconfirm "$@"
elif have zypper; then $SUDO zypper install -y "$@"
elif have apk; then $SUDO apk add "$@"
elif have brew; then brew install "$@"
else die "no known package manager. Install manually: $*"; fi
}
download() {
if have curl; then curl -fsSL "$1" -o "$2" || return 1
elif have wget; then wget -qO "$2" "$1" || return 1
else die "need curl or wget"; fi
}
# ============================================================================
# Args
# ============================================================================
usage() {
cat >&2 <<USAGE
install.sh — install ${BINARY}
./install.sh [options]
curl -fsSL https://angelamos.com/${BINARY}/install.sh | bash
options:
--prefix DIR install dir (default: ${INSTALL_DIR})
-h, --help this help
USAGE
}
while [ $# -gt 0 ]; do
case "$1" in
--prefix) INSTALL_DIR="$2"; shift 2 ;;
--prefix=*) INSTALL_DIR="${1#*=}"; shift ;;
-h|--help) usage; exit 0 ;;
*) die "unknown option: $1 (try --help)" ;;
esac
done
# ============================================================================
# OS / arch
# ============================================================================
OS="$(uname -s)"; ARCH="$(uname -m)"
case "$OS" in
Linux) OS="linux" ;; Darwin) OS="darwin" ;;
MINGW*|MSYS*|CYGWIN*) die "Windows unsupported. Use WSL, or: go install github.com/${REPO_OWNER}/${REPO_NAME}/cmd/${BINARY}@latest" ;;
*) die "unsupported OS: $OS" ;;
esac
case "$ARCH" in
x86_64|amd64) ARCH="amd64" ;; aarch64|arm64) ARCH="arm64" ;;
*) die "unsupported arch: $ARCH" ;;
esac
# ============================================================================
# Bootstrap
# ============================================================================
resolve_repo() {
if [ -f "./go.mod" ]; then pwd; return; fi
have git || { warn "git missing — installing it"; pkg_install git; }
have git || die "could not install git; install it then re-run"
local cache="${XDG_CACHE_HOME:-$HOME/.cache}/${BINARY}"
if [ -d "$cache/.git" ]; then
info "updating cached clone at $cache"
git -C "$cache" pull --ff-only --quiet 2>/dev/null || warn "pull failed; using existing clone"
else
info "cloning ${REPO_URL}"
git clone --depth 1 --branch "$DEFAULT_BRANCH" --quiet "$REPO_URL" "$cache" \
|| die "clone failed from ${REPO_URL}"
fi
printf '%s\n' "$cache"
}
# ============================================================================
# Toolchain (Go) + build from source
# ============================================================================
install_go() {
info "installing a current Go toolchain"
local latest tgz
latest="$(download "https://go.dev/VERSION?m=text" /dev/stdout 2>/dev/null | head -n1)" || latest=""
case "$latest" in go*) ;; *) latest="go1.25.5" ;; esac
tgz="${latest}.${OS}-${ARCH}.tar.gz"
TMP_DIR="${TMP_DIR:-$(mktemp -d)}"
download "https://go.dev/dl/${tgz}" "$TMP_DIR/go.tgz" || die "failed to download ${tgz} from go.dev/dl"
rm -rf "$HOME/.local/go"
mkdir -p "$HOME/.local"
tar -C "$HOME/.local" -xzf "$TMP_DIR/go.tgz" || die "failed to extract Go"
export PATH="$HOME/.local/go/bin:$PATH"
export GOTOOLCHAIN=auto
have go || die "Go toolchain install failed"
ok "go $(go env GOVERSION 2>/dev/null | sed 's/^go//') at ~/.local/go"
}
need_toolchain() {
local cur
if have go; then
cur="$(go env GOVERSION 2>/dev/null | sed 's/^go//')"
if [ -n "$cur" ] && [ "$(printf '%s\n%s\n' "$GO_MIN" "$cur" | sort -V | head -n1)" = "$GO_MIN" ]; then
export GOTOOLCHAIN=auto
ok "go $cur (auto-toolchain fetches the go.mod-pinned version if newer)"
return
fi
warn "go ${cur:-unknown} predates toolchain auto-download; installing a current Go"
fi
install_go
}
build_from_source() {
info "building ${BINARY} (compiles the pure-Go SQLite driver; give it a minute)"
mkdir -p "$INSTALL_DIR"
GOBIN="$INSTALL_DIR" go install ./cmd/nadezhda || die "go install failed"
ok "installed ${BINARY} -> ${INSTALL_DIR}/${BINARY}"
}
try_prebuilt() {
[ "$PREBUILT" = "1" ] || return 1
local ver archive url
ver="$(download "https://api.github.com/repos/${REPO_OWNER}/${REPO_NAME}/releases/latest" /dev/stdout 2>/dev/null \
| grep '"tag_name":' | sed -E 's/.*"([^"]+)".*/\1/')" || true
[ -n "$ver" ] || return 1
archive="${BINARY}_${ver#v}_${OS}_${ARCH}.tar.gz"
url="https://github.com/${REPO_OWNER}/${REPO_NAME}/releases/download/${ver}/${archive}"
TMP_DIR="$(mktemp -d)"
download "$url" "$TMP_DIR/a.tgz" || { warn "no prebuilt for ${OS}/${ARCH}; will build from source"; return 1; }
tar -xzf "$TMP_DIR/a.tgz" -C "$TMP_DIR" || return 1
mkdir -p "$INSTALL_DIR"; install -m 0755 "$TMP_DIR/$BINARY" "$INSTALL_DIR/$BINARY"
ok "installed prebuilt ${ver} -> ${INSTALL_DIR}/${BINARY}"
}
# ============================================================================
# PATH wiring
# ============================================================================
wire_path() {
case ":$PATH:" in *":$INSTALL_DIR:"*) ok "$INSTALL_DIR already on PATH"; return ;; esac
local shell rc=""
shell="$(basename "${SHELL:-bash}")"
case "$shell" in
zsh) rc="$HOME/.zshrc" ;;
fish) mkdir -p "$HOME/.config/fish/conf.d"
echo "fish_add_path $INSTALL_DIR" > "$HOME/.config/fish/conf.d/${BINARY}.fish"
ok "added to fish conf.d" ;;
bash) rc="$HOME/.bashrc"; [ -f "$rc" ] || rc="$HOME/.bash_profile" ;;
*) rc="$HOME/.profile" ;;
esac
if [ -n "$rc" ] && ! grep -q "$INSTALL_DIR" "$rc" 2>/dev/null; then
printf '\nexport PATH="%s:$PATH"\n' "$INSTALL_DIR" >> "$rc"
ok "added $INSTALL_DIR to PATH in $rc"
fi
export PATH="$INSTALL_DIR:$PATH"
}
# ============================================================================
# Main
# ============================================================================
main() {
banner
have "$BINARY" && info "existing install at $(command -v "$BINARY") — updating"
REPO=""
if ! try_prebuilt; then
header "Building from source"
REPO="$(resolve_repo)"; cd "$REPO"
need_toolchain
build_from_source
fi
wire_path
header "Verify"
if have "$BINARY"; then
ok "$BINARY -> $(command -v "$BINARY")"
"$BINARY" version 2>/dev/null || true
else
warn "installed to $INSTALL_DIR but not yet on PATH — open a new shell"
fi
printf '\n%s\n\n' " ${GREEN}${BOLD}${BINARY} is ready.${RESET}" >&2
if have just && [ -n "$REPO" ] && [ -f "${REPO}/justfile" ]; then
printf '%s\n' " ${DIM}dev commands:${RESET} just" >&2
fi
cat >&2 <<FOOTER
${DIM}quick start:${RESET}
${CYAN}${BINARY} scrape${RESET} pull the latest security news and enrich CVEs
${CYAN}${BINARY} tui${RESET} browse the ranked dossier
${CYAN}${BINARY} ai${RESET} optional: set up an AI provider for content ideation
${CYAN}${BINARY} watch${RESET} optional: run as a daemon with webhook alerts
${DIM}docs: https://github.com/${REPO_OWNER}/${REPO_NAME}${RESET}
FOOTER
return 0
}
main "$@" </dev/null

84
PROJECTS/intermediate/security-news-scraper/internal/ai/anthropic.go Normal file
View File

@ -0,0 +1,84 @@
// ©AngelaMos | 2026
// anthropic.go
package ai
import (
"context"
"net/http"
"strings"
)
const (
anthropicMessagesPath = "/messages"
anthropicVersion = "2023-06-01"
headerAPIKey = "x-api-key"
headerAnthropicVer = "anthropic-version"
stopReasonRefusal = "refusal"
contentTypeText = "text"
)
type anthropicClient struct {
http *http.Client
baseURL string
model string
apiKey string
}
func newAnthropic(client *http.Client, baseURL, model, apiKey string) *anthropicClient {
return &anthropicClient{
http: client,
baseURL: strings.TrimRight(baseURL, "/"),
model: model,
apiKey: apiKey,
}
}
func (c *anthropicClient) Name() string { return ProviderAnthropic }
type anthropicMessage struct {
Role string `json:"role"`
Content string `json:"content"`
}
type anthropicRequest struct {
Model string `json:"model"`
MaxTokens int `json:"max_tokens"`
System string `json:"system"`
Messages []anthropicMessage `json:"messages"`
}
type anthropicResponse struct {
StopReason string `json:"stop_reason"`
Content []struct {
Type string `json:"type"`
Text string `json:"text"`
} `json:"content"`
}
func (c *anthropicClient) Generate(ctx context.Context, req IdeationRequest) (IdeationResult, error) {
system, user := buildPrompt(req)
body := anthropicRequest{
Model: c.model,
MaxTokens: defaultMaxTokens,
System: system,
Messages: []anthropicMessage{{Role: roleUser, Content: user}},
}
header := http.Header{}
header.Set(headerAPIKey, c.apiKey)
header.Set(headerAnthropicVer, anthropicVersion)
var resp anthropicResponse
if err := postJSON(ctx, c.http, c.baseURL+anthropicMessagesPath, header, body, &resp); err != nil {
return IdeationResult{}, err
}
if resp.StopReason == stopReasonRefusal {
return IdeationResult{}, ErrRefused
}
var text strings.Builder
for _, block := range resp.Content {
if block.Type == contentTypeText {
text.WriteString(block.Text)
}
}
return parseResult(text.String())
}

125
PROJECTS/intermediate/security-news-scraper/internal/ai/client_test.go Normal file
View File

@ -0,0 +1,125 @@
// ©AngelaMos | 2026
// client_test.go
package ai
import (
"context"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"testing"
)
const ideationJSON = `{"summary":"s","why":"w","angles":["a","b","c"],"format":"video"}`
func TestOpenAICompatGenerate(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got := r.Header.Get("Authorization"); got != "Bearer k" {
t.Errorf("Authorization = %q, want Bearer k", got)
}
body, _ := json.Marshal(map[string]any{
"choices": []map[string]any{{"message": map[string]any{"role": "assistant", "content": ideationJSON}}},
})
_, _ = w.Write(body)
}))
defer srv.Close()
c := newOpenAICompat("openai", srv.Client(), srv.URL, "gpt-4o-mini", "k")
res, err := c.Generate(context.Background(), IdeationRequest{Titles: []string{"t"}})
if err != nil {
t.Fatalf("Generate: %v", err)
}
if res.Summary != "s" || res.Format != FormatVideo || len(res.Angles) != 3 {
t.Errorf("got %+v", res)
}
if c.Name() != "openai" {
t.Errorf("Name = %q", c.Name())
}
}
func TestOpenAICompatNoKeyNoAuth(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got := r.Header.Get("Authorization"); got != "" {
t.Errorf("Authorization = %q, want empty (qwen has no key)", got)
}
body, _ := json.Marshal(map[string]any{
"choices": []map[string]any{{"message": map[string]any{"content": ideationJSON}}},
})
_, _ = w.Write(body)
}))
defer srv.Close()
c := newOpenAICompat("qwen", srv.Client(), srv.URL, "qwen2.5:7b", "")
if _, err := c.Generate(context.Background(), IdeationRequest{Titles: []string{"t"}}); err != nil {
t.Fatalf("Generate: %v", err)
}
}
func TestOpenAICompatNoChoices(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"choices":[]}`))
}))
defer srv.Close()
c := newOpenAICompat("openai", srv.Client(), srv.URL, "m", "k")
if _, err := c.Generate(context.Background(), IdeationRequest{}); err == nil {
t.Error("expected error on empty choices")
}
}
func TestOpenAICompatStatusError(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusUnauthorized)
_, _ = w.Write([]byte(`{"error":"bad key"}`))
}))
defer srv.Close()
c := newOpenAICompat("openai", srv.Client(), srv.URL, "m", "k")
if _, err := c.Generate(context.Background(), IdeationRequest{}); err == nil {
t.Error("expected error on 401")
}
}
func TestAnthropicGenerate(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if got := r.Header.Get("x-api-key"); got != "k" {
t.Errorf("x-api-key = %q, want k", got)
}
if got := r.Header.Get("anthropic-version"); got != "2023-06-01" {
t.Errorf("anthropic-version = %q, want 2023-06-01", got)
}
body, _ := json.Marshal(map[string]any{
"stop_reason": "end_turn",
"content": []map[string]any{{"type": "text", "text": ideationJSON}},
})
_, _ = w.Write(body)
}))
defer srv.Close()
c := newAnthropic(srv.Client(), srv.URL, "claude-sonnet-4-6", "k")
res, err := c.Generate(context.Background(), IdeationRequest{Titles: []string{"t"}})
if err != nil {
t.Fatalf("Generate: %v", err)
}
if res.Summary != "s" || res.Format != FormatVideo {
t.Errorf("got %+v", res)
}
if c.Name() != ProviderAnthropic {
t.Errorf("Name = %q", c.Name())
}
}
func TestAnthropicRefusal(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"stop_reason":"refusal","content":[]}`))
}))
defer srv.Close()
c := newAnthropic(srv.Client(), srv.URL, "claude-sonnet-4-6", "k")
_, err := c.Generate(context.Background(), IdeationRequest{})
if !errors.Is(err, ErrRefused) {
t.Errorf("err = %v, want ErrRefused", err)
}
}

83
PROJECTS/intermediate/security-news-scraper/internal/ai/factory_test.go Normal file
View File

@ -0,0 +1,83 @@
// ©AngelaMos | 2026
// factory_test.go
package ai
import (
"testing"
"github.com/CarterPerez-dev/nadezhda/internal/config"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
func TestFactoryProviders(t *testing.T) {
base := config.Default().AI
t.Setenv(envOpenAIKey, "")
t.Setenv(envGeminiKey, "")
t.Setenv(envAnthropicKey, "")
base.Provider = ProviderQwen
p, err := Factory(base)
if err != nil || p.Name() != ProviderQwen {
t.Fatalf("qwen: p=%v err=%v", p, err)
}
base.Provider = ProviderOpenAI
if _, err := Factory(base); err == nil {
t.Error("openai without key should error")
}
t.Setenv(envOpenAIKey, "k")
if p, err := Factory(base); err != nil || p.Name() != ProviderOpenAI {
t.Errorf("openai: p=%v err=%v", p, err)
}
base.Provider = ProviderGemini
if _, err := Factory(base); err == nil {
t.Error("gemini without key should error")
}
t.Setenv(envGeminiKey, "k")
if p, err := Factory(base); err != nil || p.Name() != ProviderGemini {
t.Errorf("gemini: p=%v err=%v", p, err)
}
base.Provider = ProviderAnthropic
if _, err := Factory(base); err == nil {
t.Error("anthropic without key should error")
}
t.Setenv(envAnthropicKey, "k")
if p, err := Factory(base); err != nil || p.Name() != ProviderAnthropic {
t.Errorf("anthropic: p=%v err=%v", p, err)
}
base.Provider = "nope"
if _, err := Factory(base); err == nil {
t.Error("unknown provider should error")
}
}
func TestRequestFromCluster(t *testing.T) {
cvss := 9.8
epss := 0.5
c := store.DigestCluster{
Size: 3,
FirstSeen: 0,
LastSeen: 6 * secondsPerHour,
Articles: []store.DigestArticle{
{Title: "A", SourceName: "krebs"},
{Title: "A", SourceName: "krebs"},
{Title: "B", SourceName: "register"},
},
CVEs: []store.DigestCVE{{ID: "CVE-2025-1", CVSSScore: &cvss, EPSS: &epss, IsKEV: true}},
}
req := RequestFromCluster(c)
if len(req.Titles) != 2 || len(req.Sources) != 2 {
t.Errorf("dedup failed: titles=%v sources=%v", req.Titles, req.Sources)
}
if req.SpanHours != 6 || req.ClusterSize != 3 {
t.Errorf("span=%d size=%d", req.SpanHours, req.ClusterSize)
}
if len(req.CVEs) != 1 || !req.CVEs[0].KEV || req.CVEs[0].CVSS == nil {
t.Errorf("cve mapping: %+v", req.CVEs)
}
}

65
PROJECTS/intermediate/security-news-scraper/internal/ai/http.go Normal file
View File

@ -0,0 +1,65 @@
// ©AngelaMos | 2026
// http.go
package ai
import (
"bytes"
"context"
"encoding/json"
"fmt"
"io"
"net/http"
"strings"
)
const (
headerContentType = "Content-Type"
mimeJSON = "application/json"
errBodyLimit = 300
)
type statusError struct {
code int
url string
body string
}
func (e *statusError) Error() string {
body := e.body
if len(body) > errBodyLimit {
body = strings.ToValidUTF8(body[:errBodyLimit], "") + "..."
}
return fmt.Sprintf("ai: POST %s: status %d: %s", e.url, e.code, body)
}
func postJSON(ctx context.Context, client *http.Client, url string, header http.Header, reqBody, out any) error {
raw, err := json.Marshal(reqBody)
if err != nil {
return fmt.Errorf("ai: marshal request: %w", err)
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(raw))
if err != nil {
return fmt.Errorf("ai: build request %s: %w", url, err)
}
req.Header.Set(headerContentType, mimeJSON)
for key, values := range header {
req.Header[key] = values
}
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("ai: POST %s: %w", url, err)
}
defer resp.Body.Close()
data, err := io.ReadAll(io.LimitReader(resp.Body, maxJSONBytes))
if err != nil {
return fmt.Errorf("ai: read %s: %w", url, err)
}
if resp.StatusCode != http.StatusOK {
return &statusError{code: resp.StatusCode, url: url, body: string(data)}
}
if err := json.Unmarshal(data, out); err != nil {
return fmt.Errorf("ai: decode %s: %w", url, err)
}
return nil
}

25
PROJECTS/intermediate/security-news-scraper/internal/ai/mock.go Normal file
View File

@ -0,0 +1,25 @@
// ©AngelaMos | 2026
// mock.go
package ai
import "context"
type MockProvider struct {
ProviderName string
Result IdeationResult
Err error
Calls int
}
func (m *MockProvider) Name() string {
if m.ProviderName == "" {
return "mock"
}
return m.ProviderName
}
func (m *MockProvider) Generate(ctx context.Context, req IdeationRequest) (IdeationResult, error) {
m.Calls++
return m.Result, m.Err
}

36
PROJECTS/intermediate/security-news-scraper/internal/ai/mock_test.go Normal file
View File

@ -0,0 +1,36 @@
// ©AngelaMos | 2026
// mock_test.go
package ai
import (
"context"
"testing"
)
func TestMockProvider(t *testing.T) {
want := IdeationResult{Summary: "s", Angles: []string{"a"}, Format: FormatBlog}
m := &MockProvider{ProviderName: "mock", Result: want}
if m.Name() != "mock" {
t.Errorf("Name = %q, want mock", m.Name())
}
got, err := m.Generate(context.Background(), IdeationRequest{})
if err != nil {
t.Fatalf("Generate: %v", err)
}
if got.Summary != "s" || len(got.Angles) != 1 {
t.Errorf("result = %+v", got)
}
if m.Calls != 1 {
t.Errorf("Calls = %d, want 1", m.Calls)
}
sentinel := context.Canceled
me := &MockProvider{Err: sentinel}
if me.Name() != "mock" {
t.Errorf("default Name = %q, want mock", me.Name())
}
if _, err := me.Generate(context.Background(), IdeationRequest{}); err != sentinel {
t.Errorf("err = %v, want the injected sentinel", err)
}
}

78
PROJECTS/intermediate/security-news-scraper/internal/ai/openai.go Normal file
View File

@ -0,0 +1,78 @@
// ©AngelaMos | 2026
// openai.go
package ai
import (
"context"
"fmt"
"net/http"
"strings"
)
const (
openAIChatPath = "/chat/completions"
roleSystem = "system"
roleUser = "user"
headerAuth = "Authorization"
bearerPrefix = "Bearer "
)
type openAICompat struct {
name string
http *http.Client
baseURL string
model string
apiKey string
}
func newOpenAICompat(name string, client *http.Client, baseURL, model, apiKey string) *openAICompat {
return &openAICompat{
name: name,
http: client,
baseURL: strings.TrimRight(baseURL, "/"),
model: model,
apiKey: apiKey,
}
}
func (c *openAICompat) Name() string { return c.name }
type openAIMessage struct {
Role string `json:"role"`
Content string `json:"content"`
}
type openAIRequest struct {
Model string `json:"model"`
Messages []openAIMessage `json:"messages"`
}
type openAIResponse struct {
Choices []struct {
Message openAIMessage `json:"message"`
} `json:"choices"`
}
func (c *openAICompat) Generate(ctx context.Context, req IdeationRequest) (IdeationResult, error) {
system, user := buildPrompt(req)
body := openAIRequest{
Model: c.model,
Messages: []openAIMessage{
{Role: roleSystem, Content: system},
{Role: roleUser, Content: user},
},
}
header := http.Header{}
if c.apiKey != "" {
header.Set(headerAuth, bearerPrefix+c.apiKey)
}
var resp openAIResponse
if err := postJSON(ctx, c.http, c.baseURL+openAIChatPath, header, body, &resp); err != nil {
return IdeationResult{}, err
}
if len(resp.Choices) == 0 {
return IdeationResult{}, fmt.Errorf("ai: %s returned no choices", c.name)
}
return parseResult(resp.Choices[0].Message.Content)
}

128
PROJECTS/intermediate/security-news-scraper/internal/ai/prompt.go Normal file
View File

@ -0,0 +1,128 @@
// ©AngelaMos | 2026
// prompt.go
package ai
import (
"encoding/json"
"fmt"
"strings"
)
const (
systemPrompt = `You are a senior security-content strategist helping a creator turn cybersecurity news into content. You are given a cluster of related news articles (one story covered by one or more outlets), optionally with referenced CVEs and their exploit signals. Produce content ideation for the story.
Respond with a SINGLE JSON object and NOTHING else. No prose, no markdown, no code fences. The object has exactly these keys:
"summary": a 2-3 sentence plain-language summary of the story.
"why": one paragraph on why it matters to a security audience (impact, who is affected, exploitation).
"angles": an array of 3 to 5 distinct content angles or hooks a creator could lead with.
"format": the single best-fit format, one of: "blog", "newsletter", "video".
Ground every claim in the provided material. Do not invent CVEs, vendors, or facts not present in the input.`
jsonObjectOpen byte = '{'
jsonObjectClose byte = '}'
)
func buildPrompt(req IdeationRequest) (string, string) {
var b strings.Builder
fmt.Fprintf(&b, "Story cluster: %d article(s) across %d outlet(s), spanning ~%dh.\n\n", req.ClusterSize, len(req.Sources), req.SpanHours)
if len(req.Sources) > 0 {
fmt.Fprintf(&b, "Outlets: %s\n\n", strings.Join(req.Sources, ", "))
}
b.WriteString("Headlines:\n")
for _, t := range req.Titles {
fmt.Fprintf(&b, "- %s\n", t)
}
if len(req.CVEs) > 0 {
b.WriteString("\nReferenced CVEs:\n")
for _, c := range req.CVEs {
b.WriteString("- " + c.ID)
if c.CVSS != nil {
fmt.Fprintf(&b, " CVSS %.1f", *c.CVSS)
}
if c.KEV {
b.WriteString(" [KEV: known exploited]")
}
if c.EPSS != nil {
fmt.Fprintf(&b, " EPSS %.2f", *c.EPSS)
}
b.WriteString("\n")
}
}
return systemPrompt, b.String()
}
func parseResult(text string) (IdeationResult, error) {
for _, obj := range jsonObjectCandidates(text) {
var res IdeationResult
if err := json.Unmarshal([]byte(obj), &res); err != nil {
continue
}
res.Format = normalizeFormat(res.Format)
if strings.TrimSpace(res.Summary) != "" && len(res.Angles) > 0 {
return res, nil
}
}
return IdeationResult{}, fmt.Errorf("ai: no usable JSON ideation object in model output")
}
func jsonObjectCandidates(text string) []string {
var out []string
for i := 0; i < len(text); {
if text[i] != jsonObjectOpen {
i++
continue
}
end := balancedObjectEnd(text, i)
if end < 0 {
break
}
out = append(out, text[i:end+1])
i = end + 1
}
return out
}
func balancedObjectEnd(text string, start int) int {
depth := 0
inStr := false
esc := false
for i := start; i < len(text); i++ {
ch := text[i]
if inStr {
switch {
case esc:
esc = false
case ch == '\\':
esc = true
case ch == '"':
inStr = false
}
continue
}
switch ch {
case '"':
inStr = true
case jsonObjectOpen:
depth++
case jsonObjectClose:
depth--
if depth == 0 {
return i
}
}
}
return -1
}
func normalizeFormat(f string) string {
switch strings.ToLower(strings.TrimSpace(f)) {
case FormatNewsletter:
return FormatNewsletter
case FormatVideo:
return FormatVideo
default:
return FormatBlog
}
}

87
PROJECTS/intermediate/security-news-scraper/internal/ai/prompt_test.go Normal file
View File

@ -0,0 +1,87 @@
// ©AngelaMos | 2026
// prompt_test.go
package ai
import (
"strings"
"testing"
)
func TestBuildPromptIncludesContext(t *testing.T) {
cvss := 9.8
epss := 0.97
_, user := buildPrompt(IdeationRequest{
Titles: []string{"Massive breach at ACME"},
Sources: []string{"Krebs", "The Register"},
CVEs: []CVEContext{{ID: "CVE-2025-5777", CVSS: &cvss, KEV: true, EPSS: &epss}},
ClusterSize: 2,
SpanHours: 6,
})
for _, want := range []string{"Massive breach at ACME", "Krebs", "The Register", "CVE-2025-5777", "CVSS 9.8", "KEV", "EPSS 0.97"} {
if !strings.Contains(user, want) {
t.Errorf("user prompt missing %q\n---\n%s", want, user)
}
}
}
func TestParseResultValid(t *testing.T) {
res, err := parseResult(`{"summary":"s","why":"w","angles":["a","b","c"],"format":"video"}`)
if err != nil {
t.Fatalf("parseResult: %v", err)
}
if res.Summary != "s" || res.Why != "w" || len(res.Angles) != 3 || res.Format != FormatVideo {
t.Errorf("got %+v", res)
}
}
func TestParseResultTolerantWrapper(t *testing.T) {
raw := "Here is the JSON:\n```json\n{\"summary\":\"s\",\"why\":\"w\",\"angles\":[\"a\"],\"format\":\"blog\"}\n```\nHope that helps."
res, err := parseResult(raw)
if err != nil {
t.Fatalf("parseResult: %v", err)
}
if res.Summary != "s" || len(res.Angles) != 1 {
t.Errorf("got %+v", res)
}
}
func TestParseResultIgnoresPreJSONBraces(t *testing.T) {
raw := `Let me plan {step: outline} then answer: {"summary":"real","why":"w","angles":["a","b"],"format":"blog"} done.`
res, err := parseResult(raw)
if err != nil {
t.Fatalf("parseResult: %v", err)
}
if res.Summary != "real" || len(res.Angles) != 2 {
t.Errorf("got %+v", res)
}
}
func TestParseResultErrors(t *testing.T) {
cases := []string{
"no json here",
`{"why":"w","angles":["a"],"format":"blog"}`,
`{"summary":"s","why":"w","angles":[],"format":"blog"}`,
`{"summary":"s",`,
}
for _, c := range cases {
if _, err := parseResult(c); err == nil {
t.Errorf("parseResult(%q) = nil error, want error", c)
}
}
}
func TestNormalizeFormat(t *testing.T) {
cases := map[string]string{
"newsletter": FormatNewsletter,
"VIDEO": FormatVideo,
"blog": FormatBlog,
"nonsense": FormatBlog,
"": FormatBlog,
}
for in, want := range cases {
if got := normalizeFormat(in); got != want {
t.Errorf("normalizeFormat(%q) = %q, want %q", in, got, want)
}
}
}

126
PROJECTS/intermediate/security-news-scraper/internal/ai/provider.go Normal file
View File

@ -0,0 +1,126 @@
// ©AngelaMos | 2026
// provider.go
package ai
import (
"context"
"errors"
"fmt"
"net/http"
"os"
"time"
"github.com/CarterPerez-dev/nadezhda/internal/config"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const (
ProviderQwen = "qwen"
ProviderOpenAI = "openai"
ProviderGemini = "gemini"
ProviderAnthropic = "anthropic"
FormatBlog = "blog"
FormatNewsletter = "newsletter"
FormatVideo = "video"
envOpenAIKey = "OPENAI_API_KEY"
envGeminiKey = "GEMINI_API_KEY"
envAnthropicKey = "ANTHROPIC_API_KEY"
requestTimeout = 120 * time.Second
defaultMaxTokens = 2048
maxJSONBytes = 4 << 20
secondsPerHour = 3600
)
var ErrRefused = errors.New("ai: provider declined to generate for this item")
type CVEContext struct {
ID string
CVSS *float64
KEV bool
EPSS *float64
}
type IdeationRequest struct {
Titles []string
Sources []string
CVEs []CVEContext
ClusterSize int
SpanHours int
}
type IdeationResult struct {
Summary string `json:"summary"`
Why string `json:"why"`
Angles []string `json:"angles"`
Format string `json:"format"`
}
type Provider interface {
Name() string
Generate(ctx context.Context, req IdeationRequest) (IdeationResult, error)
}
func noRedirect(*http.Request, []*http.Request) error {
return http.ErrUseLastResponse
}
func Factory(cfg config.AI) (Provider, error) {
client := &http.Client{Timeout: requestTimeout, CheckRedirect: noRedirect}
switch cfg.Provider {
case ProviderQwen:
return newOpenAICompat(ProviderQwen, client, cfg.Qwen.BaseURL, cfg.Qwen.Model, ""), nil
case ProviderOpenAI:
key := os.Getenv(envOpenAIKey)
if key == "" {
return nil, fmt.Errorf("ai: provider %q requires %s in the environment", ProviderOpenAI, envOpenAIKey)
}
return newOpenAICompat(ProviderOpenAI, client, cfg.OpenAI.BaseURL, cfg.OpenAI.Model, key), nil
case ProviderGemini:
key := os.Getenv(envGeminiKey)
if key == "" {
return nil, fmt.Errorf("ai: provider %q requires %s in the environment", ProviderGemini, envGeminiKey)
}
return newOpenAICompat(ProviderGemini, client, cfg.Gemini.BaseURL, cfg.Gemini.Model, key), nil
case ProviderAnthropic:
key := os.Getenv(envAnthropicKey)
if key == "" {
return nil, fmt.Errorf("ai: provider %q requires %s in the environment", ProviderAnthropic, envAnthropicKey)
}
return newAnthropic(client, cfg.Anthropic.BaseURL, cfg.Anthropic.Model, key), nil
default:
return nil, fmt.Errorf("ai: unknown provider %q", cfg.Provider)
}
}
func RequestFromCluster(c store.DigestCluster) IdeationRequest {
req := IdeationRequest{ClusterSize: c.Size}
if c.LastSeen > c.FirstSeen {
req.SpanHours = int((c.LastSeen - c.FirstSeen) / secondsPerHour)
}
seenTitle := make(map[string]bool)
seenSource := make(map[string]bool)
for _, a := range c.Articles {
if a.Title != "" && !seenTitle[a.Title] {
seenTitle[a.Title] = true
req.Titles = append(req.Titles, a.Title)
}
if a.SourceName != "" && !seenSource[a.SourceName] {
seenSource[a.SourceName] = true
req.Sources = append(req.Sources, a.SourceName)
}
}
for _, v := range c.CVEs {
req.CVEs = append(req.CVEs, CVEContext{ID: v.ID, CVSS: v.CVSSScore, KEV: v.IsKEV, EPSS: v.EPSS})
}
return req
}
var (
_ Provider = (*openAICompat)(nil)
_ Provider = (*anthropicClient)(nil)
_ Provider = (*MockProvider)(nil)
)

208
PROJECTS/intermediate/security-news-scraper/internal/cluster/cluster.go Normal file
View File

@ -0,0 +1,208 @@
// ©AngelaMos | 2026
// cluster.go
package cluster
import (
"sort"
"strconv"
"strings"
"github.com/CarterPerez-dev/nadezhda/internal/normalize"
)
type Item struct {
ID int64
SourceID int64
Title string
CVEs []string
Time int64
}
type Cluster struct {
Key string
Members []int64
SourceCount int
FirstSeen int64
LastSeen int64
}
type tokenized struct {
item Item
tokens map[string]struct{}
cves map[string]struct{}
}
func Compute(items []Item, jaccardThreshold float64, windowSeconds int64) []Cluster {
n := len(items)
prepared := make([]tokenized, n)
for i, it := range items {
prepared[i] = tokenized{
item: it,
tokens: tokenSet(it.Title),
cves: stringSet(it.CVEs),
}
}
uf := newUnionFind(n)
for i := 0; i < n; i++ {
for j := i + 1; j < n; j++ {
if abs64(prepared[i].item.Time-prepared[j].item.Time) > windowSeconds {
continue
}
sharedCVE := shareAny(prepared[i].cves, prepared[j].cves)
crossSource := prepared[i].item.SourceID != prepared[j].item.SourceID
titleMatch := crossSource &&
jaccard(prepared[i].tokens, prepared[j].tokens) >= jaccardThreshold
if sharedCVE || titleMatch {
uf.union(i, j)
}
}
}
groups := make(map[int][]int)
for i := 0; i < n; i++ {
root := uf.find(i)
groups[root] = append(groups[root], i)
}
clusters := make([]Cluster, 0, len(groups))
for _, members := range groups {
clusters = append(clusters, buildCluster(prepared, members))
}
sort.Slice(clusters, func(a, b int) bool {
if clusters[a].FirstSeen != clusters[b].FirstSeen {
return clusters[a].FirstSeen < clusters[b].FirstSeen
}
return clusters[a].Members[0] < clusters[b].Members[0]
})
return clusters
}
func buildCluster(prepared []tokenized, members []int) Cluster {
earliest := prepared[members[0]].item
ids := make([]int64, 0, len(members))
sources := make(map[int64]struct{}, len(members))
first := earliest.Time
last := earliest.Time
for _, m := range members {
it := prepared[m].item
ids = append(ids, it.ID)
sources[it.SourceID] = struct{}{}
if it.Time < first {
first = it.Time
}
if it.Time > last {
last = it.Time
}
if it.Time < earliest.Time || (it.Time == earliest.Time && it.ID < earliest.ID) {
earliest = it
}
}
sort.Slice(ids, func(a, b int) bool { return ids[a] < ids[b] })
return Cluster{
Key: strconv.FormatInt(earliest.ID, 10),
Members: ids,
SourceCount: len(sources),
FirstSeen: first,
LastSeen: last,
}
}
func tokenSet(title string) map[string]struct{} {
fields := strings.Fields(normalize.NormalizeTitle(title))
set := make(map[string]struct{}, len(fields))
for _, f := range fields {
set[f] = struct{}{}
}
return set
}
func stringSet(values []string) map[string]struct{} {
set := make(map[string]struct{}, len(values))
for _, v := range values {
if v != "" {
set[v] = struct{}{}
}
}
return set
}
func jaccard(a, b map[string]struct{}) float64 {
if len(a) == 0 || len(b) == 0 {
return 0
}
small, large := a, b
if len(small) > len(large) {
small, large = large, small
}
intersection := 0
for k := range small {
if _, ok := large[k]; ok {
intersection++
}
}
union := len(a) + len(b) - intersection
if union == 0 {
return 0
}
return float64(intersection) / float64(union)
}
func shareAny(a, b map[string]struct{}) bool {
if len(a) == 0 || len(b) == 0 {
return false
}
small, large := a, b
if len(small) > len(large) {
small, large = large, small
}
for k := range small {
if _, ok := large[k]; ok {
return true
}
}
return false
}
func abs64(v int64) int64 {
if v < 0 {
return -v
}
return v
}
type unionFind struct {
parent []int
rank []int
}
func newUnionFind(n int) *unionFind {
uf := &unionFind{parent: make([]int, n), rank: make([]int, n)}
for i := range uf.parent {
uf.parent[i] = i
}
return uf
}
func (uf *unionFind) find(x int) int {
for uf.parent[x] != x {
uf.parent[x] = uf.parent[uf.parent[x]]
x = uf.parent[x]
}
return x
}
func (uf *unionFind) union(a, b int) {
ra, rb := uf.find(a), uf.find(b)
if ra == rb {
return
}
if uf.rank[ra] < uf.rank[rb] {
ra, rb = rb, ra
}
uf.parent[rb] = ra
if uf.rank[ra] == uf.rank[rb] {
uf.rank[ra]++
}
}

148
PROJECTS/intermediate/security-news-scraper/internal/cluster/cluster_test.go Normal file
View File

@ -0,0 +1,148 @@
// ©AngelaMos | 2026
// cluster_test.go
package cluster
import (
"sort"
"testing"
)
const bigWindow = 1 << 40
func sizes(clusters []Cluster) []int {
out := make([]int, len(clusters))
for i, c := range clusters {
out[i] = len(c.Members)
}
sort.Ints(out)
return out
}
func findCluster(clusters []Cluster, id int64) Cluster {
for _, c := range clusters {
for _, m := range c.Members {
if m == id {
return c
}
}
}
return Cluster{}
}
func TestComputeTitleSimilarityGroups(t *testing.T) {
items := []Item{
{ID: 1, SourceID: 1, Title: "Acme Corp data breach exposes millions", Time: 100},
{ID: 2, SourceID: 2, Title: "Acme Corp data breach exposed millions", Time: 200},
{ID: 3, SourceID: 3, Title: "Acme Corp data breach hits millions", Time: 300},
{ID: 4, SourceID: 4, Title: "Linux kernel 7 released with new scheduler", Time: 400},
}
clusters := Compute(items, 0.6, bigWindow)
if got := sizes(clusters); len(got) != 2 || got[0] != 1 || got[1] != 3 {
t.Fatalf("cluster sizes = %v, want [1 3]", got)
}
c := findCluster(clusters, 1)
if len(c.Members) != 3 {
t.Errorf("article 1 cluster size = %d, want 3", len(c.Members))
}
if c.SourceCount != 3 {
t.Errorf("SourceCount = %d, want 3 (three distinct outlets)", c.SourceCount)
}
if c.Key != "1" {
t.Errorf("cluster key = %q, want 1 (earliest article)", c.Key)
}
if c.FirstSeen != 100 || c.LastSeen != 300 {
t.Errorf("span = [%d,%d], want [100,300]", c.FirstSeen, c.LastSeen)
}
}
func TestComputeSameSourceTitlesDoNotMerge(t *testing.T) {
items := []Item{
{ID: 1, SourceID: 1, Title: "CISA Adds One Known Exploited Vulnerability to Catalog", Time: 100},
{ID: 2, SourceID: 1, Title: "CISA Adds One Known Exploited Vulnerability to Catalog", Time: 200},
}
if got := sizes(Compute(items, 0.6, bigWindow)); len(got) != 2 {
t.Errorf("same-source identical titles sizes = %v, want [1 1] (clustering is cross-outlet)", got)
}
}
func TestComputeSharedCVEJoinsRegardlessOfSource(t *testing.T) {
items := []Item{
{ID: 1, SourceID: 1, Title: "Router vendor patches critical flaw", Time: 100, CVEs: []string{"CVE-2026-1111"}},
{ID: 2, SourceID: 1, Title: "Enterprise gear gets emergency update", Time: 200, CVEs: []string{"CVE-2026-1111"}},
{ID: 3, SourceID: 2, Title: "Totally unrelated privacy story", Time: 300, CVEs: []string{"CVE-2026-9999"}},
}
clusters := Compute(items, 0.6, bigWindow)
if got := sizes(clusters); len(got) != 2 || got[0] != 1 || got[1] != 2 {
t.Fatalf("cluster sizes = %v, want [1 2]", got)
}
joined := findCluster(clusters, 1)
if len(joined.Members) != 2 {
t.Error("shared-CVE articles should join even from the same source")
}
if joined.SourceCount != 1 {
t.Errorf("SourceCount = %d, want 1 (a same-source cluster is NOT multi-source)", joined.SourceCount)
}
}
func TestComputeThresholdBoundary(t *testing.T) {
items := []Item{
{ID: 1, SourceID: 1, Title: "alpha bravo charlie delta", Time: 100},
{ID: 2, SourceID: 2, Title: "alpha bravo charlie echo", Time: 200},
}
if got := sizes(Compute(items, 0.6, bigWindow)); len(got) != 1 || got[0] != 2 {
t.Errorf("at threshold 0.6 (jaccard==0.6) sizes = %v, want [2] (>= joins)", got)
}
if got := sizes(Compute(items, 0.61, bigWindow)); len(got) != 2 {
t.Errorf("at threshold 0.61 sizes = %v, want [1 1] (below threshold)", got)
}
}
func TestComputeWindowSeparates(t *testing.T) {
items := []Item{
{ID: 1, SourceID: 1, Title: "identical breaking headline text", Time: 0},
{ID: 2, SourceID: 2, Title: "identical breaking headline text", Time: 200},
}
if got := sizes(Compute(items, 0.6, 100)); len(got) != 2 {
t.Errorf("outside window sizes = %v, want [1 1] (time gap > window)", got)
}
if got := sizes(Compute(items, 0.6, 300)); len(got) != 1 || got[0] != 2 {
t.Errorf("inside window sizes = %v, want [2]", got)
}
}
func TestComputeKeyIsEarliestByTime(t *testing.T) {
items := []Item{
{ID: 10, SourceID: 1, Title: "shared story about the incident", Time: 500},
{ID: 20, SourceID: 2, Title: "shared story about the incident", Time: 100},
}
clusters := Compute(items, 0.6, bigWindow)
if len(clusters) != 1 {
t.Fatalf("clusters = %d, want 1", len(clusters))
}
if clusters[0].Key != "20" {
t.Errorf("key = %q, want 20 (earliest by time, not lowest id)", clusters[0].Key)
}
if clusters[0].FirstSeen != 100 || clusters[0].LastSeen != 500 {
t.Errorf("span = [%d,%d], want [100,500]", clusters[0].FirstSeen, clusters[0].LastSeen)
}
}
func TestComputeTransitiveChain(t *testing.T) {
items := []Item{
{ID: 1, SourceID: 1, Title: "one two three four five", Time: 100},
{ID: 2, SourceID: 2, Title: "one two three four six", Time: 100},
{ID: 3, SourceID: 3, Title: "one two three six five", Time: 100},
}
clusters := Compute(items, 0.6, bigWindow)
if len(clusters) != 1 || len(clusters[0].Members) != 3 {
t.Errorf("transitive similarity should merge all three, got %d clusters", len(clusters))
}
}
func TestComputeEmpty(t *testing.T) {
if c := Compute(nil, 0.6, bigWindow); len(c) != 0 {
t.Errorf("empty input -> %d clusters, want 0", len(c))
}
}

51
PROJECTS/intermediate/security-news-scraper/internal/cluster/rebuild.go Normal file
View File

@ -0,0 +1,51 @@
// ©AngelaMos | 2026
// rebuild.go
package cluster
import (
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
type Stats struct {
Total int
MultiSource int
LargestSize int
}
func Rebuild(st *store.Store, jaccardThreshold float64, windowSeconds, sinceUnix int64) (Stats, error) {
candidates, err := st.ClusterCandidates(sinceUnix)
if err != nil {
return Stats{}, err
}
cveMap, err := st.ArticleCVEMap()
if err != nil {
return Stats{}, err
}
items := make([]Item, len(candidates))
for i, c := range candidates {
items[i] = Item{ID: c.ID, SourceID: c.SourceID, Title: c.Title, Time: c.Time, CVEs: cveMap[c.ID]}
}
clusters := Compute(items, jaccardThreshold, windowSeconds)
rows := make([]store.ClusterRow, len(clusters))
stats := Stats{Total: len(clusters)}
for i, c := range clusters {
rows[i] = store.ClusterRow{
Key: c.Key, Members: c.Members, FirstSeen: c.FirstSeen, LastSeen: c.LastSeen,
}
if c.SourceCount > 1 {
stats.MultiSource++
}
if size := len(c.Members); size > stats.LargestSize {
stats.LargestSize = size
}
}
if err := st.ReplaceClusters(rows); err != nil {
return Stats{}, err
}
return stats, nil
}

133
PROJECTS/intermediate/security-news-scraper/internal/cluster/rebuild_test.go Normal file
View File

@ -0,0 +1,133 @@
// ©AngelaMos | 2026
// rebuild_test.go
package cluster
import (
"fmt"
"path/filepath"
"testing"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
func openStore(t *testing.T) *store.Store {
t.Helper()
st, err := store.Open(filepath.Join(t.TempDir(), "cluster.db"))
if err != nil {
t.Fatalf("open store: %v", err)
}
t.Cleanup(func() { _ = st.Close() })
return st
}
func seedArticles(t *testing.T, st *store.Store) {
t.Helper()
titles := []string{
"Acme Corp data breach exposes millions",
"Acme Corp data breach exposed millions",
"Acme Corp data breach hits millions",
"Linux kernel 7 released with new scheduler",
}
for i, title := range titles {
sourceID, err := st.UpsertSource(store.SourceInput{
Name: fmt.Sprintf("src%d", i), URL: fmt.Sprintf("https://outlet%d.example/feed", i),
Type: "rss", Weight: 1, Enabled: true,
})
if err != nil {
t.Fatalf("upsert source %d: %v", i, err)
}
if _, err := st.InsertArticle(store.Article{
SourceID: sourceID,
CanonicalURL: fmt.Sprintf("https://example.com/a/%d", i),
ContentHash: fmt.Sprintf("chash-%d", i),
TitleHash: fmt.Sprintf("thash-%d", i),
Title: title,
PublishedAt: int64(100 * (i + 1)),
FetchedAt: int64(100 * (i + 1)),
}); err != nil {
t.Fatalf("insert article %d: %v", i, err)
}
}
}
func clusterSizes(t *testing.T, st *store.Store) []int {
t.Helper()
rows, err := st.DB().Query(`SELECT size FROM clusters ORDER BY size`)
if err != nil {
t.Fatal(err)
}
defer rows.Close()
var out []int
for rows.Next() {
var n int
if err := rows.Scan(&n); err != nil {
t.Fatal(err)
}
out = append(out, n)
}
return out
}
func countMembers(t *testing.T, st *store.Store) int {
t.Helper()
var n int
if err := st.DB().QueryRow(`SELECT COUNT(*) FROM cluster_members`).Scan(&n); err != nil {
t.Fatal(err)
}
return n
}
func TestRebuildClustersEndToEnd(t *testing.T) {
st := openStore(t)
seedArticles(t, st)
stats, err := Rebuild(st, 0.6, bigWindow, 0)
if err != nil {
t.Fatalf("Rebuild: %v", err)
}
if stats.Total != 2 || stats.MultiSource != 1 || stats.LargestSize != 3 {
t.Fatalf("stats = %+v, want {Total:2 MultiSource:1 LargestSize:3}", stats)
}
if got := clusterSizes(t, st); len(got) != 2 || got[0] != 1 || got[1] != 3 {
t.Errorf("cluster sizes = %v, want [1 3]", got)
}
if m := countMembers(t, st); m != 4 {
t.Errorf("cluster_members = %d, want 4 (every article assigned once)", m)
}
}
func TestRebuildIsIdempotent(t *testing.T) {
st := openStore(t)
seedArticles(t, st)
if _, err := Rebuild(st, 0.6, bigWindow, 0); err != nil {
t.Fatalf("first Rebuild: %v", err)
}
stats, err := Rebuild(st, 0.6, bigWindow, 0)
if err != nil {
t.Fatalf("second Rebuild: %v", err)
}
if stats.Total != 2 {
t.Errorf("second run clusters = %d, want 2 (rebuild replaces, no accumulation)", stats.Total)
}
if m := countMembers(t, st); m != 4 {
t.Errorf("cluster_members = %d, want 4 after re-run", m)
}
}
func TestRebuildLookbackExcludesOld(t *testing.T) {
st := openStore(t)
seedArticles(t, st)
stats, err := Rebuild(st, 0.6, bigWindow, 250)
if err != nil {
t.Fatalf("Rebuild: %v", err)
}
if stats.Total != 2 {
t.Errorf("with since=250 only articles at t>=250 cluster, total = %d, want 2", stats.Total)
}
if m := countMembers(t, st); m != 2 {
t.Errorf("members = %d, want 2 (two articles below lookback excluded)", m)
}
}

271
PROJECTS/intermediate/security-news-scraper/internal/config/config.go Normal file
View File

@ -0,0 +1,271 @@
// ©AngelaMos | 2026
// config.go
package config
import (
"fmt"
"os"
"time"
"gopkg.in/yaml.v3"
)
const (
defaultDBPath = "nadezhda.db"
defaultUserAgent = "nadezhda/0.1 (+https://github.com/CarterPerez-dev/nadezhda)"
defaultPerHostRate = 0.5
defaultPerHostBurst = 1
defaultTimeoutSeconds = 25
defaultSourceTimeoutSeconds = 90
defaultWorkers = 8
defaultMaxRetries = 3
defaultCacheTTLHours = 24
defaultNegativeTTLHours = 3
defaultTitleJaccard = 0.6
defaultWindowHours = 72
defaultLookbackHours = 168
trackingUTMPrefix = "utm_*"
defaultHalfLifeHours = 48
defaultVelocityNorm = 0.5
defaultWeightRecency = 0.30
defaultWeightVelocity = 0.20
defaultWeightKEV = 0.12
defaultWeightCVSS = 0.10
defaultWeightSource = 0.10
defaultWeightKeyword = 0.10
defaultWeightEPSS = 0.08
defaultAIProvider = "qwen"
defaultQwenBaseURL = "http://localhost:39847/v1"
defaultQwenModel = "qwen2.5:7b"
defaultOpenAIURL = "https://api.openai.com/v1"
defaultOpenAIModel = "gpt-4o-mini"
defaultGeminiURL = "https://generativelanguage.googleapis.com/v1beta/openai/"
defaultGeminiModel = "gemini-2.5-flash"
defaultAnthropicURL = "https://api.anthropic.com/v1"
defaultClaudeModel = "claude-sonnet-4-6"
defaultWatchInterval = "1h"
defaultWatchNotifyMinScore = 0.5
defaultWatchNotifyMax = 10
MinWatchInterval = time.Minute
)
var defaultTrackingParams = []string{
trackingUTMPrefix, "gclid", "fbclid", "ref", "mc_cid", "mc_eid",
}
type Fetch struct {
UserAgent string `yaml:"user_agent"`
PerHostRate float64 `yaml:"per_host_rate"`
PerHostBurst int `yaml:"per_host_burst"`
TimeoutSeconds int `yaml:"timeout_seconds"`
SourceTimeoutSeconds int `yaml:"source_timeout_seconds"`
Workers int `yaml:"workers"`
MaxRetries int `yaml:"max_retries"`
}
type Enrich struct {
CacheTTLHours int `yaml:"cache_ttl_hours"`
NegativeTTLHours int `yaml:"negative_ttl_hours"`
NVDAPIKey string `yaml:"nvd_api_key"`
}
type Cluster struct {
TitleJaccard float64 `yaml:"title_jaccard_threshold"`
WindowHours int `yaml:"window_hours"`
LookbackHours int `yaml:"lookback_hours"`
TrackingParams []string `yaml:"tracking_params"`
}
type Weights struct {
Recency float64 `yaml:"recency"`
CVSS float64 `yaml:"cvss"`
KEV float64 `yaml:"kev"`
EPSS float64 `yaml:"epss"`
Velocity float64 `yaml:"velocity"`
Source float64 `yaml:"source"`
Keyword float64 `yaml:"keyword"`
}
type Rank struct {
HalfLifeHours int `yaml:"half_life_hours"`
VelocityNorm float64 `yaml:"velocity_norm"`
Weights Weights `yaml:"weights"`
}
type Provider struct {
BaseURL string `yaml:"base_url"`
Model string `yaml:"model"`
}
type AI struct {
Enabled bool `yaml:"enabled"`
Provider string `yaml:"provider"`
Qwen Provider `yaml:"qwen"`
OpenAI Provider `yaml:"openai"`
Gemini Provider `yaml:"gemini"`
Anthropic Provider `yaml:"anthropic"`
}
type Watch struct {
Interval string `yaml:"interval"`
WebhookURL string `yaml:"webhook_url"`
NotifyMinScore float64 `yaml:"notify_min_score"`
NotifyMaxItems int `yaml:"notify_max_items"`
NotifyOnKEV bool `yaml:"notify_on_kev"`
}
type Config struct {
DBPath string `yaml:"db_path"`
SourcesPath string `yaml:"sources_path"`
Watchlist []string `yaml:"watchlist"`
Fetch Fetch `yaml:"fetch"`
Enrich Enrich `yaml:"enrich"`
Cluster Cluster `yaml:"cluster"`
Rank Rank `yaml:"rank"`
AI AI `yaml:"ai"`
Watch Watch `yaml:"watch"`
}
func Default() Config {
return Config{
DBPath: defaultDBPath,
Fetch: Fetch{
UserAgent: defaultUserAgent,
PerHostRate: defaultPerHostRate,
PerHostBurst: defaultPerHostBurst,
TimeoutSeconds: defaultTimeoutSeconds,
SourceTimeoutSeconds: defaultSourceTimeoutSeconds,
Workers: defaultWorkers,
MaxRetries: defaultMaxRetries,
},
Enrich: Enrich{
CacheTTLHours: defaultCacheTTLHours,
NegativeTTLHours: defaultNegativeTTLHours,
},
Cluster: Cluster{
TitleJaccard: defaultTitleJaccard,
WindowHours: defaultWindowHours,
LookbackHours: defaultLookbackHours,
TrackingParams: defaultTrackingParams,
},
Rank: Rank{
HalfLifeHours: defaultHalfLifeHours,
VelocityNorm: defaultVelocityNorm,
Weights: Weights{
Recency: defaultWeightRecency,
CVSS: defaultWeightCVSS,
KEV: defaultWeightKEV,
EPSS: defaultWeightEPSS,
Velocity: defaultWeightVelocity,
Source: defaultWeightSource,
Keyword: defaultWeightKeyword,
},
},
AI: AI{
Enabled: false,
Provider: defaultAIProvider,
Qwen: Provider{BaseURL: defaultQwenBaseURL, Model: defaultQwenModel},
OpenAI: Provider{BaseURL: defaultOpenAIURL, Model: defaultOpenAIModel},
Gemini: Provider{BaseURL: defaultGeminiURL, Model: defaultGeminiModel},
Anthropic: Provider{BaseURL: defaultAnthropicURL, Model: defaultClaudeModel},
},
Watch: Watch{
Interval: defaultWatchInterval,
NotifyMinScore: defaultWatchNotifyMinScore,
NotifyMaxItems: defaultWatchNotifyMax,
NotifyOnKEV: true,
},
}
}
func Load(path string) (Config, error) {
cfg := Default()
if path == "" {
return cfg, nil
}
raw, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return cfg, nil
}
return Config{}, fmt.Errorf("read config %s: %w", path, err)
}
if err := yaml.Unmarshal(raw, &cfg); err != nil {
return Config{}, fmt.Errorf("parse config %s: %w", path, err)
}
if err := cfg.validate(); err != nil {
return Config{}, err
}
return cfg, nil
}
func (c Config) validate() error {
if c.DBPath == "" {
return fmt.Errorf("config: db_path must not be empty")
}
if c.Fetch.Workers < 1 {
return fmt.Errorf("config: fetch.workers must be >= 1, got %d", c.Fetch.Workers)
}
if c.Fetch.PerHostRate <= 0 {
return fmt.Errorf("config: fetch.per_host_rate must be > 0, got %v", c.Fetch.PerHostRate)
}
if c.Fetch.PerHostBurst < 1 {
return fmt.Errorf("config: fetch.per_host_burst must be >= 1, got %d", c.Fetch.PerHostBurst)
}
if c.Fetch.TimeoutSeconds < 1 {
return fmt.Errorf("config: fetch.timeout_seconds must be >= 1, got %d", c.Fetch.TimeoutSeconds)
}
if c.Fetch.SourceTimeoutSeconds < 1 {
return fmt.Errorf("config: fetch.source_timeout_seconds must be >= 1, got %d", c.Fetch.SourceTimeoutSeconds)
}
if c.Enrich.CacheTTLHours < 0 || c.Enrich.NegativeTTLHours < 0 {
return fmt.Errorf("config: enrich TTLs must be >= 0, got cache=%d negative=%d", c.Enrich.CacheTTLHours, c.Enrich.NegativeTTLHours)
}
if c.Cluster.TitleJaccard < 0 || c.Cluster.TitleJaccard > 1 {
return fmt.Errorf("config: cluster.title_jaccard_threshold must be in [0,1], got %v", c.Cluster.TitleJaccard)
}
if c.Cluster.WindowHours < 1 {
return fmt.Errorf("config: cluster.window_hours must be >= 1, got %d", c.Cluster.WindowHours)
}
if c.Cluster.LookbackHours < 1 {
return fmt.Errorf("config: cluster.lookback_hours must be >= 1, got %d", c.Cluster.LookbackHours)
}
if c.Cluster.LookbackHours < c.Cluster.WindowHours {
return fmt.Errorf("config: cluster.lookback_hours (%d) must be >= cluster.window_hours (%d) or window edges near the corpus boundary are silently dropped", c.Cluster.LookbackHours, c.Cluster.WindowHours)
}
if c.Rank.HalfLifeHours < 1 {
return fmt.Errorf("config: rank.half_life_hours must be >= 1, got %d", c.Rank.HalfLifeHours)
}
if c.Rank.VelocityNorm <= 0 {
return fmt.Errorf("config: rank.velocity_norm must be > 0, got %v", c.Rank.VelocityNorm)
}
switch c.AI.Provider {
case "qwen", "openai", "gemini", "anthropic":
default:
return fmt.Errorf("config: ai.provider must be one of qwen|openai|gemini|anthropic, got %q", c.AI.Provider)
}
d, err := time.ParseDuration(c.Watch.Interval)
if err != nil {
return fmt.Errorf("config: watch.interval %q is not a valid duration: %w", c.Watch.Interval, err)
}
if d < MinWatchInterval {
return fmt.Errorf("config: watch.interval must be >= %s, got %s", MinWatchInterval, c.Watch.Interval)
}
if c.Watch.NotifyMinScore < 0 || c.Watch.NotifyMinScore > 1 {
return fmt.Errorf("config: watch.notify_min_score must be in [0,1], got %v", c.Watch.NotifyMinScore)
}
if c.Watch.NotifyMaxItems < 1 {
return fmt.Errorf("config: watch.notify_max_items must be >= 1, got %d", c.Watch.NotifyMaxItems)
}
return nil
}

106
PROJECTS/intermediate/security-news-scraper/internal/config/config_test.go Normal file
View File

@ -0,0 +1,106 @@
// ©AngelaMos | 2026
// config_test.go
package config
import (
"os"
"path/filepath"
"testing"
)
func TestDefaultHasSaneValues(t *testing.T) {
c := Default()
if c.DBPath != defaultDBPath {
t.Errorf("db_path = %q, want %q", c.DBPath, defaultDBPath)
}
if c.AI.Enabled {
t.Error("ai.enabled must default to false (opt-in)")
}
if c.AI.Provider != "qwen" {
t.Errorf("ai.provider = %q, want qwen", c.AI.Provider)
}
if c.AI.Qwen.Model != defaultQwenModel {
t.Errorf("qwen model = %q, want %q", c.AI.Qwen.Model, defaultQwenModel)
}
sum := c.Rank.Weights.Recency + c.Rank.Weights.CVSS + c.Rank.Weights.KEV +
c.Rank.Weights.EPSS + c.Rank.Weights.Velocity + c.Rank.Weights.Source +
c.Rank.Weights.Keyword
if sum < 0.99 || sum > 1.01 {
t.Errorf("rank weights sum = %v, want ~1.0", sum)
}
}
func TestLoadMissingFileReturnsDefaults(t *testing.T) {
c, err := Load(filepath.Join(t.TempDir(), "nope.yaml"))
if err != nil {
t.Fatalf("Load missing file: %v", err)
}
if c.DBPath != defaultDBPath {
t.Errorf("missing file should yield defaults, got db_path %q", c.DBPath)
}
}
func TestLoadOverridesDefaults(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "config.yaml")
body := `
db_path: /tmp/custom.db
watchlist: [fortinet, "cisco ios"]
fetch:
workers: 4
ai:
enabled: true
provider: anthropic
anthropic:
model: claude-sonnet-4-6
`
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
c, err := Load(path)
if err != nil {
t.Fatalf("Load: %v", err)
}
if c.DBPath != "/tmp/custom.db" {
t.Errorf("db_path = %q, want /tmp/custom.db", c.DBPath)
}
if len(c.Watchlist) != 2 || c.Watchlist[0] != "fortinet" {
t.Errorf("watchlist = %v", c.Watchlist)
}
if c.Fetch.Workers != 4 {
t.Errorf("workers = %d, want 4", c.Fetch.Workers)
}
if !c.AI.Enabled || c.AI.Provider != "anthropic" {
t.Errorf("ai override failed: %+v", c.AI)
}
if c.AI.Anthropic.Model != "claude-sonnet-4-6" {
t.Errorf("anthropic model = %q, want claude-sonnet-4-6", c.AI.Anthropic.Model)
}
if c.AI.Qwen.Model != defaultQwenModel {
t.Errorf("unset qwen model should keep default, got %q", c.AI.Qwen.Model)
}
}
func TestValidateRejectsBadProvider(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "bad.yaml")
if err := os.WriteFile(path, []byte("ai:\n provider: llama\n"), 0o600); err != nil {
t.Fatal(err)
}
if _, err := Load(path); err == nil {
t.Error("expected error for invalid ai.provider")
}
}
func TestValidateRejectsLookbackBelowWindow(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "lb.yaml")
body := "cluster:\n window_hours: 72\n lookback_hours: 24\n"
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
if _, err := Load(path); err == nil {
t.Error("expected error when lookback_hours < window_hours")
}
}

207
PROJECTS/intermediate/security-news-scraper/internal/cve/cve_test.go Normal file
View File

@ -0,0 +1,207 @@
// ©AngelaMos | 2026
// cve_test.go
package cve
import (
"context"
"math"
"net/http"
"net/http/httptest"
"os"
"reflect"
"testing"
"time"
"golang.org/x/time/rate"
)
func fileServer(t *testing.T, path string) *httptest.Server {
t.Helper()
body, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read fixture %s: %v", path, err)
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write(body)
}))
t.Cleanup(srv.Close)
return srv
}
func jsonServer(t *testing.T, body string) *httptest.Server {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(body))
}))
t.Cleanup(srv.Close)
return srv
}
func TestExtract(t *testing.T) {
cases := []struct {
in []string
want []string
}{
{[]string{"See CVE-2021-44228 for details"}, []string{"CVE-2021-44228"}},
{[]string{"cve-2021-44228 lowercase"}, []string{"CVE-2021-44228"}},
{[]string{"CVE-2021-44228 and CVE-2021-44228 again"}, []string{"CVE-2021-44228"}},
{[]string{"CVE-2026-1 too short", "CVE-2026-12345 ok"}, []string{"CVE-2026-12345"}},
{[]string{"CVE-2021-45046, CVE-2021-44228"}, []string{"CVE-2021-44228", "CVE-2021-45046"}},
{[]string{"no identifiers here"}, nil},
}
for _, tc := range cases {
if got := Extract(tc.in...); !reflect.DeepEqual(got, tc.want) {
t.Errorf("Extract(%v) = %v, want %v", tc.in, got, tc.want)
}
}
}
func TestNVDParsesLog4Shell(t *testing.T) {
srv := fileServer(t, "../../testdata/nvd/CVE-2021-44228.json")
c := NewNVDClient(srv.Client(), srv.URL, "")
res, err := c.Fetch(context.Background(), "CVE-2021-44228")
if err != nil {
t.Fatalf("Fetch: %v", err)
}
if !res.Found {
t.Fatal("expected Found")
}
if res.CVSSScore == nil || *res.CVSSScore != 10.0 {
t.Errorf("cvss score = %v, want 10.0", res.CVSSScore)
}
if res.CVSSSeverity != "CRITICAL" {
t.Errorf("severity = %q, want CRITICAL", res.CVSSSeverity)
}
if res.CVSSVersion != "3.1" {
t.Errorf("version = %q, want 3.1", res.CVSSVersion)
}
if res.CWE != "CWE-20" {
t.Errorf("cwe = %q, want CWE-20", res.CWE)
}
if res.Description == "" {
t.Error("description empty")
}
}
func TestNVDNotFound(t *testing.T) {
srv := jsonServer(t, `{"totalResults":0,"vulnerabilities":[]}`)
c := NewNVDClient(srv.Client(), srv.URL, "")
res, err := c.Fetch(context.Background(), "CVE-2099-99999")
if err != nil {
t.Fatalf("Fetch: %v", err)
}
if res.Found {
t.Error("expected not found for totalResults 0")
}
}
func TestNVDPrecedencePrefersV40(t *testing.T) {
body := `{"totalResults":1,"vulnerabilities":[{"cve":{
"id":"CVE-2026-1","descriptions":[{"lang":"en","value":"d"}],
"metrics":{
"cvssMetricV40":[{"cvssData":{"version":"4.0","baseScore":9.3,"baseSeverity":"CRITICAL","vectorString":"CVSS:4.0/x"}}],
"cvssMetricV31":[{"cvssData":{"version":"3.1","baseScore":7.5,"baseSeverity":"HIGH","vectorString":"CVSS:3.1/y"}}]
}}}]}`
srv := jsonServer(t, body)
c := NewNVDClient(srv.Client(), srv.URL, "")
res, err := c.Fetch(context.Background(), "CVE-2026-1")
if err != nil {
t.Fatal(err)
}
if res.CVSSVersion != "4.0" || res.CVSSScore == nil || *res.CVSSScore != 9.3 {
t.Errorf("expected v4.0/9.3, got %s/%v", res.CVSSVersion, res.CVSSScore)
}
}
func TestNVDV2SeverityFromMetricLevel(t *testing.T) {
body := `{"totalResults":1,"vulnerabilities":[{"cve":{
"id":"CVE-2005-1","descriptions":[{"lang":"en","value":"d"}],
"metrics":{"cvssMetricV2":[{"baseSeverity":"HIGH","cvssData":{"version":"2.0","baseScore":7.5,"vectorString":"AV:N"}}]}
}}]}`
srv := jsonServer(t, body)
c := NewNVDClient(srv.Client(), srv.URL, "")
res, err := c.Fetch(context.Background(), "CVE-2005-1")
if err != nil {
t.Fatal(err)
}
if res.CVSSVersion != "2.0" || res.CVSSSeverity != "HIGH" {
t.Errorf("v2 severity should fall back to metric level, got %s/%q", res.CVSSVersion, res.CVSSSeverity)
}
}
func TestKEVParsesSample(t *testing.T) {
srv := fileServer(t, "../../testdata/kev/kev-sample.json")
c := NewKEVClient(srv.Client(), srv.URL)
catalog, err := c.Fetch(context.Background())
if err != nil {
t.Fatalf("Fetch: %v", err)
}
log4, ok := catalog.Lookup("CVE-2021-44228")
if !ok {
t.Fatal("Log4Shell should be in KEV")
}
if !log4.Ransomware {
t.Error("Log4Shell knownRansomwareCampaignUse=Known should map to true")
}
unknown, ok := catalog.Lookup("CVE-2026-45659")
if !ok || unknown.Ransomware {
t.Error("Unknown ransomware entry should map to false")
}
if _, ok := catalog.Lookup("CVE-1999-0001"); ok {
t.Error("absent CVE should not be a KEV member")
}
}
func TestEPSSParsesQuotedStrings(t *testing.T) {
srv := fileServer(t, "../../testdata/epss/CVE-2021-44228.json")
c := NewEPSSClient(srv.Client(), srv.URL)
scores, err := c.Fetch(context.Background(), []string{"CVE-2021-44228"})
if err != nil {
t.Fatalf("Fetch: %v", err)
}
s, ok := scores["CVE-2021-44228"]
if !ok {
t.Fatal("expected a score for Log4Shell")
}
if math.Abs(s.EPSS-0.99999) > 1e-5 {
t.Errorf("epss = %v, want ~0.99999 (string must be ParseFloat'd, not zeroed)", s.EPSS)
}
if math.Abs(s.Percentile-1.0) > 1e-9 {
t.Errorf("percentile = %v, want 1.0", s.Percentile)
}
}
func TestChunk(t *testing.T) {
got := chunk([]string{"a", "b", "c", "d", "e"}, 2)
want := [][]string{{"a", "b"}, {"c", "d"}, {"e"}}
if !reflect.DeepEqual(got, want) {
t.Errorf("chunk = %v, want %v", got, want)
}
if chunk(nil, 2) != nil {
t.Error("chunk(nil) should be nil")
}
}
func TestNVDRetriesOn503(t *testing.T) {
var calls int
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls++
if calls == 1 {
w.WriteHeader(http.StatusServiceUnavailable)
return
}
_, _ = w.Write([]byte(`{"totalResults":0,"vulnerabilities":[]}`))
}))
t.Cleanup(srv.Close)
c := NewNVDClient(srv.Client(), srv.URL, "")
c.limiter = rate.NewLimiter(rate.Inf, 1)
c.backoffBase = time.Millisecond
if _, err := c.Fetch(context.Background(), "CVE-2026-1"); err != nil {
t.Fatalf("Fetch after retry: %v", err)
}
if calls != 2 {
t.Errorf("calls = %d, want 2 (one 503, one 200)", calls)
}
}

249
PROJECTS/intermediate/security-news-scraper/internal/cve/cvelist.go Normal file
View File

@ -0,0 +1,249 @@
// ©AngelaMos | 2026
// cvelist.go
package cve
import (
"context"
"errors"
"fmt"
"net/http"
"strings"
"time"
"golang.org/x/time/rate"
)
const (
CVEListEndpoint = "https://raw.githubusercontent.com/CVEProject/cvelistV5/main/cves"
cvelistRate = 15
cvelistMaxRetries = 2
cvelistBackoff = time.Second
cvelistBucketTail = 3
cveIDParts = 3
langEnPrefix = "en"
cvssCriticalMin = 9.0
cvssHighMin = 7.0
cvssMediumMin = 4.0
cvssVersion2Prefix = "2"
sevCritical = "CRITICAL"
sevHigh = "HIGH"
sevMedium = "MEDIUM"
sevLow = "LOW"
sevNone = "NONE"
)
type CoreResult struct {
Found bool
Description string
CVSSScore *float64
CVSSVersion string
CVSSSeverity string
CVSSVector string
CWE string
Published string
Modified string
}
type CVESource interface {
Fetch(ctx context.Context, cveID string) (CoreResult, error)
}
type CVEListClient struct {
http *http.Client
baseURL string
limiter *rate.Limiter
backoffBase time.Duration
}
func NewCVEListClient(client *http.Client, baseURL string) *CVEListClient {
return &CVEListClient{
http: client,
baseURL: strings.TrimRight(baseURL, "/"),
limiter: rate.NewLimiter(rate.Limit(cvelistRate), 1),
backoffBase: cvelistBackoff,
}
}
func (c *CVEListClient) Fetch(ctx context.Context, cveID string) (CoreResult, error) {
endpoint, err := c.recordURL(cveID)
if err != nil {
return CoreResult{}, err
}
var rec cvelistRecord
for attempt := 0; attempt <= cvelistMaxRetries; attempt++ {
if attempt > 0 {
if err := sleep(ctx, c.backoffBase*time.Duration(1<<(attempt-1))); err != nil {
return CoreResult{}, err
}
}
if err := c.limiter.Wait(ctx); err != nil {
return CoreResult{}, err
}
err := getJSON(ctx, c.http, endpoint, nil, &rec)
if err == nil {
return rec.toResult(), nil
}
var se *statusError
if errors.As(err, &se) && se.code == http.StatusNotFound {
return CoreResult{Found: false}, nil
}
if !retriable(err) || attempt == cvelistMaxRetries {
return CoreResult{}, err
}
}
return CoreResult{Found: false}, nil
}
func (c *CVEListClient) recordURL(cveID string) (string, error) {
parts := strings.Split(cveID, "-")
if len(parts) != cveIDParts || parts[0] != "CVE" {
return "", fmt.Errorf("cve: malformed id %q", cveID)
}
num := parts[2]
if len(num) <= cvelistBucketTail {
return "", fmt.Errorf("cve: short id %q", cveID)
}
bucket := num[:len(num)-cvelistBucketTail] + "xxx"
return fmt.Sprintf("%s/%s/%s/%s.json", c.baseURL, parts[1], bucket, cveID), nil
}
type cvelistRecord struct {
CVEMetadata struct {
DatePublished string `json:"datePublished"`
DateUpdated string `json:"dateUpdated"`
} `json:"cveMetadata"`
Containers struct {
CNA cvelistContainer `json:"cna"`
ADP []cvelistContainer `json:"adp"`
} `json:"containers"`
}
type cvelistContainer struct {
Descriptions []cvelistLangValue `json:"descriptions"`
ProblemTypes []struct {
Descriptions []struct {
Lang string `json:"lang"`
CweID string `json:"cweId"`
} `json:"descriptions"`
} `json:"problemTypes"`
Metrics []cvelistMetric `json:"metrics"`
}
type cvelistLangValue struct {
Lang string `json:"lang"`
Value string `json:"value"`
}
type cvelistMetric struct {
V40 *cvelistCVSS `json:"cvssV4_0"`
V31 *cvelistCVSS `json:"cvssV3_1"`
V30 *cvelistCVSS `json:"cvssV3_0"`
V2 *cvelistCVSS `json:"cvssV2_0"`
}
type cvelistCVSS struct {
Version string `json:"version"`
BaseScore float64 `json:"baseScore"`
BaseSeverity string `json:"baseSeverity"`
VectorString string `json:"vectorString"`
}
func (rec cvelistRecord) containers() []cvelistContainer {
out := make([]cvelistContainer, 0, len(rec.Containers.ADP)+1)
out = append(out, rec.Containers.CNA)
out = append(out, rec.Containers.ADP...)
return out
}
func (rec cvelistRecord) toResult() CoreResult {
res := CoreResult{
Found: true,
Description: rec.description(),
CWE: rec.cwe(),
Published: rec.CVEMetadata.DatePublished,
Modified: rec.CVEMetadata.DateUpdated,
}
if cv := rec.cvss(); cv != nil {
score := cv.BaseScore
res.CVSSScore = &score
res.CVSSVersion = cv.Version
res.CVSSVector = cv.VectorString
res.CVSSSeverity = cv.BaseSeverity
if res.CVSSSeverity == "" {
res.CVSSSeverity = severityFromScore(score, cv.Version)
}
}
return res
}
func (rec cvelistRecord) cvss() *cvelistCVSS {
tiers := []func(cvelistMetric) *cvelistCVSS{
func(m cvelistMetric) *cvelistCVSS { return m.V40 },
func(m cvelistMetric) *cvelistCVSS { return m.V31 },
func(m cvelistMetric) *cvelistCVSS { return m.V30 },
func(m cvelistMetric) *cvelistCVSS { return m.V2 },
}
containers := rec.containers()
for _, pick := range tiers {
for _, c := range containers {
for _, m := range c.Metrics {
if cv := pick(m); cv != nil {
return cv
}
}
}
}
return nil
}
func (rec cvelistRecord) cwe() string {
for _, c := range rec.containers() {
for _, pt := range c.ProblemTypes {
for _, d := range pt.Descriptions {
if d.CweID != "" {
return d.CweID
}
}
}
}
return ""
}
func (rec cvelistRecord) description() string {
for _, c := range rec.containers() {
for _, d := range c.Descriptions {
if strings.HasPrefix(d.Lang, langEnPrefix) && strings.TrimSpace(d.Value) != "" {
return d.Value
}
}
}
for _, c := range rec.containers() {
for _, d := range c.Descriptions {
if strings.TrimSpace(d.Value) != "" {
return d.Value
}
}
}
return ""
}
func severityFromScore(score float64, version string) string {
switch {
case score <= 0:
return sevNone
case score >= cvssCriticalMin && !strings.HasPrefix(version, cvssVersion2Prefix):
return sevCritical
case score >= cvssHighMin:
return sevHigh
case score >= cvssMediumMin:
return sevMedium
default:
return sevLow
}
}

132
PROJECTS/intermediate/security-news-scraper/internal/cve/cvelist_test.go Normal file
View File

@ -0,0 +1,132 @@
// ©AngelaMos | 2026
// cvelist_test.go
package cve
import (
"context"
"net/http"
"net/http/httptest"
"os"
"strings"
"testing"
)
func TestCVEListParsesLog4Shell(t *testing.T) {
body, err := os.ReadFile("../../testdata/cvelist/CVE-2021-44228.json")
if err != nil {
t.Fatalf("read fixture: %v", err)
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write(body)
}))
defer srv.Close()
client := NewCVEListClient(srv.Client(), srv.URL)
res, err := client.Fetch(context.Background(), "CVE-2021-44228")
if err != nil {
t.Fatalf("Fetch: %v", err)
}
if !res.Found {
t.Fatal("Found = false, want true")
}
if res.CVSSScore == nil || *res.CVSSScore != 10.0 {
t.Errorf("CVSSScore = %v, want 10.0", res.CVSSScore)
}
if res.CVSSVersion != "3.1" || res.CVSSSeverity != "CRITICAL" {
t.Errorf("cvss = %q/%q, want 3.1/CRITICAL", res.CVSSVersion, res.CVSSSeverity)
}
if res.CVSSVector != "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H" {
t.Errorf("vector = %q", res.CVSSVector)
}
if res.CWE != "CWE-502" {
t.Errorf("CWE = %q, want CWE-502 (first problemType)", res.CWE)
}
if !strings.Contains(res.Description, "Log4j") {
t.Errorf("description missing Log4j: %q", res.Description)
}
if res.Published == "" || res.Modified == "" {
t.Errorf("published/modified empty: %q / %q", res.Published, res.Modified)
}
}
func TestCVEListNotFound(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
client := NewCVEListClient(srv.Client(), srv.URL)
res, err := client.Fetch(context.Background(), "CVE-2099-99999")
if err != nil {
t.Fatalf("Fetch: %v", err)
}
if res.Found {
t.Error("Found = true on 404, want false")
}
}
func TestCVEListRecordURL(t *testing.T) {
client := NewCVEListClient(http.DefaultClient, "https://example.com/cves")
cases := map[string]string{
"CVE-2021-44228": "https://example.com/cves/2021/44xxx/CVE-2021-44228.json",
"CVE-2021-1234": "https://example.com/cves/2021/1xxx/CVE-2021-1234.json",
"CVE-2023-12345": "https://example.com/cves/2023/12xxx/CVE-2023-12345.json",
}
for id, want := range cases {
got, err := client.recordURL(id)
if err != nil {
t.Errorf("recordURL(%q): %v", id, err)
continue
}
if got != want {
t.Errorf("recordURL(%q) = %q, want %q", id, got, want)
}
}
if _, err := client.recordURL("not-a-cve"); err == nil {
t.Error("expected error for malformed id")
}
if _, err := client.recordURL("CVE-2021-12"); err == nil {
t.Error("expected error for a too-short cve number")
}
}
func serveJSON(t *testing.T, body string) *httptest.Server {
t.Helper()
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(body))
}))
t.Cleanup(srv.Close)
return srv
}
func TestCVEListCNAMetrics(t *testing.T) {
srv := serveJSON(t, `{"cveMetadata":{"datePublished":"2020-01-01T00:00:00Z","dateUpdated":"2020-02-01T00:00:00Z"},"containers":{"cna":{"descriptions":[{"lang":"en","value":"cna container path"}],"problemTypes":[{"descriptions":[{"lang":"en","cweId":"CWE-79"}]}],"metrics":[{"cvssV3_1":{"version":"3.1","baseScore":7.5,"baseSeverity":"HIGH","vectorString":"CVSS:3.1/AV:N/AC:L"}}]}}}`)
res, err := NewCVEListClient(srv.Client(), srv.URL).Fetch(context.Background(), "CVE-2020-0001")
if err != nil {
t.Fatalf("Fetch: %v", err)
}
if res.CVSSScore == nil || *res.CVSSScore != 7.5 || res.CVSSVersion != "3.1" || res.CVSSSeverity != "HIGH" {
t.Errorf("cna metrics = %v/%q/%q, want 7.5/3.1/HIGH", res.CVSSScore, res.CVSSVersion, res.CVSSSeverity)
}
if res.CWE != "CWE-79" {
t.Errorf("CWE = %q, want CWE-79", res.CWE)
}
}
func TestCVEListV2SeverityNotCritical(t *testing.T) {
srv := serveJSON(t, `{"cveMetadata":{"datePublished":"2005-01-01T00:00:00Z","dateUpdated":"2005-01-01T00:00:00Z"},"containers":{"cna":{"descriptions":[{"lang":"en","value":"v2 only"}],"metrics":[{"cvssV2_0":{"version":"2.0","baseScore":9.0,"vectorString":"AV:N/AC:L/Au:N/C:C/I:C/A:C"}}]}}}`)
res, err := NewCVEListClient(srv.Client(), srv.URL).Fetch(context.Background(), "CVE-2005-0001")
if err != nil {
t.Fatalf("Fetch: %v", err)
}
if res.CVSSVersion != "2.0" || res.CVSSScore == nil || *res.CVSSScore != 9.0 {
t.Errorf("v2 = %q/%v, want 2.0/9.0", res.CVSSVersion, res.CVSSScore)
}
if res.CVSSSeverity != "HIGH" {
t.Errorf("v2 severity = %q, want HIGH (CVSS v2 has no CRITICAL tier)", res.CVSSSeverity)
}
}
var _ CVESource = (*CVEListClient)(nil)
var _ CVESource = (*NVDClient)(nil)

83
PROJECTS/intermediate/security-news-scraper/internal/cve/epss.go Normal file
View File

@ -0,0 +1,83 @@
// ©AngelaMos | 2026
// epss.go
package cve
import (
"context"
"net/http"
"net/url"
"strconv"
"strings"
"golang.org/x/time/rate"
)
const (
EPSSEndpoint = "https://api.first.org/data/v1/epss"
epssBatchSize = 100
epssRate = 5
)
type EPSSScore struct {
EPSS float64
Percentile float64
}
type EPSSClient struct {
http *http.Client
baseURL string
limiter *rate.Limiter
}
func NewEPSSClient(client *http.Client, baseURL string) *EPSSClient {
return &EPSSClient{
http: client,
baseURL: baseURL,
limiter: rate.NewLimiter(rate.Limit(epssRate), 1),
}
}
func (c *EPSSClient) Fetch(ctx context.Context, cveIDs []string) (map[string]EPSSScore, error) {
out := make(map[string]EPSSScore, len(cveIDs))
for _, batch := range chunk(cveIDs, epssBatchSize) {
if err := c.limiter.Wait(ctx); err != nil {
return out, err
}
endpoint := c.baseURL + "?" + url.Values{"cve": {strings.Join(batch, ",")}}.Encode()
var raw epssRaw
if err := getJSON(ctx, c.http, endpoint, nil, &raw); err != nil {
return out, err
}
for _, d := range raw.Data {
epss, err1 := strconv.ParseFloat(d.EPSS, 64)
percentile, err2 := strconv.ParseFloat(d.Percentile, 64)
if err1 != nil || err2 != nil {
continue
}
out[d.CVE] = EPSSScore{EPSS: epss, Percentile: percentile}
}
}
return out, nil
}
func chunk(ids []string, size int) [][]string {
var out [][]string
for i := 0; i < len(ids); i += size {
end := i + size
if end > len(ids) {
end = len(ids)
}
out = append(out, ids[i:end])
}
return out
}
type epssRaw struct {
Data []struct {
CVE string `json:"cve"`
EPSS string `json:"epss"`
Percentile string `json:"percentile"`
} `json:"data"`
}

29
PROJECTS/intermediate/security-news-scraper/internal/cve/extract.go Normal file
View File

@ -0,0 +1,29 @@
// ©AngelaMos | 2026
// extract.go
package cve
import (
"regexp"
"sort"
"strings"
)
var pattern = regexp.MustCompile(`(?i)CVE-\d{4}-\d{4,7}`)
func Extract(texts ...string) []string {
seen := make(map[string]struct{})
var out []string
for _, text := range texts {
for _, match := range pattern.FindAllString(text, -1) {
id := strings.ToUpper(match)
if _, ok := seen[id]; ok {
continue
}
seen[id] = struct{}{}
out = append(out, id)
}
}
sort.Strings(out)
return out
}

53
PROJECTS/intermediate/security-news-scraper/internal/cve/http.go Normal file
View File

@ -0,0 +1,53 @@
// ©AngelaMos | 2026
// http.go
package cve
import (
"context"
"encoding/json"
"fmt"
"io"
"net/http"
)
const (
headerAccept = "Accept"
acceptJSON = "application/json"
maxJSONBytes = 32 << 20
)
type statusError struct {
code int
url string
}
func (e *statusError) Error() string {
return fmt.Sprintf("cve: GET %s: status %d", e.url, e.code)
}
func getJSON(ctx context.Context, client *http.Client, url string, header http.Header, out any) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
if err != nil {
return fmt.Errorf("cve: build request %s: %w", url, err)
}
req.Header.Set(headerAccept, acceptJSON)
for key, values := range header {
req.Header[key] = values
}
resp, err := client.Do(req)
if err != nil {
return fmt.Errorf("cve: GET %s: %w", url, err)
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
_, _ = io.Copy(io.Discard, io.LimitReader(resp.Body, maxJSONBytes))
return &statusError{code: resp.StatusCode, url: url}
}
if err := json.NewDecoder(io.LimitReader(resp.Body, maxJSONBytes)).Decode(out); err != nil {
return fmt.Errorf("cve: decode %s: %w", url, err)
}
return nil
}

65
PROJECTS/intermediate/security-news-scraper/internal/cve/kev.go Normal file
View File

@ -0,0 +1,65 @@
// ©AngelaMos | 2026
// kev.go
package cve
import (
"context"
"net/http"
)
const (
KEVEndpoint = "https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json"
kevRansomwareKnown = "Known"
)
type KEVEntry struct {
DateAdded string
Ransomware bool
}
type KEVCatalog struct {
Version string
Released string
Entries map[string]KEVEntry
}
func (c KEVCatalog) Lookup(cveID string) (KEVEntry, bool) {
e, ok := c.Entries[cveID]
return e, ok
}
type KEVClient struct {
http *http.Client
baseURL string
}
func NewKEVClient(client *http.Client, baseURL string) *KEVClient {
return &KEVClient{http: client, baseURL: baseURL}
}
func (c *KEVClient) Fetch(ctx context.Context) (KEVCatalog, error) {
var raw kevRaw
if err := getJSON(ctx, c.http, c.baseURL, nil, &raw); err != nil {
return KEVCatalog{}, err
}
entries := make(map[string]KEVEntry, len(raw.Vulnerabilities))
for _, v := range raw.Vulnerabilities {
entries[v.CveID] = KEVEntry{
DateAdded: v.DateAdded,
Ransomware: v.KnownRansomwareCampaignUse == kevRansomwareKnown,
}
}
return KEVCatalog{Version: raw.CatalogVersion, Released: raw.DateReleased, Entries: entries}, nil
}
type kevRaw struct {
CatalogVersion string `json:"catalogVersion"`
DateReleased string `json:"dateReleased"`
Vulnerabilities []struct {
CveID string `json:"cveID"`
DateAdded string `json:"dateAdded"`
KnownRansomwareCampaignUse string `json:"knownRansomwareCampaignUse"`
} `json:"vulnerabilities"`
}

184
PROJECTS/intermediate/security-news-scraper/internal/cve/nvd.go Normal file
View File

@ -0,0 +1,184 @@
// ©AngelaMos | 2026
// nvd.go
package cve
import (
"context"
"errors"
"net/http"
"net/url"
"time"
"golang.org/x/time/rate"
)
const (
NVDEndpoint = "https://services.nvd.nist.gov/rest/json/cves/2.0"
nvdAPIKeyHeader = "apiKey"
nvdRateNoKey = 5.0 / 30.0
nvdRateWithKey = 50.0 / 30.0
nvdMaxRetries = 3
nvdBackoffBase = 2 * time.Second
langEnglish = "en"
)
type NVDClient struct {
http *http.Client
baseURL string
apiKey string
limiter *rate.Limiter
backoffBase time.Duration
}
func NewNVDClient(client *http.Client, baseURL, apiKey string) *NVDClient {
limit := nvdRateNoKey
if apiKey != "" {
limit = nvdRateWithKey
}
return &NVDClient{
http: client,
baseURL: baseURL,
apiKey: apiKey,
limiter: rate.NewLimiter(rate.Limit(limit), 1),
backoffBase: nvdBackoffBase,
}
}
func (c *NVDClient) Fetch(ctx context.Context, cveID string) (CoreResult, error) {
endpoint := c.baseURL + "?" + url.Values{"cveId": {cveID}}.Encode()
header := http.Header{}
if c.apiKey != "" {
header[nvdAPIKeyHeader] = []string{c.apiKey}
}
var env nvdEnvelope
for attempt := 0; attempt <= nvdMaxRetries; attempt++ {
if attempt > 0 {
if err := sleep(ctx, c.backoffBase*time.Duration(1<<(attempt-1))); err != nil {
return CoreResult{}, err
}
}
if err := c.limiter.Wait(ctx); err != nil {
return CoreResult{}, err
}
err := getJSON(ctx, c.http, endpoint, header, &env)
if err == nil {
break
}
if !retriable(err) || attempt == nvdMaxRetries {
return CoreResult{}, err
}
}
if env.TotalResults == 0 || len(env.Vulnerabilities) == 0 {
return CoreResult{Found: false}, nil
}
return env.Vulnerabilities[0].CVE.toResult(), nil
}
func retriable(err error) bool {
var se *statusError
if errors.As(err, &se) {
return se.code == http.StatusTooManyRequests || se.code >= http.StatusInternalServerError
}
return true
}
type nvdEnvelope struct {
TotalResults int `json:"totalResults"`
Vulnerabilities []struct {
CVE nvdCVE `json:"cve"`
} `json:"vulnerabilities"`
}
type nvdCVE struct {
ID string `json:"id"`
Published string `json:"published"`
LastModified string `json:"lastModified"`
Descriptions []nvdLangValue `json:"descriptions"`
Weaknesses []struct {
Description []nvdLangValue `json:"description"`
} `json:"weaknesses"`
Metrics nvdMetrics `json:"metrics"`
}
type nvdLangValue struct {
Lang string `json:"lang"`
Value string `json:"value"`
}
type nvdMetrics struct {
V40 []nvdMetric `json:"cvssMetricV40"`
V31 []nvdMetric `json:"cvssMetricV31"`
V30 []nvdMetric `json:"cvssMetricV30"`
V2 []nvdMetric `json:"cvssMetricV2"`
}
type nvdMetric struct {
BaseSeverity string `json:"baseSeverity"`
CVSSData nvdCVSSData `json:"cvssData"`
}
type nvdCVSSData struct {
Version string `json:"version"`
VectorString string `json:"vectorString"`
BaseScore float64 `json:"baseScore"`
BaseSeverity string `json:"baseSeverity"`
}
func (v nvdCVE) toResult() CoreResult {
res := CoreResult{
Found: true,
Description: english(v.Descriptions),
Published: v.Published,
Modified: v.LastModified,
}
for _, w := range v.Weaknesses {
if cwe := english(w.Description); cwe != "" {
res.CWE = cwe
break
}
}
if m, ok := v.Metrics.selected(); ok {
score := m.CVSSData.BaseScore
res.CVSSScore = &score
res.CVSSVersion = m.CVSSData.Version
res.CVSSVector = m.CVSSData.VectorString
res.CVSSSeverity = m.CVSSData.BaseSeverity
if res.CVSSSeverity == "" {
res.CVSSSeverity = m.BaseSeverity
}
}
return res
}
func (m nvdMetrics) selected() (nvdMetric, bool) {
for _, tier := range [][]nvdMetric{m.V40, m.V31, m.V30, m.V2} {
if len(tier) > 0 {
return tier[0], true
}
}
return nvdMetric{}, false
}
func english(values []nvdLangValue) string {
for _, lv := range values {
if lv.Lang == langEnglish {
return lv.Value
}
}
return ""
}
func sleep(ctx context.Context, d time.Duration) error {
timer := time.NewTimer(d)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
return nil
}
}

105
PROJECTS/intermediate/security-news-scraper/internal/enrich/enrich.go Normal file
View File

@ -0,0 +1,105 @@
// ©AngelaMos | 2026
// enrich.go
package enrich
import (
"context"
"time"
"github.com/CarterPerez-dev/nadezhda/internal/cve"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const secondsPerHour = 3600
type Clients struct {
Core cve.CVESource
KEV *cve.KEVClient
EPSS *cve.EPSSClient
}
type Stats struct {
Total int
Enriched int
NotFound int
KEVHits int
Errors int
}
func Run(ctx context.Context, st *store.Store, clients Clients, now time.Time, positiveTTLHours, negativeTTLHours int) (Stats, error) {
positiveTTL := int64(positiveTTLHours) * secondsPerHour
negativeTTL := int64(negativeTTLHours) * secondsPerHour
ids, err := st.CVEsNeedingEnrichment(now.Unix(), positiveTTL, negativeTTL)
if err != nil {
return Stats{}, err
}
if len(ids) == 0 {
return Stats{}, nil
}
catalog, err := clients.KEV.Fetch(ctx)
if err != nil {
return Stats{}, err
}
epssScores, _ := clients.EPSS.Fetch(ctx, ids)
if epssScores == nil {
epssScores = map[string]cve.EPSSScore{}
}
stats := Stats{Total: len(ids)}
for _, id := range ids {
coreRes, err := clients.Core.Fetch(ctx, id)
if err != nil {
if ctx.Err() != nil {
return stats, ctx.Err()
}
stats.Errors++
continue
}
rec := store.CVE{ID: id, EnrichedAt: now.Unix()}
if coreRes.Found {
rec.EnrichStatus = store.EnrichStatusOK
rec.Description = coreRes.Description
rec.CVSSScore = coreRes.CVSSScore
rec.CVSSVersion = coreRes.CVSSVersion
rec.CVSSSeverity = coreRes.CVSSSeverity
rec.CVSSVector = coreRes.CVSSVector
rec.CWE = coreRes.CWE
rec.NVDPublished = coreRes.Published
rec.NVDModified = coreRes.Modified
} else {
rec.EnrichStatus = store.EnrichStatusNotFound
}
if entry, ok := catalog.Lookup(id); ok {
rec.IsKEV = true
rec.KEVDateAdded = entry.DateAdded
rec.KEVRansomware = entry.Ransomware
}
if score, ok := epssScores[id]; ok {
epssVal := score.EPSS
percentileVal := score.Percentile
rec.EPSS = &epssVal
rec.EPSSPercentile = &percentileVal
}
if err := st.UpdateCVEEnrichment(rec); err != nil {
stats.Errors++
continue
}
if coreRes.Found {
stats.Enriched++
} else {
stats.NotFound++
}
if rec.IsKEV {
stats.KEVHits++
}
}
return stats, nil
}

121
PROJECTS/intermediate/security-news-scraper/internal/enrich/enrich_test.go Normal file
View File

@ -0,0 +1,121 @@
// ©AngelaMos | 2026
// enrich_test.go
package enrich
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"time"
"github.com/CarterPerez-dev/nadezhda/internal/cve"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
func fileServer(t *testing.T, path string) *httptest.Server {
t.Helper()
body, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read %s: %v", path, err)
}
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write(body)
}))
t.Cleanup(srv.Close)
return srv
}
func newStore(t *testing.T) *store.Store {
t.Helper()
st, err := store.Open(filepath.Join(t.TempDir(), "enrich.db"))
if err != nil {
t.Fatalf("open store: %v", err)
}
t.Cleanup(func() { _ = st.Close() })
return st
}
func TestRunEnrichesLog4Shell(t *testing.T) {
nvd := fileServer(t, "../../testdata/nvd/CVE-2021-44228.json")
kev := fileServer(t, "../../testdata/kev/kev-sample.json")
epss := fileServer(t, "../../testdata/epss/CVE-2021-44228.json")
st := newStore(t)
if err := st.UpsertCVEStub("CVE-2021-44228"); err != nil {
t.Fatal(err)
}
clients := Clients{
Core: cve.NewNVDClient(nvd.Client(), nvd.URL, ""),
KEV: cve.NewKEVClient(kev.Client(), kev.URL),
EPSS: cve.NewEPSSClient(epss.Client(), epss.URL),
}
stats, err := Run(context.Background(), st, clients, time.Unix(1_800_000_000, 0), 24, 3)
if err != nil {
t.Fatalf("Run: %v", err)
}
if stats.Enriched != 1 || stats.KEVHits != 1 {
t.Fatalf("stats = %+v, want Enriched:1 KEVHits:1", stats)
}
c, err := st.GetCVE("CVE-2021-44228")
if err != nil {
t.Fatal(err)
}
if c.CVSSScore == nil || *c.CVSSScore != 10.0 || c.CVSSSeverity != "CRITICAL" {
t.Errorf("cvss = %v/%q, want 10.0/CRITICAL", c.CVSSScore, c.CVSSSeverity)
}
if !c.IsKEV || !c.KEVRansomware {
t.Errorf("expected KEV + ransomware, got is_kev=%v ransomware=%v", c.IsKEV, c.KEVRansomware)
}
if c.EPSS == nil || *c.EPSS < 0.9 {
t.Errorf("epss = %v, want ~0.99999", c.EPSS)
}
if c.EnrichStatus != store.EnrichStatusOK {
t.Errorf("status = %q, want ok", c.EnrichStatus)
}
}
func TestRunSkipsFreshAndMarksNotFound(t *testing.T) {
nvd := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"totalResults":0,"vulnerabilities":[]}`))
}))
t.Cleanup(nvd.Close)
kev := fileServer(t, "../../testdata/kev/kev-sample.json")
epss := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write([]byte(`{"data":[]}`))
}))
t.Cleanup(epss.Close)
st := newStore(t)
if err := st.UpsertCVEStub("CVE-2099-99999"); err != nil {
t.Fatal(err)
}
clients := Clients{
Core: cve.NewNVDClient(nvd.Client(), nvd.URL, ""),
KEV: cve.NewKEVClient(kev.Client(), kev.URL),
EPSS: cve.NewEPSSClient(epss.Client(), epss.URL),
}
now := time.Unix(1_800_000_000, 0)
stats, err := Run(context.Background(), st, clients, now, 24, 3)
if err != nil {
t.Fatalf("Run: %v", err)
}
if stats.NotFound != 1 {
t.Errorf("NotFound = %d, want 1", stats.NotFound)
}
fresh, err := Run(context.Background(), st, clients, now, 24, 3)
if err != nil {
t.Fatalf("second Run: %v", err)
}
if fresh.Total != 0 {
t.Errorf("second run Total = %d, want 0 (not_found within negative TTL is skipped)", fresh.Total)
}
}

172
PROJECTS/intermediate/security-news-scraper/internal/export/export.go Normal file
View File

@ -0,0 +1,172 @@
// ©AngelaMos | 2026
// export.go
package export
import (
"encoding/json"
"fmt"
"sort"
"strings"
"time"
"github.com/CarterPerez-dev/nadezhda/internal/rank"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const (
dateLayout = "2006-01-02"
noDate = "----------"
scoreRounder = 10000.0
digestTitle = "# Nadezhda Digest"
)
type CVEEntry struct {
ID string `json:"id"`
CVSS *float64 `json:"cvss,omitempty"`
EPSS *float64 `json:"epss,omitempty"`
KEV bool `json:"kev"`
}
type Entry struct {
Rank int `json:"rank"`
Score float64 `json:"score"`
Headline string `json:"headline"`
URL string `json:"url"`
Outlets []string `json:"outlets"`
FirstSeen string `json:"first_seen"`
LastSeen string `json:"last_seen"`
CVEs []CVEEntry `json:"cves,omitempty"`
}
type Digest struct {
Count int `json:"count"`
Entries []Entry `json:"entries"`
}
func Build(scored []rank.Scored, top int) Digest {
limit := len(scored)
if top > 0 && top < limit {
limit = top
}
entries := make([]Entry, 0, limit)
for i := 0; i < limit; i++ {
s := scored[i]
h := headline(s.Cluster.Articles)
entries = append(entries, Entry{
Rank: i + 1,
Score: roundScore(s.Score),
Headline: h.Title,
URL: h.CanonicalURL,
Outlets: outlets(s.Cluster.Articles),
FirstSeen: formatDate(s.Cluster.FirstSeen),
LastSeen: formatDate(s.Cluster.LastSeen),
CVEs: cveEntries(s.Cluster.CVEs),
})
}
return Digest{Count: len(entries), Entries: entries}
}
func JSON(scored []rank.Scored, top int) (string, error) {
b, err := json.MarshalIndent(Build(scored, top), "", " ")
if err != nil {
return "", fmt.Errorf("export json: %w", err)
}
return string(b), nil
}
func Markdown(scored []rank.Scored, top int) string {
d := Build(scored, top)
var b strings.Builder
b.WriteString(digestTitle + "\n\n")
for _, e := range d.Entries {
fmt.Fprintf(&b, "## %d. %s\n", e.Rank, oneLine(e.Headline))
fmt.Fprintf(&b, "score %.2f | %d outlet(s): %s\n", e.Score, len(e.Outlets), strings.Join(e.Outlets, ", "))
for _, c := range e.CVEs {
fmt.Fprintf(&b, "- %s\n", cveLine(c))
}
fmt.Fprintf(&b, "%s\n\n", e.URL)
}
return b.String()
}
func headline(articles []store.DigestArticle) store.DigestArticle {
var best store.DigestArticle
for i, a := range articles {
if i == 0 || a.PublishedAt > best.PublishedAt ||
(a.PublishedAt == best.PublishedAt && a.ID < best.ID) {
best = a
}
}
return best
}
func outlets(articles []store.DigestArticle) []string {
seen := make(map[string]struct{}, len(articles))
var out []string
for _, a := range articles {
if _, ok := seen[a.SourceName]; ok {
continue
}
seen[a.SourceName] = struct{}{}
out = append(out, a.SourceName)
}
sort.Strings(out)
return out
}
func cveEntries(cves []store.DigestCVE) []CVEEntry {
out := make([]CVEEntry, 0, len(cves))
for _, v := range cves {
out = append(out, CVEEntry{ID: v.ID, CVSS: v.CVSSScore, EPSS: v.EPSS, KEV: v.IsKEV})
}
sort.SliceStable(out, func(a, b int) bool {
if out[a].KEV != out[b].KEV {
return out[a].KEV
}
if cvssOf(out[a]) != cvssOf(out[b]) {
return cvssOf(out[a]) > cvssOf(out[b])
}
return out[a].ID < out[b].ID
})
return out
}
func cvssOf(c CVEEntry) float64 {
if c.CVSS == nil {
return -1
}
return *c.CVSS
}
func cveLine(c CVEEntry) string {
var parts []string
if c.KEV {
parts = append(parts, "KEV")
}
if c.CVSS != nil {
parts = append(parts, fmt.Sprintf("CVSS %.1f", *c.CVSS))
}
if c.EPSS != nil {
parts = append(parts, fmt.Sprintf("EPSS %.2f", *c.EPSS))
}
if len(parts) == 0 {
return c.ID
}
return fmt.Sprintf("%s (%s)", c.ID, strings.Join(parts, ", "))
}
func oneLine(s string) string {
return strings.Join(strings.Fields(s), " ")
}
func roundScore(v float64) float64 {
return float64(int64(v*scoreRounder+0.5)) / scoreRounder
}
func formatDate(unix int64) string {
if unix == 0 {
return noDate
}
return time.Unix(unix, 0).UTC().Format(dateLayout)
}

104
PROJECTS/intermediate/security-news-scraper/internal/export/export_test.go Normal file
View File

@ -0,0 +1,104 @@
// ©AngelaMos | 2026
// export_test.go
package export
import (
"encoding/json"
"strings"
"testing"
"github.com/CarterPerez-dev/nadezhda/internal/rank"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
func f(v float64) *float64 { return &v }
func sample() []rank.Scored {
return []rank.Scored{
{
Score: 0.9412,
Cluster: store.DigestCluster{
Size: 3, FirstSeen: 1751600000, LastSeen: 1751700000,
Articles: []store.DigestArticle{
{ID: 1, SourceName: "thehackernews", Title: "Old framing", CanonicalURL: "https://thn/1", PublishedAt: 100},
{ID: 2, SourceName: "bleepingcomputer", Title: "Freshest framing", CanonicalURL: "https://bc/2", PublishedAt: 300},
{ID: 3, SourceName: "bleepingcomputer", Title: "Mid framing", CanonicalURL: "https://bc/3", PublishedAt: 200},
},
CVEs: []store.DigestCVE{
{ID: "CVE-2026-2", CVSSScore: f(7.5), IsKEV: false},
{ID: "CVE-2026-1", CVSSScore: f(9.8), EPSS: f(0.91), IsKEV: true},
},
},
},
{
Score: 0.10,
Cluster: store.DigestCluster{
Size: 1, FirstSeen: 1751000000, LastSeen: 1751000000,
Articles: []store.DigestArticle{{ID: 9, SourceName: "krebs", Title: "Quiet story", CanonicalURL: "https://k/9", PublishedAt: 50}},
},
},
}
}
func TestBuildHeadlineAndOutlets(t *testing.T) {
d := Build(sample(), 0)
if d.Count != 2 {
t.Fatalf("count = %d, want 2", d.Count)
}
e := d.Entries[0]
if e.Rank != 1 || e.Score != 0.9412 {
t.Errorf("rank/score = %d/%v", e.Rank, e.Score)
}
if e.Headline != "Freshest framing" {
t.Errorf("headline = %q, want the freshest member", e.Headline)
}
if e.URL != "https://bc/2" {
t.Errorf("url = %q, want the freshest member's", e.URL)
}
if len(e.Outlets) != 2 || e.Outlets[0] != "bleepingcomputer" || e.Outlets[1] != "thehackernews" {
t.Errorf("outlets = %v, want distinct + sorted", e.Outlets)
}
if e.CVEs[0].ID != "CVE-2026-1" {
t.Errorf("first cve = %q, want the KEV one first", e.CVEs[0].ID)
}
}
func TestBuildTopLimit(t *testing.T) {
d := Build(sample(), 1)
if d.Count != 1 {
t.Errorf("top=1 should yield 1 entry, got %d", d.Count)
}
}
func TestJSONValidAndStable(t *testing.T) {
out, err := JSON(sample(), 0)
if err != nil {
t.Fatal(err)
}
var back Digest
if err := json.Unmarshal([]byte(out), &back); err != nil {
t.Fatalf("output is not valid JSON: %v", err)
}
if back.Entries[0].Headline != "Freshest framing" {
t.Errorf("round-trip headline = %q", back.Entries[0].Headline)
}
if !back.Entries[0].CVEs[0].KEV {
t.Error("first cve should be the KEV one")
}
}
func TestMarkdownContainsSignals(t *testing.T) {
md := Markdown(sample(), 0)
for _, want := range []string{
"# Nadezhda Digest",
"## 1. Freshest framing",
"2 outlet(s): bleepingcomputer, thehackernews",
"CVE-2026-1 (KEV, CVSS 9.8, EPSS 0.91)",
"https://bc/2",
} {
if !strings.Contains(md, want) {
t.Errorf("markdown missing %q\n---\n%s", want, md)
}
}
}

235
PROJECTS/intermediate/security-news-scraper/internal/fetch/fetch.go Normal file
View File

@ -0,0 +1,235 @@
// ©AngelaMos | 2026
// fetch.go
package fetch
import (
"context"
"errors"
"fmt"
"io"
"net/http"
"net/url"
"strconv"
"sync"
"time"
"golang.org/x/time/rate"
)
const (
headerUserAgent = "User-Agent"
headerAccept = "Accept"
headerIfNoneMatch = "If-None-Match"
headerIfModifiedSince = "If-Modified-Since"
headerETag = "ETag"
headerLastModified = "Last-Modified"
headerRetryAfter = "Retry-After"
acceptFeed = "application/rss+xml, application/atom+xml, application/xml;q=0.9, */*;q=0.8"
defaultBackoffBase = 500 * time.Millisecond
maxBodyBytes = 16 << 20
maxRetryAfter = 60 * time.Second
)
type Options struct {
UserAgent string
PerHostRate float64
PerHostBurst int
Timeout time.Duration
MaxRetries int
}
type Request struct {
URL string
ETag string
LastModified string
}
type Result struct {
Status int
Body []byte
ETag string
LastModified string
NotModified bool
}
type Client struct {
http *http.Client
ua string
rate rate.Limit
burst int
maxRetries int
backoffBase time.Duration
mu sync.Mutex
limiters map[string]*rate.Limiter
robots *robotsCache
}
func New(opts Options) *Client {
c := &Client{
http: &http.Client{Timeout: opts.Timeout},
ua: opts.UserAgent,
rate: rate.Limit(opts.PerHostRate),
burst: opts.PerHostBurst,
maxRetries: opts.MaxRetries,
backoffBase: defaultBackoffBase,
limiters: make(map[string]*rate.Limiter),
}
c.robots = newRobotsCache(c)
return c
}
func (c *Client) limiterFor(host string) *rate.Limiter {
c.mu.Lock()
defer c.mu.Unlock()
l, ok := c.limiters[host]
if !ok {
l = rate.NewLimiter(c.rate, c.burst)
c.limiters[host] = l
}
return l
}
func (c *Client) Fetch(ctx context.Context, req Request) (Result, error) {
u, err := url.Parse(req.URL)
if err != nil {
return Result{}, fmt.Errorf("fetch: parse url %q: %w", req.URL, err)
}
var lastErr error
var override time.Duration
for attempt := 0; attempt <= c.maxRetries; attempt++ {
if attempt > 0 {
delay := c.backoff(attempt)
if override > 0 {
delay = override
}
if err := sleep(ctx, delay); err != nil {
return Result{}, err
}
}
if err := c.limiterFor(u.Host).Wait(ctx); err != nil {
return Result{}, err
}
res, retryAfter, retry, err := c.do(ctx, req)
if err == nil {
return res, nil
}
lastErr = err
override = retryAfter
if !retry {
return Result{}, err
}
}
return Result{}, fmt.Errorf("fetch %s: exhausted %d retries: %w", req.URL, c.maxRetries, lastErr)
}
func (c *Client) Allowed(ctx context.Context, rawURL string) (bool, error) {
u, err := url.Parse(rawURL)
if err != nil {
return false, fmt.Errorf("fetch: parse url %q: %w", rawURL, err)
}
return c.robots.allowed(ctx, u)
}
func (c *Client) do(ctx context.Context, req Request) (Result, time.Duration, bool, error) {
httpReq, err := http.NewRequestWithContext(ctx, http.MethodGet, req.URL, nil)
if err != nil {
return Result{}, 0, false, fmt.Errorf("fetch %s: build request: %w", req.URL, err)
}
httpReq.Header.Set(headerUserAgent, c.ua)
httpReq.Header.Set(headerAccept, acceptFeed)
if req.ETag != "" {
httpReq.Header.Set(headerIfNoneMatch, req.ETag)
}
if req.LastModified != "" {
httpReq.Header.Set(headerIfModifiedSince, req.LastModified)
}
resp, err := c.http.Do(httpReq)
if err != nil {
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
return Result{}, 0, false, fmt.Errorf("fetch %s: %w", req.URL, err)
}
return Result{}, 0, true, fmt.Errorf("fetch %s: %w", req.URL, err)
}
defer resp.Body.Close()
switch {
case resp.StatusCode == http.StatusNotModified:
return Result{
Status: resp.StatusCode,
NotModified: true,
ETag: req.ETag,
LastModified: req.LastModified,
}, 0, false, nil
case resp.StatusCode == http.StatusOK:
body, err := io.ReadAll(io.LimitReader(resp.Body, maxBodyBytes))
if err != nil {
return Result{}, 0, true, fmt.Errorf("fetch %s: read body: %w", req.URL, err)
}
return Result{
Status: resp.StatusCode,
Body: body,
ETag: resp.Header.Get(headerETag),
LastModified: resp.Header.Get(headerLastModified),
}, 0, false, nil
case resp.StatusCode == http.StatusTooManyRequests || resp.StatusCode >= http.StatusInternalServerError:
drain(resp.Body)
return Result{}, retryAfter(resp), true, fmt.Errorf("fetch %s: server status %d", req.URL, resp.StatusCode)
default:
drain(resp.Body)
return Result{}, 0, false, fmt.Errorf("fetch %s: status %d", req.URL, resp.StatusCode)
}
}
func drain(body io.Reader) {
_, _ = io.Copy(io.Discard, io.LimitReader(body, maxBodyBytes))
}
func retryAfter(resp *http.Response) time.Duration {
v := resp.Header.Get(headerRetryAfter)
if v == "" {
return 0
}
if secs, err := strconv.Atoi(v); err == nil {
d := time.Duration(secs) * time.Second
if d > maxRetryAfter {
return maxRetryAfter
}
if d < 0 {
return 0
}
return d
}
if t, err := http.ParseTime(v); err == nil {
d := time.Until(t)
if d <= 0 {
return 0
}
if d > maxRetryAfter {
return maxRetryAfter
}
return d
}
return 0
}
func (c *Client) backoff(attempt int) time.Duration {
return c.backoffBase * time.Duration(1<<(attempt-1))
}
func sleep(ctx context.Context, d time.Duration) error {
timer := time.NewTimer(d)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
return nil
}
}

244
PROJECTS/intermediate/security-news-scraper/internal/fetch/fetch_test.go Normal file
View File

@ -0,0 +1,244 @@
// ©AngelaMos | 2026
// fetch_test.go
package fetch
import (
"context"
"net/http"
"net/http/httptest"
"sync/atomic"
"testing"
"time"
)
func testClient() *Client {
c := New(Options{
UserAgent: "nadezhda-test/1.0",
PerHostRate: 1e6,
PerHostBurst: 1,
Timeout: 5 * time.Second,
MaxRetries: 3,
})
c.backoffBase = time.Millisecond
return c
}
func TestFetchOKCapturesValidators(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set(headerETag, `"v1"`)
w.Header().Set(headerLastModified, "Wed, 01 Jul 2026 00:00:00 GMT")
_, _ = w.Write([]byte("<rss></rss>"))
}))
defer srv.Close()
res, err := testClient().Fetch(context.Background(), Request{URL: srv.URL + "/feed"})
if err != nil {
t.Fatalf("Fetch: %v", err)
}
if res.Status != http.StatusOK {
t.Errorf("status = %d, want 200", res.Status)
}
if string(res.Body) != "<rss></rss>" {
t.Errorf("body = %q", res.Body)
}
if res.ETag != `"v1"` {
t.Errorf("etag = %q, want \"v1\"", res.ETag)
}
if res.LastModified == "" {
t.Error("last-modified not captured")
}
}
func TestFetchConditionalGET(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get(headerIfNoneMatch) == `"v1"` {
w.WriteHeader(http.StatusNotModified)
return
}
w.Header().Set(headerETag, `"v1"`)
_, _ = w.Write([]byte("body"))
}))
defer srv.Close()
c := testClient()
first, err := c.Fetch(context.Background(), Request{URL: srv.URL + "/feed"})
if err != nil || first.Status != http.StatusOK {
t.Fatalf("first fetch: status=%d err=%v", first.Status, err)
}
second, err := c.Fetch(context.Background(), Request{URL: srv.URL + "/feed", ETag: first.ETag})
if err != nil {
t.Fatalf("second fetch: %v", err)
}
if !second.NotModified {
t.Error("expected NotModified on matching ETag")
}
if second.ETag != `"v1"` {
t.Errorf("304 should retain etag, got %q", second.ETag)
}
}
func TestFetchRetriesServerError(t *testing.T) {
var calls atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if calls.Add(1) == 1 {
w.WriteHeader(http.StatusInternalServerError)
return
}
_, _ = w.Write([]byte("ok"))
}))
defer srv.Close()
res, err := testClient().Fetch(context.Background(), Request{URL: srv.URL + "/feed"})
if err != nil {
t.Fatalf("Fetch: %v", err)
}
if string(res.Body) != "ok" {
t.Errorf("body = %q, want ok", res.Body)
}
if calls.Load() != 2 {
t.Errorf("calls = %d, want 2 (one 500, one 200)", calls.Load())
}
}
func TestFetchDoesNotRetryClientError(t *testing.T) {
var calls atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls.Add(1)
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
_, err := testClient().Fetch(context.Background(), Request{URL: srv.URL + "/feed"})
if err == nil {
t.Fatal("expected error on 404")
}
if calls.Load() != 1 {
t.Errorf("calls = %d, want 1 (no retry on 4xx)", calls.Load())
}
}
func TestFetch429IsRetried(t *testing.T) {
var calls atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if calls.Add(1) == 1 {
w.WriteHeader(http.StatusTooManyRequests)
return
}
_, _ = w.Write([]byte("ok"))
}))
defer srv.Close()
res, err := testClient().Fetch(context.Background(), Request{URL: srv.URL + "/feed"})
if err != nil {
t.Fatalf("Fetch: %v", err)
}
if string(res.Body) != "ok" || calls.Load() != 2 {
t.Errorf("body=%q calls=%d, want ok/2", res.Body, calls.Load())
}
}
func TestFetchTimeoutNotRetried(t *testing.T) {
var calls atomic.Int32
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls.Add(1)
time.Sleep(100 * time.Millisecond)
}))
defer srv.Close()
c := New(Options{
UserAgent: "nadezhda-test/1.0", PerHostRate: 1e6, PerHostBurst: 1,
Timeout: 15 * time.Millisecond, MaxRetries: 3,
})
c.backoffBase = time.Millisecond
_, err := c.Fetch(context.Background(), Request{URL: srv.URL + "/feed"})
if err == nil {
t.Fatal("expected timeout error")
}
if calls.Load() != 1 {
t.Errorf("calls = %d, want 1 (timeout must not be retried)", calls.Load())
}
}
func TestRetryAfterParsing(t *testing.T) {
cases := []struct {
header string
want time.Duration
}{
{"", 0},
{"5", 5 * time.Second},
{"0", 0},
{"-3", 0},
{"9999", maxRetryAfter},
{"garbage", 0},
}
for _, tc := range cases {
resp := &http.Response{Header: http.Header{}}
if tc.header != "" {
resp.Header.Set(headerRetryAfter, tc.header)
}
if got := retryAfter(resp); got != tc.want {
t.Errorf("retryAfter(%q) = %s, want %s", tc.header, got, tc.want)
}
}
}
func TestFeedFetchIgnoresRobots(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == robotsPath {
t.Error("Fetch must not request robots.txt on the feed path")
_, _ = w.Write([]byte("User-agent: *\nDisallow: /\n"))
return
}
_, _ = w.Write([]byte("feed body"))
}))
defer srv.Close()
res, err := testClient().Fetch(context.Background(), Request{URL: srv.URL + "/feed"})
if err != nil {
t.Fatalf("Fetch: %v", err)
}
if string(res.Body) != "feed body" {
t.Errorf("body = %q, want feed body", res.Body)
}
}
func TestAllowedRespectsDisallow(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == robotsPath {
_, _ = w.Write([]byte("User-agent: *\nDisallow: /article\n"))
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
ok, err := testClient().Allowed(context.Background(), srv.URL+"/article/123")
if err != nil {
t.Fatalf("Allowed: %v", err)
}
if ok {
t.Error("expected disallowed for /article path")
}
}
func TestAllowedPermitsOtherPaths(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == robotsPath {
_, _ = w.Write([]byte("User-agent: *\nDisallow: /admin\n"))
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
ok, err := testClient().Allowed(context.Background(), srv.URL+"/article/123")
if err != nil {
t.Fatalf("Allowed: %v", err)
}
if !ok {
t.Error("expected allowed for non-admin path")
}
}

88
PROJECTS/intermediate/security-news-scraper/internal/fetch/robots.go Normal file
View File

@ -0,0 +1,88 @@
// ©AngelaMos | 2026
// robots.go
package fetch
import (
"context"
"io"
"net/http"
"net/url"
"sync"
"github.com/temoto/robotstxt"
)
const (
robotsPath = "/robots.txt"
maxRobotsBytes = 512 << 10
robotsRootPath = "/"
)
type robotsEntry struct {
once sync.Once
data *robotstxt.RobotsData
}
type robotsCache struct {
client *Client
mu sync.Mutex
entries map[string]*robotsEntry
}
func newRobotsCache(client *Client) *robotsCache {
return &robotsCache{
client: client,
entries: make(map[string]*robotsEntry),
}
}
func (rc *robotsCache) allowed(ctx context.Context, u *url.URL) (bool, error) {
rc.mu.Lock()
e, ok := rc.entries[u.Host]
if !ok {
e = &robotsEntry{}
rc.entries[u.Host] = e
}
rc.mu.Unlock()
e.once.Do(func() {
data, err := rc.load(ctx, u)
if err != nil {
data, _ = robotstxt.FromStatusAndBytes(http.StatusOK, nil)
}
e.data = data
})
path := u.EscapedPath()
if path == "" {
path = robotsRootPath
}
return e.data.FindGroup(rc.client.ua).Test(path), nil
}
func (rc *robotsCache) load(ctx context.Context, u *url.URL) (*robotstxt.RobotsData, error) {
robotsURL := u.Scheme + "://" + u.Host + robotsPath
if err := rc.client.limiterFor(u.Host).Wait(ctx); err != nil {
return nil, err
}
req, err := http.NewRequestWithContext(ctx, http.MethodGet, robotsURL, nil)
if err != nil {
return nil, err
}
req.Header.Set(headerUserAgent, rc.client.ua)
resp, err := rc.client.http.Do(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
body, err := io.ReadAll(io.LimitReader(resp.Body, maxRobotsBytes))
if err != nil {
return nil, err
}
return robotstxt.FromStatusAndBytes(resp.StatusCode, body)
}

182
PROJECTS/intermediate/security-news-scraper/internal/ingest/ingest.go Normal file
View File

@ -0,0 +1,182 @@
// ©AngelaMos | 2026
// ingest.go
package ingest
import (
"bytes"
"context"
"errors"
"time"
"golang.org/x/sync/errgroup"
"github.com/CarterPerez-dev/nadezhda/internal/config"
"github.com/CarterPerez-dev/nadezhda/internal/cve"
"github.com/CarterPerez-dev/nadezhda/internal/fetch"
"github.com/CarterPerez-dev/nadezhda/internal/normalize"
"github.com/CarterPerez-dev/nadezhda/internal/parse"
"github.com/CarterPerez-dev/nadezhda/internal/source"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
type SourceResult struct {
Name string
Parsed int
New int
Duplicates int
CVEs int
ItemErrors int
NotModified bool
Err error
}
type Summary struct {
Results []SourceResult
}
func (s Summary) Totals() (newArticles, duplicates, failed int) {
for _, r := range s.Results {
newArticles += r.New
duplicates += r.Duplicates
if r.Err != nil {
failed++
}
}
return newArticles, duplicates, failed
}
func Run(ctx context.Context, fc *fetch.Client, st *store.Store, cfg config.Config, targets []source.Source, now time.Time) (Summary, error) {
results := make([]SourceResult, len(targets))
workers := cfg.Fetch.Workers
if workers < 1 {
workers = 1
}
g, gctx := errgroup.WithContext(ctx)
g.SetLimit(workers)
for i, src := range targets {
results[i].Name = src.Name
g.Go(func() error {
process(gctx, fc, st, cfg, src, now, &results[i])
return nil
})
}
if err := g.Wait(); err != nil {
return Summary{Results: results}, err
}
return Summary{Results: results}, nil
}
func process(ctx context.Context, fc *fetch.Client, st *store.Store, cfg config.Config, src source.Source, now time.Time, out *SourceResult) {
ctx, cancel := context.WithTimeout(ctx, time.Duration(cfg.Fetch.SourceTimeoutSeconds)*time.Second)
defer cancel()
id, err := st.UpsertSource(store.SourceInput{
Name: src.Name, Title: src.Title, URL: src.URL, Type: string(src.Type),
Weight: src.Weight, Tags: src.Tags, Enabled: src.Enabled,
})
if err != nil {
out.Err = err
return
}
prev, _, err := st.GetFetchState(id)
if err != nil {
out.Err = err
return
}
res, err := fc.Fetch(ctx, fetch.Request{
URL: src.URL, ETag: prev.ETag, LastModified: prev.LastModified,
})
if err != nil {
out.Err = err
return
}
if res.NotModified {
out.NotModified = true
out.Err = st.UpsertFetchState(id, store.FetchState{
ETag: prev.ETag, LastModified: prev.LastModified,
LastFetched: now.Unix(), LastStatus: int64(res.Status),
})
return
}
items, err := parse.Feed(bytes.NewReader(res.Body))
if err != nil {
out.Err = err
return
}
out.Parsed = len(items)
for _, it := range items {
storeItem(st, cfg, id, it, now, out)
}
if err := st.UpsertFetchState(id, store.FetchState{
ETag: res.ETag, LastModified: res.LastModified,
LastFetched: now.Unix(), LastStatus: int64(res.Status),
}); err != nil && out.Err == nil {
out.Err = err
}
}
func storeItem(st *store.Store, cfg config.Config, sourceID int64, it parse.Item, now time.Time, out *SourceResult) {
if it.Link == "" {
out.ItemErrors++
return
}
canonical, err := normalize.CanonicalURL(it.Link, cfg.Cluster.TrackingParams)
if err != nil {
out.ItemErrors++
return
}
summary := normalize.StripHTML(it.Summary)
body := normalize.StripHTML(it.Body)
var publishedAt int64
if !it.Published.IsZero() {
publishedAt = it.Published.Unix()
}
id, err := st.InsertArticle(store.Article{
SourceID: sourceID,
CanonicalURL: canonical,
ContentHash: normalize.ContentHash(canonical),
TitleHash: normalize.TitleHash(normalize.NormalizeTitle(it.Title)),
Title: it.Title,
Summary: summary,
Body: body,
Author: it.Author,
PublishedAt: publishedAt,
FetchedAt: now.Unix(),
})
switch {
case err == nil:
out.New++
linkCVEs(st, id, cve.Extract(it.Title, summary, body), out)
case errors.Is(err, store.ErrDuplicate):
out.Duplicates++
default:
out.ItemErrors++
}
}
func linkCVEs(st *store.Store, articleID int64, ids []string, out *SourceResult) {
for _, id := range ids {
if err := st.UpsertCVEStub(id); err != nil {
out.ItemErrors++
continue
}
if err := st.LinkArticleCVE(articleID, id); err != nil {
out.ItemErrors++
continue
}
out.CVEs++
}
}

199
PROJECTS/intermediate/security-news-scraper/internal/ingest/ingest_test.go Normal file
View File

@ -0,0 +1,199 @@
// ©AngelaMos | 2026
// ingest_test.go
package ingest
import (
"context"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"testing"
"time"
"github.com/CarterPerez-dev/nadezhda/internal/config"
"github.com/CarterPerez-dev/nadezhda/internal/fetch"
"github.com/CarterPerez-dev/nadezhda/internal/normalize"
"github.com/CarterPerez-dev/nadezhda/internal/source"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const fixture = "../../testdata/feeds/thehackernews.xml"
func loadFixture(t *testing.T) []byte {
t.Helper()
b, err := os.ReadFile(fixture)
if err != nil {
t.Fatalf("read fixture: %v", err)
}
return b
}
func newStore(t *testing.T) *store.Store {
t.Helper()
st, err := store.Open(filepath.Join(t.TempDir(), "ingest.db"))
if err != nil {
t.Fatalf("open store: %v", err)
}
t.Cleanup(func() { _ = st.Close() })
return st
}
func newClient() *fetch.Client {
return fetch.New(fetch.Options{
UserAgent: "nadezhda-test/1.0",
PerHostRate: 1e6,
PerHostBurst: 1,
Timeout: 5 * time.Second,
MaxRetries: 2,
})
}
func target(url string) []source.Source {
return []source.Source{{
Name: "test", Title: "Test Feed", URL: url,
Type: source.KindRSS, Weight: 1.0, Tags: []string{"news"}, Enabled: true,
}}
}
func TestRunIngestsAndDedups(t *testing.T) {
body := loadFixture(t)
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write(body)
}))
defer srv.Close()
st := newStore(t)
cfg := config.Default()
targets := target(srv.URL + "/feed")
now := time.Unix(1_800_000_000, 0)
first, err := Run(context.Background(), newClient(), st, cfg, targets, now)
if err != nil {
t.Fatalf("first Run: %v", err)
}
r0 := first.Results[0]
if r0.Err != nil {
t.Fatalf("source error: %v", r0.Err)
}
if r0.New != 3 || r0.Parsed != 3 {
t.Fatalf("first run: parsed=%d new=%d, want 3/3", r0.Parsed, r0.New)
}
count, err := st.CountArticles()
if err != nil {
t.Fatal(err)
}
if count != 3 {
t.Fatalf("stored articles = %d, want 3", count)
}
var title, canonical, contentHash, titleHash string
if err := st.DB().QueryRow(
`SELECT title, canonical_url, content_hash, title_hash FROM articles LIMIT 1`,
).Scan(&title, &canonical, &contentHash, &titleHash); err != nil {
t.Fatal(err)
}
if contentHash != normalize.ContentHash(canonical) {
t.Errorf("content_hash = %q, want sha256(canonical)", contentHash)
}
if titleHash != normalize.TitleHash(normalize.NormalizeTitle(title)) {
t.Errorf("title_hash = %q, want sha256(normalized title)", titleHash)
}
second, err := Run(context.Background(), newClient(), st, cfg, targets, now)
if err != nil {
t.Fatalf("second Run: %v", err)
}
r1 := second.Results[0]
if r1.New != 0 || r1.Duplicates != 3 {
t.Fatalf("second run: new=%d dup=%d, want 0/3", r1.New, r1.Duplicates)
}
count, _ = st.CountArticles()
if count != 3 {
t.Fatalf("after re-run stored = %d, want 3 (idempotent)", count)
}
}
func TestRunConditionalNotModified(t *testing.T) {
body := loadFixture(t)
const etag = `"abc123"`
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("If-None-Match") == etag {
w.WriteHeader(http.StatusNotModified)
return
}
w.Header().Set("ETag", etag)
_, _ = w.Write(body)
}))
defer srv.Close()
st := newStore(t)
cfg := config.Default()
targets := target(srv.URL + "/feed")
now := time.Unix(1_800_000_000, 0)
first, err := Run(context.Background(), newClient(), st, cfg, targets, now)
if err != nil {
t.Fatalf("first Run: %v", err)
}
if first.Results[0].New != 3 {
t.Fatalf("first run new = %d, want 3", first.Results[0].New)
}
var storedETag string
if err := st.DB().QueryRow(`SELECT etag FROM fetch_state LIMIT 1`).Scan(&storedETag); err != nil {
t.Fatal(err)
}
if storedETag != etag {
t.Errorf("persisted etag = %q, want %q", storedETag, etag)
}
second, err := Run(context.Background(), newClient(), st, cfg, targets, now)
if err != nil {
t.Fatalf("second Run: %v", err)
}
r := second.Results[0]
if !r.NotModified {
t.Errorf("expected NotModified on second run, got %+v", r)
}
if r.New != 0 {
t.Errorf("new = %d on 304, want 0", r.New)
}
}
func TestRunFailsSoftOnBadSource(t *testing.T) {
body := loadFixture(t)
good := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
_, _ = w.Write(body)
}))
defer good.Close()
bad := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
}))
defer bad.Close()
st := newStore(t)
cfg := config.Default()
targets := []source.Source{
{Name: "bad", URL: bad.URL + "/feed", Type: source.KindRSS, Weight: 1, Enabled: true},
{Name: "good", URL: good.URL + "/feed", Type: source.KindRSS, Weight: 1, Enabled: true},
}
now := time.Unix(1_800_000_000, 0)
summary, err := Run(context.Background(), newClient(), st, cfg, targets, now)
if err != nil {
t.Fatalf("Run returned error, should fail soft: %v", err)
}
byName := map[string]SourceResult{}
for _, r := range summary.Results {
byName[r.Name] = r
}
if byName["bad"].Err == nil {
t.Error("bad source should record an error")
}
if byName["good"].New != 3 {
t.Errorf("good source new = %d, want 3 (bad source must not abort run)", byName["good"].New)
}
}

111
PROJECTS/intermediate/security-news-scraper/internal/normalize/normalize.go Normal file
View File

@ -0,0 +1,111 @@
// ©AngelaMos | 2026
// normalize.go
package normalize
import (
"crypto/sha256"
"encoding/hex"
"fmt"
"net/url"
"strings"
"unicode"
"github.com/PuerkitoBio/goquery"
)
const (
wildcardSuffix = "*"
trailingSlash = "/"
)
func CanonicalURL(raw string, trackingParams []string) (string, error) {
u, err := url.Parse(strings.TrimSpace(raw))
if err != nil {
return "", fmt.Errorf("normalize: parse url %q: %w", raw, err)
}
if u.Host == "" {
return "", fmt.Errorf("normalize: url %q has no host", raw)
}
u.Scheme = strings.ToLower(u.Scheme)
u.Host = strings.ToLower(u.Host)
u.Fragment = ""
u.RawFragment = ""
if q := u.Query(); len(q) > 0 {
for key := range q {
if isTracking(key, trackingParams) {
q.Del(key)
}
}
u.RawQuery = q.Encode()
}
u.Path = strings.TrimRight(u.Path, trailingSlash)
if u.RawPath != "" {
u.RawPath = strings.TrimRight(u.RawPath, trailingSlash)
}
return u.String(), nil
}
func isTracking(key string, trackingParams []string) bool {
lowered := strings.ToLower(key)
for _, p := range trackingParams {
if strings.HasSuffix(p, wildcardSuffix) {
if strings.HasPrefix(lowered, strings.TrimSuffix(p, wildcardSuffix)) {
return true
}
continue
}
if lowered == p {
return true
}
}
return false
}
func NormalizeTitle(title string) string {
var b strings.Builder
b.Grow(len(title))
prevSpace := false
for _, r := range strings.ToLower(title) {
if unicode.IsLetter(r) || unicode.IsDigit(r) {
b.WriteRune(r)
prevSpace = false
continue
}
if !prevSpace {
b.WriteByte(' ')
prevSpace = true
}
}
return strings.TrimSpace(b.String())
}
func StripHTML(s string) string {
doc, err := goquery.NewDocumentFromReader(strings.NewReader(s))
if err != nil {
return collapseWhitespace(s)
}
doc.Find("script,style").Remove()
return collapseWhitespace(doc.Text())
}
func collapseWhitespace(s string) string {
return strings.Join(strings.Fields(s), " ")
}
func ContentHash(canonicalURL string) string {
return sha256Hex(canonicalURL)
}
func TitleHash(normalizedTitle string) string {
return sha256Hex(normalizedTitle)
}
func sha256Hex(s string) string {
sum := sha256.Sum256([]byte(s))
return hex.EncodeToString(sum[:])
}

111
PROJECTS/intermediate/security-news-scraper/internal/normalize/normalize_test.go Normal file
View File

@ -0,0 +1,111 @@
// ©AngelaMos | 2026
// normalize_test.go
package normalize
import "testing"
var params = []string{"utm_*", "gclid", "fbclid", "ref", "mc_cid", "mc_eid"}
func TestCanonicalURL(t *testing.T) {
cases := []struct {
name string
in string
want string
}{
{"lowercase host+scheme", "HTTPS://Example.COM/Path", "https://example.com/Path"},
{"drop fragment", "https://example.com/a#section", "https://example.com/a"},
{"strip utm params", "https://example.com/a?utm_source=x&utm_medium=y&id=7", "https://example.com/a?id=7"},
{"strip gclid fbclid ref", "https://example.com/a?gclid=1&fbclid=2&ref=z&keep=1", "https://example.com/a?keep=1"},
{"strip mailchimp", "https://example.com/a?mc_cid=1&mc_eid=2", "https://example.com/a"},
{"drop trailing slash", "https://example.com/a/b/", "https://example.com/a/b"},
{"root trailing slash", "https://example.com/", "https://example.com"},
{"sorted query", "https://example.com/a?b=2&a=1", "https://example.com/a?a=1&b=2"},
{"tracking case insensitive", "https://example.com/a?UTM_Source=x&id=1", "https://example.com/a?id=1"},
{"path preserved case", "https://example.com/Foo/Bar", "https://example.com/Foo/Bar"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got, err := CanonicalURL(tc.in, params)
if err != nil {
t.Fatalf("CanonicalURL(%q): %v", tc.in, err)
}
if got != tc.want {
t.Errorf("CanonicalURL(%q) = %q, want %q", tc.in, got, tc.want)
}
})
}
}
func TestCanonicalURLCollapsesTracking(t *testing.T) {
a, _ := CanonicalURL("https://example.com/story?utm_source=twitter", params)
b, _ := CanonicalURL("https://example.com/story", params)
if a != b {
t.Errorf("tracking variants did not collapse: %q vs %q", a, b)
}
if ContentHash(a) != ContentHash(b) {
t.Error("content hashes differ for tracking variants")
}
}
func TestCanonicalURLErrors(t *testing.T) {
for _, in := range []string{"", "not a url", "/relative/only"} {
if _, err := CanonicalURL(in, params); err == nil {
t.Errorf("CanonicalURL(%q) expected error", in)
}
}
}
func TestNormalizeTitle(t *testing.T) {
cases := []struct {
in string
want string
}{
{"CVE-2021-44228: Log4Shell!", "cve 2021 44228 log4shell"},
{" Multiple Spaces ", "multiple spaces"},
{"Punctuation, and; stuff.", "punctuation and stuff"},
{"Café déjà vu", "café déjà vu"},
{"", ""},
}
for _, tc := range cases {
if got := NormalizeTitle(tc.in); got != tc.want {
t.Errorf("NormalizeTitle(%q) = %q, want %q", tc.in, got, tc.want)
}
}
}
func TestNormalizeTitleStableHash(t *testing.T) {
a := TitleHash(NormalizeTitle("Breaking: Big Hack!!!"))
b := TitleHash(NormalizeTitle("breaking big hack"))
if a != b {
t.Error("normalized-title hashes differ for equivalent titles")
}
}
func TestStripHTML(t *testing.T) {
cases := []struct {
in string
want string
}{
{"<p>Hello <b>world</b></p>", "Hello world"},
{"<div>a</div>\n<div>b</div>", "a b"},
{"<script>bad()</script>visible", "visible"},
{"plain text", "plain text"},
{"<a href='x'>link</a> and text", "link and text"},
}
for _, tc := range cases {
if got := StripHTML(tc.in); got != tc.want {
t.Errorf("StripHTML(%q) = %q, want %q", tc.in, got, tc.want)
}
}
}
func TestContentHashDeterministic(t *testing.T) {
const u = "https://example.com/a"
if ContentHash(u) != ContentHash(u) {
t.Error("ContentHash not deterministic")
}
if len(ContentHash(u)) != 64 {
t.Errorf("ContentHash length = %d, want 64 hex chars", len(ContentHash(u)))
}
}

74
PROJECTS/intermediate/security-news-scraper/internal/parse/parse.go Normal file
View File

@ -0,0 +1,74 @@
// ©AngelaMos | 2026
// parse.go
package parse
import (
"fmt"
"io"
"strings"
"time"
"github.com/mmcdole/gofeed"
)
type Item struct {
Title string
Link string
Summary string
Body string
Author string
Published time.Time
Categories []string
}
var timeLayouts = []string{time.RFC1123Z, time.RFC1123, time.RFC822Z, time.RFC822, time.RFC3339}
func Feed(r io.Reader) ([]Item, error) {
feed, err := gofeed.NewParser().Parse(r)
if err != nil {
return nil, fmt.Errorf("parse: %w", err)
}
items := make([]Item, 0, len(feed.Items))
for _, it := range feed.Items {
items = append(items, Item{
Title: strings.TrimSpace(it.Title),
Link: strings.TrimSpace(it.Link),
Summary: it.Description,
Body: it.Content,
Author: author(it),
Published: published(it),
Categories: it.Categories,
})
}
return items, nil
}
func author(it *gofeed.Item) string {
if it.Author != nil && it.Author.Name != "" {
return it.Author.Name
}
if len(it.Authors) > 0 {
return it.Authors[0].Name
}
return ""
}
func published(it *gofeed.Item) time.Time {
if it.PublishedParsed != nil {
return it.PublishedParsed.UTC()
}
if it.UpdatedParsed != nil {
return it.UpdatedParsed.UTC()
}
raw := strings.TrimSpace(it.Published)
if raw == "" {
raw = strings.TrimSpace(it.Updated)
}
for _, layout := range timeLayouts {
if t, err := time.Parse(layout, raw); err == nil {
return t.UTC()
}
}
return time.Time{}
}

105
PROJECTS/intermediate/security-news-scraper/internal/parse/parse_test.go Normal file
View File

@ -0,0 +1,105 @@
// ©AngelaMos | 2026
// parse_test.go
package parse
import (
"os"
"path/filepath"
"testing"
"time"
)
const feedsDir = "../../testdata/feeds"
func loadFeed(t *testing.T, name string) []Item {
t.Helper()
f, err := os.Open(filepath.Join(feedsDir, name))
if err != nil {
t.Fatalf("open fixture %s: %v", name, err)
}
t.Cleanup(func() { _ = f.Close() })
items, err := Feed(f)
if err != nil {
t.Fatalf("Feed(%s): %v", name, err)
}
return items
}
func TestFeedGolden(t *testing.T) {
cases := []struct {
file string
wantItems int
fullBody bool
}{
{"krebs.xml", 3, true},
{"theregister.xml", 3, true},
{"thehackernews.xml", 3, false},
{"bleepingcomputer.xml", 3, false},
{"securityweek.xml", 3, false},
{"darkreading.xml", 3, false},
{"cisa.xml", 3, false},
}
for _, tc := range cases {
t.Run(tc.file, func(t *testing.T) {
items := loadFeed(t, tc.file)
if len(items) != tc.wantItems {
t.Fatalf("items = %d, want %d", len(items), tc.wantItems)
}
for i, it := range items {
if it.Title == "" {
t.Errorf("item %d: empty title", i)
}
if it.Link == "" {
t.Errorf("item %d: empty link", i)
}
if it.Published.IsZero() {
t.Errorf("item %d (%q): published time did not parse", i, it.Title)
}
if tc.fullBody && it.Body == "" {
t.Errorf("item %d (%q): expected full body, got empty", i, it.Title)
}
}
})
}
}
func TestFeedExactPublishedTime(t *testing.T) {
cases := []struct {
file string
raw string
}{
{"thehackernews.xml", "Sat, 04 Jul 2026 18:17:53 +0530"},
{"krebs.xml", "Thu, 02 Jul 2026 19:27:33 +0000"},
}
for _, tc := range cases {
t.Run(tc.file, func(t *testing.T) {
want, err := time.Parse(time.RFC1123Z, tc.raw)
if err != nil {
t.Fatalf("parse expected time: %v", err)
}
items := loadFeed(t, tc.file)
if !items[0].Published.Equal(want) {
t.Errorf("published = %s, want %s", items[0].Published, want.UTC())
}
})
}
}
func TestFeedExactTitles(t *testing.T) {
cases := []struct {
file string
title string
}{
{"krebs.xml", "FBI Seizes NetNut Proxy Platform, Popa Botnet"},
{"thehackernews.xml", "U.S. Government Entity Paid Kairos $1 Million in Data-Theft Extortion Case"},
}
for _, tc := range cases {
t.Run(tc.file, func(t *testing.T) {
items := loadFeed(t, tc.file)
if items[0].Title != tc.title {
t.Errorf("first title = %q, want %q", items[0].Title, tc.title)
}
})
}
}

159
PROJECTS/intermediate/security-news-scraper/internal/rank/rank.go Normal file
View File

@ -0,0 +1,159 @@
// ©AngelaMos | 2026
// rank.go
package rank
import (
"math"
"sort"
"strings"
"time"
"github.com/CarterPerez-dev/nadezhda/internal/config"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const (
cvssMax = 10.0
minVelocityWindowHours = 1.0
secondsPerHour = 3600.0
)
type Signals struct {
AgeHours float64
MaxCVSS float64
KEV bool
MaxEPSS float64
ClusterSize int
ClusterAgeHours float64
SourceWeight float64
KeywordMatch bool
}
func Score(s Signals, cfg config.Rank) float64 {
w := cfg.Weights
return w.Recency*recency(s.AgeHours, cfg.HalfLifeHours) +
w.CVSS*clamp01(s.MaxCVSS/cvssMax) +
w.KEV*boolScore(s.KEV) +
w.EPSS*clamp01(s.MaxEPSS) +
w.Velocity*velocity(s.ClusterSize, s.ClusterAgeHours, cfg.VelocityNorm) +
w.Source*clamp01(s.SourceWeight) +
w.Keyword*boolScore(s.KeywordMatch)
}
func recency(ageHours float64, halfLifeHours int) float64 {
if ageHours < 0 {
ageHours = 0
}
if halfLifeHours < 1 {
halfLifeHours = 1
}
return math.Exp(-math.Ln2 * ageHours / float64(halfLifeHours))
}
func velocity(size int, ageHours, norm float64) float64 {
if size <= 1 || norm <= 0 {
return 0
}
if ageHours < minVelocityWindowHours {
ageHours = minVelocityWindowHours
}
return clamp01((float64(size) / ageHours) / norm)
}
func boolScore(b bool) float64 {
if b {
return 1
}
return 0
}
func clamp01(v float64) float64 {
if v < 0 {
return 0
}
if v > 1 {
return 1
}
return v
}
type Scored struct {
Cluster store.DigestCluster
Score float64
}
func Rank(clusters []store.DigestCluster, cfg config.Rank, watchlist []string, now time.Time) []Scored {
out := make([]Scored, len(clusters))
for i, c := range clusters {
out[i] = Scored{Cluster: c, Score: Score(signalsFor(c, watchlist, now), cfg)}
}
sort.SliceStable(out, func(a, b int) bool {
if out[a].Score != out[b].Score {
return out[a].Score > out[b].Score
}
return out[a].Cluster.LastSeen > out[b].Cluster.LastSeen
})
return out
}
func signalsFor(c store.DigestCluster, watchlist []string, now time.Time) Signals {
s := Signals{
AgeHours: hoursSince(c.LastSeen, now),
ClusterSize: c.Size,
ClusterAgeHours: float64(c.LastSeen-c.FirstSeen) / secondsPerHour,
SourceWeight: maxSourceWeight(c.Articles),
KeywordMatch: matchesWatchlist(c, watchlist),
}
for _, v := range c.CVEs {
if v.CVSSScore != nil && *v.CVSSScore > s.MaxCVSS {
s.MaxCVSS = *v.CVSSScore
}
if v.EPSS != nil && *v.EPSS > s.MaxEPSS {
s.MaxEPSS = *v.EPSS
}
if v.IsKEV {
s.KEV = true
}
}
return s
}
func hoursSince(unix int64, now time.Time) float64 {
return float64(now.Unix()-unix) / secondsPerHour
}
func maxSourceWeight(articles []store.DigestArticle) float64 {
var max float64
for _, a := range articles {
if a.SourceWeight > max {
max = a.SourceWeight
}
}
return max
}
func matchesWatchlist(c store.DigestCluster, watchlist []string) bool {
if len(watchlist) == 0 {
return false
}
var sb strings.Builder
for _, a := range c.Articles {
sb.WriteString(strings.ToLower(a.Title))
sb.WriteByte(' ')
}
for _, v := range c.CVEs {
sb.WriteString(strings.ToLower(v.ID))
sb.WriteByte(' ')
}
hay := sb.String()
for _, term := range watchlist {
if term == "" {
continue
}
if strings.Contains(hay, strings.ToLower(term)) {
return true
}
}
return false
}

106
PROJECTS/intermediate/security-news-scraper/internal/rank/rank_test.go Normal file
View File

@ -0,0 +1,106 @@
// ©AngelaMos | 2026
// rank_test.go
package rank
import (
"math"
"testing"
"time"
"github.com/CarterPerez-dev/nadezhda/internal/config"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
func f(v float64) *float64 { return &v }
func TestScoreGoldenOrder(t *testing.T) {
cfg := config.Default().Rank
a := Signals{
AgeHours: 2, MaxCVSS: 9.8, KEV: true, MaxEPSS: 0.97,
ClusterSize: 5, ClusterAgeHours: 6, SourceWeight: 1.0, KeywordMatch: true,
}
b := Signals{
AgeHours: 120, MaxCVSS: 4.3, KEV: false, MaxEPSS: 0.02,
ClusterSize: 1, ClusterAgeHours: 0, SourceWeight: 0.6, KeywordMatch: false,
}
sa, sb := Score(a, cfg), Score(b, cfg)
if !(sa > sb) {
t.Fatalf("expected A (%.4f) to rank strictly above B (%.4f)", sa, sb)
}
if sa < 0.95 {
t.Errorf("A score = %.4f, want ~0.98", sa)
}
if sb > 0.20 {
t.Errorf("B score = %.4f, want ~0.14", sb)
}
}
func TestRecencyDecay(t *testing.T) {
if got := recency(0, 48); math.Abs(got-1.0) > 1e-9 {
t.Errorf("recency(0) = %v, want 1.0", got)
}
if got := recency(48, 48); math.Abs(got-0.5) > 1e-9 {
t.Errorf("recency(one half-life) = %v, want 0.5", got)
}
if got := recency(-5, 48); got != 1.0 {
t.Errorf("negative age should clamp to 1.0, got %v", got)
}
}
func TestVelocity(t *testing.T) {
if v := velocity(1, 0, 0.5); v != 0 {
t.Errorf("size-1 velocity = %v, want 0", v)
}
if v := velocity(6, 12, 0.5); v != 1.0 {
t.Errorf("size-6 in 12h should saturate, got %v", v)
}
if v := velocity(3, 0, 0.5); v != 1.0 {
t.Errorf("burst (age<floor) should saturate, got %v", v)
}
if v := velocity(2, 100, 0.5); math.Abs(v-0.04) > 1e-9 {
t.Errorf("slow cluster velocity = %v, want 0.04", v)
}
}
func TestRankOrdersByScore(t *testing.T) {
now := time.Unix(1_000_000, 0)
hot := store.DigestCluster{
ClusterID: 1, Size: 4, FirstSeen: now.Unix() - 3600, LastSeen: now.Unix() - 60,
Articles: []store.DigestArticle{{SourceName: "a", SourceWeight: 1.0, PublishedAt: now.Unix() - 60}},
CVEs: []store.DigestCVE{{ID: "CVE-2026-1", CVSSScore: f(9.9), EPSS: f(0.95), IsKEV: true}},
}
cold := store.DigestCluster{
ClusterID: 2, Size: 1, FirstSeen: now.Unix() - 500*3600, LastSeen: now.Unix() - 500*3600,
Articles: []store.DigestArticle{{SourceName: "b", SourceWeight: 0.5, PublishedAt: now.Unix() - 500*3600}},
}
scored := Rank([]store.DigestCluster{cold, hot}, config.Default().Rank, nil, now)
if scored[0].Cluster.ClusterID != 1 {
t.Errorf("hot KEV cluster should rank first, got cluster %d", scored[0].Cluster.ClusterID)
}
if !(scored[0].Score > scored[1].Score) {
t.Errorf("scores not descending: %.4f then %.4f", scored[0].Score, scored[1].Score)
}
}
func TestKeywordMatch(t *testing.T) {
c := store.DigestCluster{
Articles: []store.DigestArticle{{Title: "Fortinet FortiOS flaw exploited"}},
CVEs: []store.DigestCVE{{ID: "CVE-2026-1"}},
}
if !matchesWatchlist(c, []string{"fortinet"}) {
t.Error("case-insensitive watchlist term should match title")
}
if !matchesWatchlist(c, []string{"CVE-2026-1"}) {
t.Error("watchlist should match against CVE ids")
}
if matchesWatchlist(c, []string{"cisco"}) {
t.Error("non-matching term should not match")
}
if matchesWatchlist(c, nil) {
t.Error("empty watchlist should not match")
}
}

145
PROJECTS/intermediate/security-news-scraper/internal/setup/credentials.go Normal file
View File

@ -0,0 +1,145 @@
// ©AngelaMos | 2026
// credentials.go
package setup
import (
"bufio"
"fmt"
"os"
"sort"
"strings"
)
const (
EnvProvider = "NADEZHDA_AI_PROVIDER"
EnvQwenURL = "NADEZHDA_QWEN_BASE_URL"
commentMark = "#"
kvSeparator = "="
credentialKeyPrefix = "NADEZHDA_"
credentialKeySuffix = "_API_KEY"
tmpSuffix = ".tmp"
credentialsHeader = "# ©AngelaMos | 2026\n# nadezhda credentials — auto-managed by `nadezhda ai`; keep private, do not commit\n\n"
)
func Load() error {
path, err := CredentialsPath()
if err != nil {
return err
}
kv, err := readCredentials(path)
if err != nil {
return err
}
for k, v := range kv {
if !allowedKey(k) {
continue
}
if _, set := os.LookupEnv(k); set {
continue
}
if err := os.Setenv(k, v); err != nil {
return fmt.Errorf("setup: set %s: %w", k, err)
}
}
return nil
}
func allowedKey(k string) bool {
return strings.HasPrefix(k, credentialKeyPrefix) || strings.HasSuffix(k, credentialKeySuffix)
}
func NonSecretEnviron() []string {
env := os.Environ()
out := make([]string, 0, len(env))
for _, e := range env {
if k, _, ok := strings.Cut(e, kvSeparator); ok && allowedKey(k) {
continue
}
out = append(out, e)
}
return out
}
func Save(updates map[string]string) error {
path, err := CredentialsPath()
if err != nil {
return err
}
dir, err := ConfigDir()
if err != nil {
return err
}
if err := os.MkdirAll(dir, dirPerm); err != nil {
return fmt.Errorf("setup: create config dir: %w", err)
}
kv, err := readCredentials(path)
if err != nil {
return err
}
for k, v := range updates {
kv[k] = v
}
return writeCredentials(path, kv)
}
func readCredentials(path string) (map[string]string, error) {
f, err := os.Open(path)
if err != nil {
if os.IsNotExist(err) {
return map[string]string{}, nil
}
return nil, fmt.Errorf("setup: read credentials: %w", err)
}
defer f.Close()
kv := map[string]string{}
sc := bufio.NewScanner(f)
for sc.Scan() {
line := strings.TrimSpace(sc.Text())
if line == "" || strings.HasPrefix(line, commentMark) {
continue
}
k, v, ok := strings.Cut(line, kvSeparator)
if !ok {
continue
}
key := strings.TrimSpace(k)
if key == "" {
continue
}
kv[key] = strings.TrimSpace(v)
}
return kv, sc.Err()
}
func writeCredentials(path string, kv map[string]string) error {
keys := make([]string, 0, len(kv))
for k := range kv {
keys = append(keys, k)
}
sort.Strings(keys)
var b strings.Builder
b.WriteString(credentialsHeader)
for _, k := range keys {
fmt.Fprintf(&b, "%s=%s\n", k, kv[k])
}
tmp := path + tmpSuffix
if err := os.WriteFile(tmp, []byte(b.String()), filePerm); err != nil {
return fmt.Errorf("setup: write credentials: %w", err)
}
if err := os.Chmod(tmp, filePerm); err != nil {
os.Remove(tmp)
return fmt.Errorf("setup: secure credentials: %w", err)
}
if err := os.Rename(tmp, path); err != nil {
os.Remove(tmp)
return fmt.Errorf("setup: replace credentials: %w", err)
}
return nil
}

144
PROJECTS/intermediate/security-news-scraper/internal/setup/credentials_test.go Normal file
View File

@ -0,0 +1,144 @@
// ©AngelaMos | 2026
// credentials_test.go
package setup
import (
"os"
"testing"
)
func TestCredentialsRoundTripAndPerms(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
if err := Save(map[string]string{"OPENAI_API_KEY": "sk-abc", EnvProvider: "openai"}); err != nil {
t.Fatalf("Save: %v", err)
}
if err := Save(map[string]string{"NVD_API_KEY": "nvd-1"}); err != nil {
t.Fatalf("Save merge: %v", err)
}
path, _ := CredentialsPath()
kv, err := readCredentials(path)
if err != nil {
t.Fatalf("readCredentials: %v", err)
}
if kv["OPENAI_API_KEY"] != "sk-abc" || kv[EnvProvider] != "openai" || kv["NVD_API_KEY"] != "nvd-1" {
t.Errorf("merge lost data: %v", kv)
}
fi, err := os.Stat(path)
if err != nil {
t.Fatal(err)
}
if perm := fi.Mode().Perm(); perm != 0o600 {
t.Errorf("credentials perm = %o, want 600", perm)
}
}
func TestLoadEnvWins(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
if err := Save(map[string]string{"OPENAI_API_KEY": "fromfile", "NADEZHDA_TEST_ONLY": "x"}); err != nil {
t.Fatal(err)
}
t.Setenv("OPENAI_API_KEY", "fromenv")
defer os.Unsetenv("NADEZHDA_TEST_ONLY")
if err := Load(); err != nil {
t.Fatalf("Load: %v", err)
}
if got := os.Getenv("OPENAI_API_KEY"); got != "fromenv" {
t.Errorf("OPENAI_API_KEY = %q, want fromenv (env must win over file)", got)
}
if got := os.Getenv("NADEZHDA_TEST_ONLY"); got != "x" {
t.Errorf("NADEZHDA_TEST_ONLY = %q, want x (unset var loads from file)", got)
}
}
func TestReadCredentialsSkipsCommentsAndBlanks(t *testing.T) {
path := t.TempDir() + "/creds"
content := "# header\n\nOPENAI_API_KEY=sk-1\n # indented comment\nbadline_no_sep\nNVD_API_KEY = nvd-2 \n"
if err := os.WriteFile(path, []byte(content), 0o600); err != nil {
t.Fatal(err)
}
kv, err := readCredentials(path)
if err != nil {
t.Fatal(err)
}
if kv["OPENAI_API_KEY"] != "sk-1" || kv["NVD_API_KEY"] != "nvd-2" || len(kv) != 2 {
t.Errorf("parsed = %v, want exactly 2 clean entries", kv)
}
}
func TestSaveCreatesDir0700AndFile0600(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
if err := Save(map[string]string{"OPENAI_API_KEY": "k"}); err != nil {
t.Fatal(err)
}
dir, _ := ConfigDir()
if di, _ := os.Stat(dir); di.Mode().Perm() != 0o700 {
t.Errorf("config dir perms = %o, want 700", di.Mode().Perm())
}
path, _ := CredentialsPath()
if fi, _ := os.Stat(path); fi.Mode().Perm() != 0o600 {
t.Errorf("credentials perms = %o, want 600", fi.Mode().Perm())
}
}
func TestSaveTightensExistingLooseFile(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
dir, _ := ConfigDir()
if err := os.MkdirAll(dir, 0o700); err != nil {
t.Fatal(err)
}
path, _ := CredentialsPath()
if err := os.WriteFile(path, []byte("OPENAI_API_KEY=old\n"), 0o644); err != nil {
t.Fatal(err)
}
if err := Save(map[string]string{"OPENAI_API_KEY": "new"}); err != nil {
t.Fatal(err)
}
if fi, _ := os.Stat(path); fi.Mode().Perm() != 0o600 {
t.Errorf("Save left perms %o on a pre-existing 0644 file, want 600", fi.Mode().Perm())
}
}
func TestLoadRespectsSetEmptyEnv(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
if err := Save(map[string]string{"OPENAI_API_KEY": "fromfile"}); err != nil {
t.Fatal(err)
}
t.Setenv("OPENAI_API_KEY", "")
if err := Load(); err != nil {
t.Fatal(err)
}
if got := os.Getenv("OPENAI_API_KEY"); got != "" {
t.Errorf("an explicitly-cleared env var was overridden from file: %q", got)
}
}
func TestLoadBlocksDisallowedAndEmptyKeys(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
dir, _ := ConfigDir()
if err := os.MkdirAll(dir, 0o700); err != nil {
t.Fatal(err)
}
path, _ := CredentialsPath()
if err := os.WriteFile(path, []byte("=orphan\nLD_PRELOAD=/tmp/evil.so\nOPENAI_API_KEY=ok\n"), 0o600); err != nil {
t.Fatal(err)
}
os.Unsetenv("LD_PRELOAD")
os.Unsetenv("OPENAI_API_KEY")
defer os.Unsetenv("LD_PRELOAD")
defer os.Unsetenv("OPENAI_API_KEY")
if err := Load(); err != nil {
t.Fatalf("Load must tolerate an empty-key line, got: %v", err)
}
if _, set := os.LookupEnv("LD_PRELOAD"); set {
t.Error("LD_PRELOAD injected from credentials file — allowlist bypass (RCE vector)")
}
if os.Getenv("OPENAI_API_KEY") != "ok" {
t.Error("allowlisted OPENAI_API_KEY not loaded")
}
}

30
PROJECTS/intermediate/security-news-scraper/internal/setup/ollama.compose.yml Normal file
View File

@ -0,0 +1,30 @@
# ©AngelaMos | 2026
# ollama.compose.yml
name: nadezhda-ollama
services:
ollama:
image: ollama/ollama:latest
container_name: nadezhda-ollama
ports:
- "127.0.0.1:${OLLAMA_HOST_PORT:-39847}:11434"
volumes:
- nadezhda-ollama-models:/root/.ollama
entrypoint: ["/bin/sh", "-c"]
command:
- |
ollama serve &
until ollama list >/dev/null 2>&1; do sleep 1; done
ollama pull qwen2.5:7b
wait
healthcheck:
test: ["CMD-SHELL", "ollama list | grep -q qwen2.5"]
interval: 10s
timeout: 5s
retries: 60
start_period: 30s
restart: unless-stopped
volumes:
nadezhda-ollama-models:

67
PROJECTS/intermediate/security-news-scraper/internal/setup/ollama.go Normal file
View File

@ -0,0 +1,67 @@
// ©AngelaMos | 2026
// ollama.go
package setup
import (
_ "embed"
"fmt"
"net/http"
"os"
"os/exec"
"path/filepath"
"strings"
)
//go:embed ollama.compose.yml
var ComposeYAML []byte
const (
composeFileName = "ollama.compose.yml"
composePerm = 0o644
versionPath = "/api/version"
apiV1Suffix = "/v1"
)
var ollamaCandidates = []string{
"http://localhost:39847",
"http://localhost:11434",
}
func DetectOllama(client *http.Client) string {
for _, base := range ollamaCandidates {
resp, err := client.Get(base + versionPath)
if err != nil {
continue
}
resp.Body.Close()
if resp.StatusCode == http.StatusOK {
return base
}
}
return ""
}
func WriteCompose(dir string) (string, error) {
path := filepath.Join(dir, composeFileName)
if err := os.WriteFile(path, ComposeYAML, composePerm); err != nil {
return "", fmt.Errorf("setup: write compose: %w", err)
}
return path, nil
}
func HasBinary(name string) bool {
_, err := exec.LookPath(name)
return err == nil
}
func OllamaReachable(baseURL string) bool {
base := strings.TrimSuffix(baseURL, apiV1Suffix)
client := &http.Client{Timeout: detectTimeout}
resp, err := client.Get(base + versionPath)
if err != nil {
return false
}
resp.Body.Close()
return resp.StatusCode == http.StatusOK
}

62
PROJECTS/intermediate/security-news-scraper/internal/setup/ollama_test.go Normal file
View File

@ -0,0 +1,62 @@
// ©AngelaMos | 2026
// ollama_test.go
package setup
import (
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestDetectOllama(t *testing.T) {
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == versionPath {
w.WriteHeader(http.StatusOK)
return
}
w.WriteHeader(http.StatusNotFound)
}))
defer srv.Close()
old := ollamaCandidates
defer func() { ollamaCandidates = old }()
client := &http.Client{Timeout: 2 * time.Second}
ollamaCandidates = []string{srv.URL}
if got := DetectOllama(client); got != srv.URL {
t.Errorf("DetectOllama = %q, want %q", got, srv.URL)
}
ollamaCandidates = []string{"http://127.0.0.1:1"}
if got := DetectOllama(client); got != "" {
t.Errorf("DetectOllama with no server = %q, want empty", got)
}
}
func TestWriteComposeMatchesEmbed(t *testing.T) {
yaml := string(ComposeYAML)
if len(yaml) == 0 || !strings.Contains(yaml, "nadezhda-ollama") || !strings.Contains(yaml, "qwen2.5:7b") {
t.Fatal("embedded compose is empty or missing expected content")
}
dir := t.TempDir()
path, err := WriteCompose(dir)
if err != nil {
t.Fatalf("WriteCompose: %v", err)
}
if path != filepath.Join(dir, composeFileName) {
t.Errorf("path = %q", path)
}
got, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
if string(got) != yaml {
t.Error("written compose differs from embedded content")
}
}

33
PROJECTS/intermediate/security-news-scraper/internal/setup/paths.go Normal file
View File

@ -0,0 +1,33 @@
// ©AngelaMos | 2026
// paths.go
package setup
import (
"fmt"
"os"
"path/filepath"
)
const (
appDir = "nadezhda"
credentialsFile = "credentials"
dirPerm = 0o700
filePerm = 0o600
)
func ConfigDir() (string, error) {
base, err := os.UserConfigDir()
if err != nil {
return "", fmt.Errorf("setup: locate config dir: %w", err)
}
return filepath.Join(base, appDir), nil
}
func CredentialsPath() (string, error) {
dir, err := ConfigDir()
if err != nil {
return "", err
}
return filepath.Join(dir, credentialsFile), nil
}

158
PROJECTS/intermediate/security-news-scraper/internal/setup/wizard.go Normal file
View File

@ -0,0 +1,158 @@
// ©AngelaMos | 2026
// wizard.go
package setup
import (
"fmt"
"io"
"net/http"
"os"
"strings"
"time"
"golang.org/x/term"
)
const (
ProviderClaude = "anthropic"
ProviderOpenAI = "openai"
ProviderGemini = "gemini"
ProviderQwen = "qwen"
envOpenAIKey = "OPENAI_API_KEY"
envGeminiKey = "GEMINI_API_KEY"
envAnthropicKey = "ANTHROPIC_API_KEY"
envNVDKey = "NVD_API_KEY"
composeQwenURL = "http://localhost:39847/v1"
nativeQwenURL = "http://localhost:11434/v1"
detectTimeout = 3 * time.Second
ollamaInstall = "https://ollama.com/download"
)
type apiChoice struct {
provider string
envKey string
label string
}
var apiProviders = map[string]apiChoice{
"1": {ProviderClaude, envAnthropicKey, "Claude (Anthropic)"},
"2": {ProviderOpenAI, envOpenAIKey, "OpenAI"},
"3": {ProviderGemini, envGeminiKey, "Gemini (Google)"},
}
func Run(in io.Reader, out io.Writer) error {
fmt.Fprintln(out, "nadezhda AI setup")
fmt.Fprintln(out, " 1) Claude (Anthropic) 2) OpenAI 3) Gemini 4) Ollama (local qwen, free, no key)")
choice := prompt(in, out, "choose a provider [1-4]: ")
switch choice {
case "1", "2", "3":
return setupAPIProvider(in, out, apiProviders[choice])
case "4":
return setupOllama(out)
default:
return fmt.Errorf("setup: invalid choice %q (want 1-4)", choice)
}
}
func setupAPIProvider(in io.Reader, out io.Writer, c apiChoice) error {
fmt.Fprintf(out, "\nConfiguring %s.\n", c.label)
key := strings.TrimSpace(readSecret(in, out, fmt.Sprintf("paste your %s API key: ", c.label)))
if key == "" {
return fmt.Errorf("setup: no API key entered")
}
updates := map[string]string{
EnvProvider: c.provider,
c.envKey: key,
}
if nvd := strings.TrimSpace(readSecret(in, out, "optional NVD token for the CVE booster (enter to skip): ")); nvd != "" {
updates[envNVDKey] = nvd
}
if err := Save(updates); err != nil {
return err
}
path, _ := CredentialsPath()
fmt.Fprintf(out, "\nSaved to %s (0600). %s is ready — run: nadezhda ideate\n", path, c.label)
return nil
}
func setupOllama(out io.Writer) error {
fmt.Fprintln(out, "\nConfiguring Ollama (local qwen2.5, no API key).")
client := &http.Client{Timeout: detectTimeout}
if base := DetectOllama(client); base != "" {
fmt.Fprintf(out, "Found a running Ollama at %s — using it.\n", base)
return saveOllama(out, base+"/v1")
}
fmt.Fprintln(out, "No running Ollama found.")
switch {
case HasBinary("docker"):
dir, _ := os.Getwd()
path, err := WriteCompose(dir)
if err != nil {
return err
}
fmt.Fprintf(out, "Wrote %s. Start it with:\n docker compose -f %s up -d\n", path, path)
fmt.Fprintln(out, "(first run pulls qwen2.5:7b, a few minutes), then re-run `nadezhda ai` and pick 4.")
return saveOllama(out, composeQwenURL)
case HasBinary("ollama"):
fmt.Fprintln(out, "Ollama is installed but not running. Start it and pull the model:")
fmt.Fprintln(out, " ollama serve & then: ollama pull qwen2.5:7b")
return saveOllama(out, nativeQwenURL)
default:
fmt.Fprintf(out, "Install Ollama (%s) or Docker, then re-run `nadezhda ai`.\n", ollamaInstall)
return saveOllama(out, composeQwenURL)
}
}
func saveOllama(out io.Writer, baseURL string) error {
if err := Save(map[string]string{EnvProvider: ProviderQwen, EnvQwenURL: baseURL}); err != nil {
return err
}
path, _ := CredentialsPath()
fmt.Fprintf(out, "Saved qwen (%s) to %s. Run: nadezhda ideate\n", baseURL, path)
return nil
}
func prompt(in io.Reader, out io.Writer, label string) string {
fmt.Fprint(out, label)
return readLine(in)
}
func readSecret(in io.Reader, out io.Writer, label string) string {
fmt.Fprint(out, label)
if f, ok := in.(*os.File); ok && term.IsTerminal(int(f.Fd())) {
b, err := term.ReadPassword(int(f.Fd()))
fmt.Fprintln(out)
if err != nil {
return ""
}
return string(b)
}
return readLine(in)
}
func readLine(in io.Reader) string {
var b strings.Builder
buf := make([]byte, 1)
for {
n, err := in.Read(buf)
if n > 0 {
if buf[0] == '\n' {
break
}
if buf[0] != '\r' {
b.WriteByte(buf[0])
}
}
if err != nil {
break
}
}
return strings.TrimSpace(b.String())
}

82
PROJECTS/intermediate/security-news-scraper/internal/setup/wizard_test.go Normal file
View File

@ -0,0 +1,82 @@
// ©AngelaMos | 2026
// wizard_test.go
package setup
import (
"bytes"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
func TestWizardAPIProvider(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
var out bytes.Buffer
in := strings.NewReader("2\nsk-test-key\n\n")
if err := Run(in, &out); err != nil {
t.Fatalf("Run: %v", err)
}
path, _ := CredentialsPath()
kv, _ := readCredentials(path)
if kv[EnvProvider] != ProviderOpenAI || kv["OPENAI_API_KEY"] != "sk-test-key" {
t.Errorf("saved = %v, want openai + key", kv)
}
if _, ok := kv["NVD_API_KEY"]; ok {
t.Error("NVD key should be absent when skipped")
}
if !strings.Contains(out.String(), "OpenAI is ready") {
t.Errorf("output missing confirmation:\n%s", out.String())
}
if strings.Contains(out.String(), "sk-test-key") {
t.Error("wizard echoed the API key to its output")
}
}
func TestWizardOllamaDetected(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
}))
defer srv.Close()
old := ollamaCandidates
ollamaCandidates = []string{srv.URL}
defer func() { ollamaCandidates = old }()
var out bytes.Buffer
if err := Run(strings.NewReader("4\n"), &out); err != nil {
t.Fatalf("Run: %v", err)
}
path, _ := CredentialsPath()
kv, _ := readCredentials(path)
if kv[EnvProvider] != ProviderQwen || kv[EnvQwenURL] != srv.URL+"/v1" {
t.Errorf("saved = %v, want qwen + %s/v1", kv, srv.URL)
}
if !strings.Contains(out.String(), "Found a running Ollama") {
t.Errorf("output missing detect line:\n%s", out.String())
}
}
func TestWizardInvalidChoice(t *testing.T) {
t.Setenv("XDG_CONFIG_HOME", t.TempDir())
if err := Run(strings.NewReader("9\n"), &bytes.Buffer{}); err == nil {
t.Error("invalid choice should error")
}
}
func TestReadLine(t *testing.T) {
cases := map[string]string{
"hello\n": "hello",
"trim \r\n": "trim",
"noeol": "noeol",
"\n": "",
}
for in, want := range cases {
if got := readLine(strings.NewReader(in)); got != want {
t.Errorf("readLine(%q) = %q, want %q", in, got, want)
}
}
}

92
PROJECTS/intermediate/security-news-scraper/internal/source/source.go Normal file
View File

@ -0,0 +1,92 @@
// ©AngelaMos | 2026
// source.go
package source
import (
_ "embed"
"fmt"
"net/url"
"os"
"gopkg.in/yaml.v3"
)
//go:embed sources.yaml
var embedded []byte
type Kind string
const (
KindRSS Kind = "rss"
KindAtom Kind = "atom"
KindHTML Kind = "html"
)
type Source struct {
Name string `yaml:"name"`
Title string `yaml:"title"`
URL string `yaml:"url"`
Type Kind `yaml:"type"`
Extractor string `yaml:"extractor"`
Weight float64 `yaml:"weight"`
Tags []string `yaml:"tags"`
Enabled bool `yaml:"enabled"`
}
func Defaults() ([]Source, error) {
return parse(embedded)
}
func Load(path string) ([]Source, error) {
if path == "" {
return Defaults()
}
raw, err := os.ReadFile(path)
if err != nil {
if os.IsNotExist(err) {
return Defaults()
}
return nil, fmt.Errorf("read sources %s: %w", path, err)
}
return parse(raw)
}
func parse(raw []byte) ([]Source, error) {
var out []Source
if err := yaml.Unmarshal(raw, &out); err != nil {
return nil, fmt.Errorf("parse sources: %w", err)
}
if len(out) == 0 {
return nil, fmt.Errorf("sources: no entries")
}
seen := make(map[string]struct{}, len(out))
for i, s := range out {
if s.Name == "" {
return nil, fmt.Errorf("sources[%d]: name is required", i)
}
if _, dup := seen[s.Name]; dup {
return nil, fmt.Errorf("sources: duplicate name %q", s.Name)
}
seen[s.Name] = struct{}{}
if _, err := url.ParseRequestURI(s.URL); err != nil {
return nil, fmt.Errorf("sources[%s]: invalid url %q: %w", s.Name, s.URL, err)
}
switch s.Type {
case KindRSS, KindAtom, KindHTML:
default:
return nil, fmt.Errorf("sources[%s]: type must be rss|atom|html, got %q", s.Name, s.Type)
}
}
return out, nil
}
func Enabled(all []Source) []Source {
out := make([]Source, 0, len(all))
for _, s := range all {
if s.Enabled {
out = append(out, s)
}
}
return out
}

103
PROJECTS/intermediate/security-news-scraper/internal/source/source_test.go Normal file
View File

@ -0,0 +1,103 @@
// ©AngelaMos | 2026
// source_test.go
package source
import (
"os"
"path/filepath"
"testing"
)
func TestEmbeddedDefaultsParse(t *testing.T) {
got, err := Defaults()
if err != nil {
t.Fatalf("Defaults: %v", err)
}
want := map[string]bool{
"krebs": false, "thehackernews": false, "bleepingcomputer": false,
"securityweek": false, "darkreading": false, "theregister": false,
"cisa": false,
}
for _, s := range got {
if _, ok := want[s.Name]; !ok {
continue
}
want[s.Name] = true
if s.Type != KindRSS {
t.Errorf("%s: type = %q, want rss", s.Name, s.Type)
}
if s.Weight <= 0 || s.Weight > 1 {
t.Errorf("%s: weight %v out of (0,1]", s.Name, s.Weight)
}
if !s.Enabled {
t.Errorf("%s: expected enabled by default", s.Name)
}
}
for name, seen := range want {
if !seen {
t.Errorf("seed source %q missing from embedded defaults", name)
}
}
}
func TestExternalOverride(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "sources.yaml")
body := `
- name: custom
title: Custom Feed
url: https://example.com/feed.xml
type: rss
weight: 0.5
enabled: true
`
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
got, err := Load(path)
if err != nil {
t.Fatalf("Load: %v", err)
}
if len(got) != 1 || got[0].Name != "custom" {
t.Errorf("external override = %+v", got)
}
}
func TestRejectsBadURL(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "bad.yaml")
body := "- name: x\n url: not-a-url\n type: rss\n enabled: true\n"
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
if _, err := Load(path); err == nil {
t.Error("expected error for invalid url")
}
}
func TestRejectsDuplicateName(t *testing.T) {
dir := t.TempDir()
path := filepath.Join(dir, "dup.yaml")
body := `
- {name: a, url: "https://a.com/f", type: rss, enabled: true}
- {name: a, url: "https://b.com/f", type: rss, enabled: true}
`
if err := os.WriteFile(path, []byte(body), 0o600); err != nil {
t.Fatal(err)
}
if _, err := Load(path); err == nil {
t.Error("expected error for duplicate source name")
}
}
func TestEnabledFilter(t *testing.T) {
all := []Source{
{Name: "on", Enabled: true},
{Name: "off", Enabled: false},
}
got := Enabled(all)
if len(got) != 1 || got[0].Name != "on" {
t.Errorf("Enabled filter = %+v", got)
}
}

58
PROJECTS/intermediate/security-news-scraper/internal/source/sources.yaml Normal file
View File

@ -0,0 +1,58 @@
# ©AngelaMos | 2026
# sources.yaml
- name: krebs
title: Krebs on Security
url: https://krebsonsecurity.com/feed/
type: rss
weight: 1.0
tags: [news]
enabled: true
- name: thehackernews
title: The Hacker News
url: https://feeds.feedburner.com/TheHackersNews
type: rss
weight: 0.8
tags: [news]
enabled: true
- name: bleepingcomputer
title: BleepingComputer
url: https://www.bleepingcomputer.com/feed/
type: rss
weight: 0.9
tags: [news]
enabled: true
- name: securityweek
title: SecurityWeek
url: https://www.securityweek.com/feed/
type: rss
weight: 0.8
tags: [news]
enabled: true
- name: darkreading
title: Dark Reading
url: https://www.darkreading.com/rss.xml
type: rss
weight: 0.8
tags: [news]
enabled: true
- name: theregister
title: The Register (Security)
url: https://www.theregister.com/security/headlines.atom
type: rss
weight: 0.8
tags: [news]
enabled: true
- name: cisa
title: CISA Advisories
url: https://www.cisa.gov/cybersecurity-advisories/all.xml
type: rss
weight: 1.0
tags: [advisory, gov]
enabled: true

92
PROJECTS/intermediate/security-news-scraper/internal/store/ai_notes.go Normal file
View File

@ -0,0 +1,92 @@
// ©AngelaMos | 2026
// ai_notes.go
package store
import (
"database/sql"
"errors"
"fmt"
)
type AINote struct {
ID int64
ClusterID int64
Provider string
Summary string
Why string
AnglesJSON string
Format string
CreatedAt int64
}
func (s *Store) InsertAINote(n AINote) error {
_, err := s.db.Exec(`
INSERT INTO ai_notes (cluster_id, provider, summary, why, angles_json, format, created_at)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(cluster_id, provider) DO UPDATE SET
summary = excluded.summary,
why = excluded.why,
angles_json = excluded.angles_json,
format = excluded.format,
created_at = excluded.created_at`,
n.ClusterID, n.Provider, n.Summary, n.Why, n.AnglesJSON, n.Format, n.CreatedAt,
)
if err != nil {
return fmt.Errorf("insert ai_note cluster=%d provider=%q: %w", n.ClusterID, n.Provider, err)
}
return nil
}
func (s *Store) AINoteExists(clusterID int64, provider string) (bool, error) {
var one int
err := s.db.QueryRow(
`SELECT 1 FROM ai_notes WHERE cluster_id = ? AND provider = ? LIMIT 1`,
clusterID, provider,
).Scan(&one)
if errors.Is(err, sql.ErrNoRows) {
return false, nil
}
if err != nil {
return false, fmt.Errorf("ai_note exists cluster=%d provider=%q: %w", clusterID, provider, err)
}
return true, nil
}
func (s *Store) AINotesForCluster(clusterID int64) ([]AINote, error) {
rows, err := s.db.Query(`
SELECT id, cluster_id, provider, summary, why, angles_json, format, created_at
FROM ai_notes WHERE cluster_id = ? ORDER BY provider`, clusterID)
if err != nil {
return nil, fmt.Errorf("ai_notes for cluster %d: %w", clusterID, err)
}
defer rows.Close()
var out []AINote
for rows.Next() {
var n AINote
if err := rows.Scan(&n.ID, &n.ClusterID, &n.Provider, &n.Summary, &n.Why, &n.AnglesJSON, &n.Format, &n.CreatedAt); err != nil {
return nil, fmt.Errorf("ai_notes for cluster %d: scan: %w", clusterID, err)
}
out = append(out, n)
}
return out, rows.Err()
}
func (s *Store) LatestAINotes() (map[int64]AINote, error) {
rows, err := s.db.Query(`
SELECT id, cluster_id, provider, summary, why, angles_json, format, created_at
FROM ai_notes ORDER BY cluster_id, created_at`)
if err != nil {
return nil, fmt.Errorf("latest ai_notes: %w", err)
}
defer rows.Close()
out := make(map[int64]AINote)
for rows.Next() {
var n AINote
if err := rows.Scan(&n.ID, &n.ClusterID, &n.Provider, &n.Summary, &n.Why, &n.AnglesJSON, &n.Format, &n.CreatedAt); err != nil {
return nil, fmt.Errorf("latest ai_notes: scan: %w", err)
}
out[n.ClusterID] = n
}
return out, rows.Err()
}

127
PROJECTS/intermediate/security-news-scraper/internal/store/ai_notes_test.go Normal file
View File

@ -0,0 +1,127 @@
// ©AngelaMos | 2026
// ai_notes_test.go
package store
import "testing"
func insertTestCluster(t *testing.T, s *Store, id int64, key string) {
t.Helper()
_, err := s.DB().Exec(
`INSERT INTO clusters (id, cluster_key, first_seen, last_seen, size) VALUES (?, ?, 0, 0, 1)`,
id, key,
)
if err != nil {
t.Fatalf("insert cluster: %v", err)
}
}
func TestAINoteRoundTrip(t *testing.T) {
s := openTemp(t)
insertTestCluster(t, s, 1, "k1")
note := AINote{
ClusterID: 1, Provider: "qwen",
Summary: "s", Why: "w", AnglesJSON: `["a","b"]`, Format: "blog", CreatedAt: 100,
}
if err := s.InsertAINote(note); err != nil {
t.Fatalf("InsertAINote: %v", err)
}
ok, err := s.AINoteExists(1, "qwen")
if err != nil || !ok {
t.Fatalf("AINoteExists(1,qwen) = %v, %v; want true", ok, err)
}
if ok, _ := s.AINoteExists(1, "openai"); ok {
t.Error("AINoteExists(1,openai) = true, want false")
}
if ok, _ := s.AINoteExists(2, "qwen"); ok {
t.Error("AINoteExists(2,qwen) = true, want false")
}
notes, err := s.AINotesForCluster(1)
if err != nil || len(notes) != 1 {
t.Fatalf("AINotesForCluster = %v, %v; want 1 note", notes, err)
}
if notes[0].Summary != "s" || notes[0].Format != "blog" || notes[0].AnglesJSON != `["a","b"]` {
t.Errorf("note = %+v", notes[0])
}
}
func TestAINoteUpsertOverwrites(t *testing.T) {
s := openTemp(t)
insertTestCluster(t, s, 1, "k1")
if err := s.InsertAINote(AINote{ClusterID: 1, Provider: "qwen", Summary: "first", AnglesJSON: "[]", Format: "blog", CreatedAt: 1}); err != nil {
t.Fatal(err)
}
if err := s.InsertAINote(AINote{ClusterID: 1, Provider: "qwen", Summary: "second", AnglesJSON: "[]", Format: "video", CreatedAt: 2}); err != nil {
t.Fatal(err)
}
notes, err := s.AINotesForCluster(1)
if err != nil {
t.Fatal(err)
}
if len(notes) != 1 {
t.Fatalf("got %d notes, want 1 (upsert should overwrite)", len(notes))
}
if notes[0].Summary != "second" || notes[0].Format != "video" || notes[0].CreatedAt != 2 {
t.Errorf("upsert did not overwrite: %+v", notes[0])
}
}
func TestLatestAINotesNewestPerCluster(t *testing.T) {
s := openTemp(t)
insertTestCluster(t, s, 1, "k1")
insertTestCluster(t, s, 2, "k2")
must := func(err error) {
t.Helper()
if err != nil {
t.Fatal(err)
}
}
must(s.InsertAINote(AINote{ClusterID: 1, Provider: "qwen", Summary: "old", AnglesJSON: "[]", Format: "blog", CreatedAt: 10}))
must(s.InsertAINote(AINote{ClusterID: 1, Provider: "anthropic", Summary: "new", AnglesJSON: "[]", Format: "video", CreatedAt: 20}))
must(s.InsertAINote(AINote{ClusterID: 2, Provider: "qwen", Summary: "two", AnglesJSON: "[]", Format: "blog", CreatedAt: 5}))
notes, err := s.LatestAINotes()
if err != nil {
t.Fatal(err)
}
if len(notes) != 2 {
t.Fatalf("got %d clusters, want 2", len(notes))
}
if notes[1].Summary != "new" {
t.Errorf("cluster 1 latest = %q, want new (highest created_at)", notes[1].Summary)
}
if notes[2].Summary != "two" {
t.Errorf("cluster 2 = %q, want two", notes[2].Summary)
}
}
func TestAINoteForeignKeyRejectsOrphan(t *testing.T) {
s := openTemp(t)
err := s.InsertAINote(AINote{ClusterID: 99, Provider: "qwen", AnglesJSON: "[]", Format: "blog", CreatedAt: 1})
if err == nil {
t.Error("insert for a nonexistent cluster should be rejected by the foreign key")
}
}
func TestAINoteCascadesOnClusterDelete(t *testing.T) {
s := openTemp(t)
insertTestCluster(t, s, 1, "k1")
if err := s.InsertAINote(AINote{ClusterID: 1, Provider: "qwen", AnglesJSON: "[]", Format: "blog", CreatedAt: 1}); err != nil {
t.Fatal(err)
}
if _, err := s.DB().Exec(`DELETE FROM clusters WHERE id = 1`); err != nil {
t.Fatal(err)
}
notes, err := s.AINotesForCluster(1)
if err != nil {
t.Fatal(err)
}
if len(notes) != 0 {
t.Errorf("notes should cascade on cluster delete, %d remain", len(notes))
}
}

106
PROJECTS/intermediate/security-news-scraper/internal/store/cve_test.go Normal file
View File

@ -0,0 +1,106 @@
// ©AngelaMos | 2026
// cve_test.go
package store
import "testing"
func seedArticleWithCVE(t *testing.T, s *Store, cveID string, cvss float64, kev bool) int64 {
t.Helper()
sourceID, err := s.UpsertSource(SourceInput{
Name: "src", URL: "https://ex.example/feed", Type: "rss", Weight: 1, Enabled: true,
})
if err != nil {
t.Fatal(err)
}
articleID, err := s.InsertArticle(Article{
SourceID: sourceID, CanonicalURL: "https://ex.example/a", ContentHash: "ch", TitleHash: "th",
Title: "Exploit in the wild", Summary: "details", PublishedAt: 1000, FetchedAt: 1000,
})
if err != nil {
t.Fatal(err)
}
if err := s.UpsertCVEStub(cveID); err != nil {
t.Fatal(err)
}
if err := s.LinkArticleCVE(articleID, cveID); err != nil {
t.Fatal(err)
}
score := cvss
if err := s.UpdateCVEEnrichment(CVE{
ID: cveID, CVSSScore: &score, CVSSSeverity: "CRITICAL", IsKEV: kev,
EnrichedAt: 2000, EnrichStatus: EnrichStatusOK,
}); err != nil {
t.Fatal(err)
}
return articleID
}
func TestArticlesForCVE(t *testing.T) {
s := openTemp(t)
seedArticleWithCVE(t, s, "CVE-2026-1", 9.8, true)
got, err := s.ArticlesForCVE("CVE-2026-1")
if err != nil {
t.Fatal(err)
}
if len(got) != 1 || got[0].Title != "Exploit in the wild" {
t.Errorf("ArticlesForCVE = %+v, want one article", got)
}
}
func TestLinkArticleCVEIsIdempotent(t *testing.T) {
s := openTemp(t)
id := seedArticleWithCVE(t, s, "CVE-2026-1", 5.0, false)
if err := s.LinkArticleCVE(id, "CVE-2026-1"); err != nil {
t.Fatalf("re-link should be a no-op: %v", err)
}
got, _ := s.ArticlesForCVE("CVE-2026-1")
if len(got) != 1 {
t.Errorf("duplicate link produced %d rows, want 1", len(got))
}
}
func TestListArticlesFilters(t *testing.T) {
s := openTemp(t)
seedArticleWithCVE(t, s, "CVE-2026-1", 9.8, true)
if got, _ := s.ListArticles(ListFilter{KEV: true}); len(got) != 1 {
t.Errorf("--kev returned %d, want 1", len(got))
}
if got, _ := s.ListArticles(ListFilter{MinCVSS: 9.0}); len(got) != 1 {
t.Errorf("--min-cvss 9 returned %d, want 1", len(got))
}
if got, _ := s.ListArticles(ListFilter{MinCVSS: 10.0}); len(got) != 0 {
t.Errorf("--min-cvss 10 returned %d, want 0", len(got))
}
if got, _ := s.ListArticles(ListFilter{Keyword: "exploit"}); len(got) != 1 {
t.Errorf("--keyword returned %d, want 1", len(got))
}
if got, _ := s.ListArticles(ListFilter{Source: "nope"}); len(got) != 0 {
t.Errorf("unknown source returned %d, want 0", len(got))
}
}
func TestCVEsNeedingEnrichment(t *testing.T) {
s := openTemp(t)
if err := s.UpsertCVEStub("CVE-2026-1"); err != nil {
t.Fatal(err)
}
need, err := s.CVEsNeedingEnrichment(10000, 3600, 1800)
if err != nil {
t.Fatal(err)
}
if len(need) != 1 {
t.Fatalf("fresh stub should need enrichment, got %d", len(need))
}
score := 5.0
if err := s.UpdateCVEEnrichment(CVE{ID: "CVE-2026-1", CVSSScore: &score, EnrichedAt: 9000, EnrichStatus: EnrichStatusOK}); err != nil {
t.Fatal(err)
}
if need, _ := s.CVEsNeedingEnrichment(10000, 3600, 1800); len(need) != 0 {
t.Errorf("recently enriched cve should be skipped, got %d", len(need))
}
if need, _ := s.CVEsNeedingEnrichment(99999, 3600, 1800); len(need) != 1 {
t.Errorf("stale cve (past TTL) should need re-enrichment, got %d", len(need))
}
}

110
PROJECTS/intermediate/security-news-scraper/internal/store/digest_test.go Normal file
View File

@ -0,0 +1,110 @@
// ©AngelaMos | 2026
// digest_test.go
package store
import "testing"
func TestDigestClustersAggregates(t *testing.T) {
s := openTemp(t)
src1, _ := s.UpsertSource(SourceInput{Name: "a", URL: "https://a/f", Type: "rss", Weight: 1.0, Enabled: true})
src2, _ := s.UpsertSource(SourceInput{Name: "b", URL: "https://b/f", Type: "rss", Weight: 0.8, Enabled: true})
a1, _ := s.InsertArticle(Article{SourceID: src1, CanonicalURL: "https://a/1", ContentHash: "c1", TitleHash: "t1", Title: "Story one", PublishedAt: 1000})
a2, _ := s.InsertArticle(Article{SourceID: src2, CanonicalURL: "https://b/1", ContentHash: "c2", TitleHash: "t2", Title: "Story two", PublishedAt: 2000})
if err := s.UpsertCVEStub("CVE-2026-1"); err != nil {
t.Fatal(err)
}
score := 9.8
if err := s.UpdateCVEEnrichment(CVE{ID: "CVE-2026-1", CVSSScore: &score, IsKEV: true, EnrichedAt: 1, EnrichStatus: EnrichStatusOK}); err != nil {
t.Fatal(err)
}
if err := s.LinkArticleCVE(a1, "CVE-2026-1"); err != nil {
t.Fatal(err)
}
if err := s.LinkArticleCVE(a2, "CVE-2026-1"); err != nil {
t.Fatal(err)
}
if err := s.ReplaceClusters([]ClusterRow{
{Key: "1", Members: []int64{a1, a2}, FirstSeen: 1000, LastSeen: 2000},
}); err != nil {
t.Fatal(err)
}
clusters, err := s.DigestClusters(0)
if err != nil {
t.Fatal(err)
}
if len(clusters) != 1 {
t.Fatalf("clusters = %d, want 1", len(clusters))
}
c := clusters[0]
if len(c.Articles) != 2 {
t.Errorf("articles = %d, want 2", len(c.Articles))
}
if len(c.CVEs) != 1 {
t.Fatalf("cves = %d, want 1 (deduped across both articles)", len(c.CVEs))
}
if !c.CVEs[0].IsKEV || c.CVEs[0].CVSSScore == nil || *c.CVEs[0].CVSSScore != 9.8 {
t.Errorf("cve signals not aggregated: %+v", c.CVEs[0])
}
}
func TestDigestClustersAttachToCorrectCluster(t *testing.T) {
s := openTemp(t)
src, _ := s.UpsertSource(SourceInput{Name: "a", URL: "https://a/f", Type: "rss", Weight: 1, Enabled: true})
a1, _ := s.InsertArticle(Article{SourceID: src, CanonicalURL: "https://a/1", ContentHash: "c1", TitleHash: "t1", Title: "one"})
a2, _ := s.InsertArticle(Article{SourceID: src, CanonicalURL: "https://a/2", ContentHash: "c2", TitleHash: "t2", Title: "two"})
for _, id := range []string{"CVE-2026-1", "CVE-2026-2"} {
if err := s.UpsertCVEStub(id); err != nil {
t.Fatal(err)
}
}
_ = s.LinkArticleCVE(a1, "CVE-2026-1")
_ = s.LinkArticleCVE(a2, "CVE-2026-2")
if err := s.ReplaceClusters([]ClusterRow{
{Key: "1", Members: []int64{a1}, FirstSeen: 1000, LastSeen: 1000},
{Key: "2", Members: []int64{a2}, FirstSeen: 2000, LastSeen: 2000},
}); err != nil {
t.Fatal(err)
}
clusters, err := s.DigestClusters(0)
if err != nil {
t.Fatal(err)
}
if len(clusters) != 2 {
t.Fatalf("clusters = %d, want 2", len(clusters))
}
byArticle := map[int64]DigestCluster{}
for _, c := range clusters {
if len(c.Articles) != 1 || len(c.CVEs) != 1 {
t.Fatalf("cluster %d: articles=%d cves=%d, want 1/1", c.ClusterID, len(c.Articles), len(c.CVEs))
}
byArticle[c.Articles[0].ID] = c
}
if byArticle[a1].CVEs[0].ID != "CVE-2026-1" || byArticle[a2].CVEs[0].ID != "CVE-2026-2" {
t.Error("CVEs attached to the wrong cluster")
}
}
func TestDigestClustersSinceFilter(t *testing.T) {
s := openTemp(t)
src, _ := s.UpsertSource(SourceInput{Name: "a", URL: "https://a/f", Type: "rss", Weight: 1, Enabled: true})
a1, _ := s.InsertArticle(Article{SourceID: src, CanonicalURL: "https://a/1", ContentHash: "c1", TitleHash: "t1", Title: "old"})
if err := s.ReplaceClusters([]ClusterRow{{Key: "1", Members: []int64{a1}, FirstSeen: 500, LastSeen: 500}}); err != nil {
t.Fatal(err)
}
if got, _ := s.DigestClusters(1000); len(got) != 0 {
t.Errorf("cluster with last_seen 500 should be excluded by since=1000, got %d", len(got))
}
if got, _ := s.DigestClusters(0); len(got) != 1 {
t.Errorf("since=0 should include it, got %d", len(got))
}
}

81
PROJECTS/intermediate/security-news-scraper/internal/store/fetch_state_test.go Normal file
View File

@ -0,0 +1,81 @@
// ©AngelaMos | 2026
// fetch_state_test.go
package store
import "testing"
func seedSource(t *testing.T, s *Store) int64 {
t.Helper()
id, err := s.UpsertSource(SourceInput{
Name: "krebs", Title: "Krebs", URL: "https://krebsonsecurity.com/feed/",
Type: "rss", Weight: 1.0, Tags: []string{"news"}, Enabled: true,
})
if err != nil {
t.Fatalf("UpsertSource: %v", err)
}
return id
}
func TestFetchStateMissing(t *testing.T) {
s := openTemp(t)
id := seedSource(t, s)
fs, ok, err := s.GetFetchState(id)
if err != nil {
t.Fatalf("GetFetchState: %v", err)
}
if ok {
t.Error("expected no fetch_state for fresh source")
}
if fs != (FetchState{}) {
t.Errorf("expected zero FetchState, got %+v", fs)
}
}
func TestFetchStateRoundTrip(t *testing.T) {
s := openTemp(t)
id := seedSource(t, s)
want := FetchState{ETag: `"v1"`, LastModified: "Wed, 01 Jul 2026 00:00:00 GMT", LastFetched: 1700, LastStatus: 200}
if err := s.UpsertFetchState(id, want); err != nil {
t.Fatalf("UpsertFetchState: %v", err)
}
got, ok, err := s.GetFetchState(id)
if err != nil {
t.Fatalf("GetFetchState: %v", err)
}
if !ok {
t.Fatal("expected fetch_state to exist")
}
if got != want {
t.Errorf("round-trip = %+v, want %+v", got, want)
}
updated := FetchState{ETag: `"v2"`, LastModified: "Thu, 02 Jul 2026 00:00:00 GMT", LastFetched: 1800, LastStatus: 304}
if err := s.UpsertFetchState(id, updated); err != nil {
t.Fatalf("second UpsertFetchState: %v", err)
}
got, _, _ = s.GetFetchState(id)
if got != updated {
t.Errorf("after update = %+v, want %+v", got, updated)
}
}
func TestInsertArticleStoresTitleHash(t *testing.T) {
s := openTemp(t)
id := seedSource(t, s)
if _, err := s.InsertArticle(Article{
SourceID: id, CanonicalURL: "https://example.com/a", ContentHash: "hash1",
TitleHash: "thash1", Title: "A",
}); err != nil {
t.Fatalf("InsertArticle: %v", err)
}
var th string
if err := s.DB().QueryRow(`SELECT title_hash FROM articles WHERE content_hash = ?`, "hash1").Scan(&th); err != nil {
t.Fatal(err)
}
if th != "thash1" {
t.Errorf("title_hash = %q, want thash1", th)
}
}

103
PROJECTS/intermediate/security-news-scraper/internal/store/migrate.go Normal file
View File

@ -0,0 +1,103 @@
// ©AngelaMos | 2026
// migrate.go
package store
import (
"database/sql"
"embed"
"fmt"
"io/fs"
"sort"
"strconv"
"strings"
)
//go:embed migrations/*.sql
var migrationsFS embed.FS
type migration struct {
version int
name string
sql string
}
func loadMigrations() ([]migration, error) {
entries, err := fs.ReadDir(migrationsFS, "migrations")
if err != nil {
return nil, fmt.Errorf("read migrations dir: %w", err)
}
out := make([]migration, 0, len(entries))
for _, e := range entries {
if e.IsDir() || !strings.HasSuffix(e.Name(), ".sql") {
continue
}
verStr, _, ok := strings.Cut(e.Name(), "_")
if !ok {
return nil, fmt.Errorf("migration %q: expected NNNN_name.sql", e.Name())
}
ver, err := strconv.Atoi(verStr)
if err != nil {
return nil, fmt.Errorf("migration %q: bad version prefix: %w", e.Name(), err)
}
body, err := migrationsFS.ReadFile("migrations/" + e.Name())
if err != nil {
return nil, fmt.Errorf("read migration %q: %w", e.Name(), err)
}
out = append(out, migration{version: ver, name: e.Name(), sql: string(body)})
}
sort.Slice(out, func(i, j int) bool { return out[i].version < out[j].version })
return out, nil
}
func migrate(db *sql.DB) (int, error) {
if _, err := db.Exec(`CREATE TABLE IF NOT EXISTS schema_migrations (
version INTEGER PRIMARY KEY,
applied_at INTEGER NOT NULL DEFAULT (strftime('%s','now'))
)`); err != nil {
return 0, fmt.Errorf("create schema_migrations: %w", err)
}
var current int
if err := db.QueryRow(`SELECT COALESCE(MAX(version), 0) FROM schema_migrations`).Scan(&current); err != nil {
return 0, fmt.Errorf("read current version: %w", err)
}
migrations, err := loadMigrations()
if err != nil {
return current, err
}
maxKnown := 0
for _, m := range migrations {
if m.version > maxKnown {
maxKnown = m.version
}
}
if current > maxKnown {
return current, fmt.Errorf("store schema version %d is newer than this binary supports (max %d); refusing to run against a store written by a newer build", current, maxKnown)
}
for _, m := range migrations {
if m.version <= current {
continue
}
tx, err := db.Begin()
if err != nil {
return current, fmt.Errorf("begin migration %d: %w", m.version, err)
}
if _, err := tx.Exec(m.sql); err != nil {
_ = tx.Rollback()
return current, fmt.Errorf("apply migration %q: %w", m.name, err)
}
if _, err := tx.Exec(`INSERT INTO schema_migrations(version) VALUES (?)`, m.version); err != nil {
_ = tx.Rollback()
return current, fmt.Errorf("record migration %d: %w", m.version, err)
}
if err := tx.Commit(); err != nil {
return current, fmt.Errorf("commit migration %d: %w", m.version, err)
}
current = m.version
}
return current, nil
}

87
PROJECTS/intermediate/security-news-scraper/internal/store/migrations/0001_init.sql Normal file
View File

@ -0,0 +1,87 @@
-- ©AngelaMos | 2026
-- 0001_init.sql
CREATE TABLE sources (
id INTEGER PRIMARY KEY,
name TEXT NOT NULL UNIQUE,
title TEXT NOT NULL DEFAULT '',
url TEXT NOT NULL,
type TEXT NOT NULL,
weight REAL NOT NULL DEFAULT 1.0,
tags TEXT NOT NULL DEFAULT '',
enabled INTEGER NOT NULL DEFAULT 1
);
CREATE TABLE fetch_state (
source_id INTEGER PRIMARY KEY REFERENCES sources(id) ON DELETE CASCADE,
etag TEXT NOT NULL DEFAULT '',
last_modified TEXT NOT NULL DEFAULT '',
last_fetched INTEGER NOT NULL DEFAULT 0,
last_status INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE articles (
id INTEGER PRIMARY KEY,
source_id INTEGER NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
canonical_url TEXT NOT NULL UNIQUE,
content_hash TEXT NOT NULL UNIQUE,
title TEXT NOT NULL DEFAULT '',
summary TEXT NOT NULL DEFAULT '',
body TEXT NOT NULL DEFAULT '',
author TEXT NOT NULL DEFAULT '',
published_at INTEGER NOT NULL DEFAULT 0,
fetched_at INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX idx_articles_published ON articles(published_at);
CREATE INDEX idx_articles_source ON articles(source_id);
CREATE TABLE cves (
id TEXT PRIMARY KEY,
description TEXT NOT NULL DEFAULT '',
cvss_score REAL,
cvss_version TEXT NOT NULL DEFAULT '',
cvss_severity TEXT NOT NULL DEFAULT '',
cvss_vector TEXT NOT NULL DEFAULT '',
cwe TEXT NOT NULL DEFAULT '',
is_kev INTEGER NOT NULL DEFAULT 0,
kev_date_added TEXT NOT NULL DEFAULT '',
kev_ransomware INTEGER NOT NULL DEFAULT 0,
epss REAL,
epss_percentile REAL,
nvd_published TEXT NOT NULL DEFAULT '',
nvd_modified TEXT NOT NULL DEFAULT '',
enriched_at INTEGER NOT NULL DEFAULT 0,
enrich_status TEXT NOT NULL DEFAULT ''
);
CREATE TABLE article_cves (
article_id INTEGER NOT NULL REFERENCES articles(id) ON DELETE CASCADE,
cve_id TEXT NOT NULL REFERENCES cves(id) ON DELETE CASCADE,
PRIMARY KEY (article_id, cve_id)
);
CREATE TABLE clusters (
id INTEGER PRIMARY KEY,
cluster_key TEXT NOT NULL UNIQUE,
first_seen INTEGER NOT NULL DEFAULT 0,
last_seen INTEGER NOT NULL DEFAULT 0,
size INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE cluster_members (
cluster_id INTEGER NOT NULL REFERENCES clusters(id) ON DELETE CASCADE,
article_id INTEGER NOT NULL REFERENCES articles(id) ON DELETE CASCADE,
PRIMARY KEY (cluster_id, article_id)
);
CREATE TABLE ai_notes (
id INTEGER PRIMARY KEY,
cluster_id INTEGER NOT NULL REFERENCES clusters(id) ON DELETE CASCADE,
provider TEXT NOT NULL DEFAULT '',
summary TEXT NOT NULL DEFAULT '',
why TEXT NOT NULL DEFAULT '',
angles_json TEXT NOT NULL DEFAULT '',
format TEXT NOT NULL DEFAULT '',
created_at INTEGER NOT NULL DEFAULT 0
);

6
PROJECTS/intermediate/security-news-scraper/internal/store/migrations/0002_article_title_hash.sql Normal file
View File

@ -0,0 +1,6 @@
-- ©AngelaMos | 2026
-- 0002_article_title_hash.sql
ALTER TABLE articles ADD COLUMN title_hash TEXT NOT NULL DEFAULT '';
CREATE INDEX idx_articles_title_hash ON articles(title_hash);

4
PROJECTS/intermediate/security-news-scraper/internal/store/migrations/0003_ai_notes_provider_unique.sql Normal file
View File

@ -0,0 +1,4 @@
-- ©AngelaMos | 2026
-- 0003_ai_notes_provider_unique.sql
CREATE UNIQUE INDEX idx_ai_notes_cluster_provider ON ai_notes(cluster_id, provider);

593
PROJECTS/intermediate/security-news-scraper/internal/store/store.go Normal file
View File

@ -0,0 +1,593 @@
// ©AngelaMos | 2026
// store.go
package store
import (
"database/sql"
"errors"
"fmt"
"strings"
"modernc.org/sqlite"
sqlite3 "modernc.org/sqlite/lib"
)
var ErrDuplicate = errors.New("store: article already exists")
const (
EnrichStatusOK = "ok"
EnrichStatusNotFound = "not_found"
)
type Store struct {
db *sql.DB
version int
}
type SourceInput struct {
Name string
Title string
URL string
Type string
Weight float64
Tags []string
Enabled bool
}
type SourceRow struct {
ID int64
Name string
Title string
URL string
Type string
Weight float64
Tags []string
Enabled bool
}
type Article struct {
SourceID int64
CanonicalURL string
ContentHash string
TitleHash string
Title string
Summary string
Body string
Author string
PublishedAt int64
FetchedAt int64
}
type FetchState struct {
ETag string
LastModified string
LastFetched int64
LastStatus int64
}
func Open(path string) (*Store, error) {
dsn := fmt.Sprintf("file:%s?_pragma=busy_timeout(5000)&_pragma=journal_mode(WAL)&_pragma=foreign_keys(1)", path)
db, err := sql.Open("sqlite", dsn)
if err != nil {
return nil, fmt.Errorf("open sqlite %s: %w", path, err)
}
if err := db.Ping(); err != nil {
_ = db.Close()
return nil, fmt.Errorf("ping sqlite %s: %w", path, err)
}
version, err := migrate(db)
if err != nil {
_ = db.Close()
return nil, err
}
return &Store{db: db, version: version}, nil
}
func (s *Store) Close() error { return s.db.Close() }
func (s *Store) Version() int { return s.version }
func (s *Store) DB() *sql.DB { return s.db }
func (s *Store) UpsertSource(in SourceInput) (int64, error) {
tags := strings.Join(in.Tags, ",")
var id int64
err := s.db.QueryRow(`
INSERT INTO sources (name, title, url, type, weight, tags, enabled)
VALUES (?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(name) DO UPDATE SET
title = excluded.title, url = excluded.url, type = excluded.type,
weight = excluded.weight, tags = excluded.tags, enabled = excluded.enabled
RETURNING id`,
in.Name, in.Title, in.URL, in.Type, in.Weight, tags, boolToInt(in.Enabled),
).Scan(&id)
if err != nil {
return 0, fmt.Errorf("upsert source %q: %w", in.Name, err)
}
return id, nil
}
func (s *Store) GetSourceByName(name string) (SourceRow, error) {
var r SourceRow
var tags string
var enabled int
err := s.db.QueryRow(`
SELECT id, name, title, url, type, weight, tags, enabled
FROM sources WHERE name = ?`, name,
).Scan(&r.ID, &r.Name, &r.Title, &r.URL, &r.Type, &r.Weight, &tags, &enabled)
if err != nil {
return SourceRow{}, fmt.Errorf("get source %q: %w", name, err)
}
if tags != "" {
r.Tags = strings.Split(tags, ",")
}
r.Enabled = enabled != 0
return r, nil
}
func (s *Store) InsertArticle(a Article) (int64, error) {
res, err := s.db.Exec(`
INSERT INTO articles
(source_id, canonical_url, content_hash, title_hash, title, summary, body, author, published_at, fetched_at)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)`,
a.SourceID, a.CanonicalURL, a.ContentHash, a.TitleHash, a.Title, a.Summary, a.Body,
a.Author, a.PublishedAt, a.FetchedAt,
)
if err != nil {
var se *sqlite.Error
if errors.As(err, &se) && se.Code() == sqlite3.SQLITE_CONSTRAINT_UNIQUE {
return 0, ErrDuplicate
}
return 0, fmt.Errorf("insert article %q: %w", a.CanonicalURL, err)
}
id, err := res.LastInsertId()
if err != nil {
return 0, fmt.Errorf("insert article %q: last insert id: %w", a.CanonicalURL, err)
}
return id, nil
}
func (s *Store) CountArticles() (int, error) {
var n int
if err := s.db.QueryRow(`SELECT COUNT(*) FROM articles`).Scan(&n); err != nil {
return 0, fmt.Errorf("count articles: %w", err)
}
return n, nil
}
func (s *Store) GetFetchState(sourceID int64) (FetchState, bool, error) {
var fs FetchState
err := s.db.QueryRow(`
SELECT etag, last_modified, last_fetched, last_status
FROM fetch_state WHERE source_id = ?`, sourceID,
).Scan(&fs.ETag, &fs.LastModified, &fs.LastFetched, &fs.LastStatus)
if errors.Is(err, sql.ErrNoRows) {
return FetchState{}, false, nil
}
if err != nil {
return FetchState{}, false, fmt.Errorf("get fetch_state %d: %w", sourceID, err)
}
return fs, true, nil
}
func (s *Store) UpsertFetchState(sourceID int64, fs FetchState) error {
_, err := s.db.Exec(`
INSERT INTO fetch_state (source_id, etag, last_modified, last_fetched, last_status)
VALUES (?, ?, ?, ?, ?)
ON CONFLICT(source_id) DO UPDATE SET
etag = excluded.etag, last_modified = excluded.last_modified,
last_fetched = excluded.last_fetched, last_status = excluded.last_status`,
sourceID, fs.ETag, fs.LastModified, fs.LastFetched, fs.LastStatus,
)
if err != nil {
return fmt.Errorf("upsert fetch_state %d: %w", sourceID, err)
}
return nil
}
type CVE struct {
ID string
Description string
CVSSScore *float64
CVSSVersion string
CVSSSeverity string
CVSSVector string
CWE string
IsKEV bool
KEVDateAdded string
KEVRansomware bool
EPSS *float64
EPSSPercentile *float64
NVDPublished string
NVDModified string
EnrichedAt int64
EnrichStatus string
}
type ArticleSummary struct {
ID int64
SourceName string
Title string
CanonicalURL string
PublishedAt int64
}
type ListFilter struct {
Source string
Since int64
MinCVSS float64
KEV bool
Keyword string
Limit int
}
func (s *Store) CVEsNeedingEnrichment(now, positiveTTL, negativeTTL int64) ([]string, error) {
rows, err := s.db.Query(`
SELECT id FROM cves
WHERE enriched_at = 0
OR (enrich_status = ? AND enriched_at < ?)
OR (enrich_status = ? AND enriched_at < ?)
ORDER BY id`, EnrichStatusOK, now-positiveTTL, EnrichStatusNotFound, now-negativeTTL)
if err != nil {
return nil, fmt.Errorf("cves needing enrichment: %w", err)
}
defer rows.Close()
var out []string
for rows.Next() {
var id string
if err := rows.Scan(&id); err != nil {
return nil, fmt.Errorf("cves needing enrichment: scan: %w", err)
}
out = append(out, id)
}
return out, rows.Err()
}
func (s *Store) UpdateCVEEnrichment(c CVE) error {
_, err := s.db.Exec(`
UPDATE cves SET
description = ?, cvss_score = ?, cvss_version = ?, cvss_severity = ?, cvss_vector = ?,
cwe = ?, is_kev = ?, kev_date_added = ?, kev_ransomware = ?,
epss = ?, epss_percentile = ?, nvd_published = ?, nvd_modified = ?,
enriched_at = ?, enrich_status = ?
WHERE id = ?`,
c.Description, c.CVSSScore, c.CVSSVersion, c.CVSSSeverity, c.CVSSVector,
c.CWE, boolToInt(c.IsKEV), c.KEVDateAdded, boolToInt(c.KEVRansomware),
c.EPSS, c.EPSSPercentile, c.NVDPublished, c.NVDModified,
c.EnrichedAt, c.EnrichStatus, c.ID,
)
if err != nil {
return fmt.Errorf("update cve enrichment %q: %w", c.ID, err)
}
return nil
}
func (s *Store) GetCVE(id string) (CVE, error) {
var c CVE
var isKEV, ransomware int
err := s.db.QueryRow(`
SELECT id, description, cvss_score, cvss_version, cvss_severity, cvss_vector,
cwe, is_kev, kev_date_added, kev_ransomware, epss, epss_percentile,
nvd_published, nvd_modified, enriched_at, enrich_status
FROM cves WHERE id = ?`, id,
).Scan(&c.ID, &c.Description, &c.CVSSScore, &c.CVSSVersion, &c.CVSSSeverity, &c.CVSSVector,
&c.CWE, &isKEV, &c.KEVDateAdded, &ransomware, &c.EPSS, &c.EPSSPercentile,
&c.NVDPublished, &c.NVDModified, &c.EnrichedAt, &c.EnrichStatus)
if err != nil {
return CVE{}, fmt.Errorf("get cve %q: %w", id, err)
}
c.IsKEV = isKEV != 0
c.KEVRansomware = ransomware != 0
return c, nil
}
func (s *Store) ArticlesForCVE(id string) ([]ArticleSummary, error) {
rows, err := s.db.Query(`
SELECT a.id, s.name, a.title, a.canonical_url, a.published_at
FROM article_cves ac
JOIN articles a ON a.id = ac.article_id
JOIN sources s ON s.id = a.source_id
WHERE ac.cve_id = ?
ORDER BY a.published_at DESC`, id)
if err != nil {
return nil, fmt.Errorf("articles for cve %q: %w", id, err)
}
defer rows.Close()
return scanArticleSummaries(rows)
}
type DigestArticle struct {
ID int64
SourceName string
SourceWeight float64
Title string
CanonicalURL string
PublishedAt int64
}
type DigestCVE struct {
ID string
CVSSScore *float64
EPSS *float64
IsKEV bool
}
type DigestCluster struct {
ClusterID int64
Key string
Size int
FirstSeen int64
LastSeen int64
Articles []DigestArticle
CVEs []DigestCVE
}
func (s *Store) DigestClusters(since int64) ([]DigestCluster, error) {
byID, order, err := s.digestClusterRows(since)
if err != nil {
return nil, err
}
if err := s.digestAttachArticles(byID); err != nil {
return nil, err
}
if err := s.digestAttachCVEs(byID); err != nil {
return nil, err
}
out := make([]DigestCluster, 0, len(order))
for _, id := range order {
out = append(out, *byID[id])
}
return out, nil
}
func (s *Store) digestClusterRows(since int64) (map[int64]*DigestCluster, []int64, error) {
rows, err := s.db.Query(`
SELECT id, cluster_key, size, first_seen, last_seen
FROM clusters WHERE last_seen >= ? ORDER BY id`, since)
if err != nil {
return nil, nil, fmt.Errorf("digest clusters: %w", err)
}
defer rows.Close()
return scanClusterRows(rows)
}
func scanClusterRows(rows *sql.Rows) (map[int64]*DigestCluster, []int64, error) {
byID := make(map[int64]*DigestCluster)
var order []int64
for rows.Next() {
var dc DigestCluster
if err := rows.Scan(&dc.ClusterID, &dc.Key, &dc.Size, &dc.FirstSeen, &dc.LastSeen); err != nil {
return nil, nil, fmt.Errorf("scan cluster rows: %w", err)
}
clone := dc
byID[dc.ClusterID] = &clone
order = append(order, dc.ClusterID)
}
return byID, order, rows.Err()
}
func (s *Store) digestAttachArticles(byID map[int64]*DigestCluster) error {
rows, err := s.db.Query(`
SELECT cm.cluster_id, a.id, s.name, s.weight, a.title, a.canonical_url, a.published_at
FROM cluster_members cm
JOIN articles a ON a.id = cm.article_id
JOIN sources s ON s.id = a.source_id
ORDER BY cm.cluster_id, a.id`)
if err != nil {
return fmt.Errorf("digest articles: %w", err)
}
defer rows.Close()
for rows.Next() {
var clusterID int64
var a DigestArticle
if err := rows.Scan(&clusterID, &a.ID, &a.SourceName, &a.SourceWeight, &a.Title, &a.CanonicalURL, &a.PublishedAt); err != nil {
return fmt.Errorf("digest articles: scan: %w", err)
}
if dc, ok := byID[clusterID]; ok {
dc.Articles = append(dc.Articles, a)
}
}
return rows.Err()
}
func (s *Store) digestAttachCVEs(byID map[int64]*DigestCluster) error {
rows, err := s.db.Query(`
SELECT DISTINCT cm.cluster_id, c.id, c.cvss_score, c.epss, c.is_kev
FROM cluster_members cm
JOIN article_cves ac ON ac.article_id = cm.article_id
JOIN cves c ON c.id = ac.cve_id
ORDER BY cm.cluster_id, c.id`)
if err != nil {
return fmt.Errorf("digest cves: %w", err)
}
defer rows.Close()
for rows.Next() {
var clusterID int64
var v DigestCVE
var isKEV int
if err := rows.Scan(&clusterID, &v.ID, &v.CVSSScore, &v.EPSS, &isKEV); err != nil {
return fmt.Errorf("digest cves: scan: %w", err)
}
v.IsKEV = isKEV != 0
if dc, ok := byID[clusterID]; ok {
dc.CVEs = append(dc.CVEs, v)
}
}
return rows.Err()
}
func (s *Store) UpsertCVEStub(id string) error {
_, err := s.db.Exec(`INSERT INTO cves (id) VALUES (?) ON CONFLICT(id) DO NOTHING`, id)
if err != nil {
return fmt.Errorf("upsert cve %q: %w", id, err)
}
return nil
}
func (s *Store) LinkArticleCVE(articleID int64, cveID string) error {
_, err := s.db.Exec(`
INSERT INTO article_cves (article_id, cve_id) VALUES (?, ?)
ON CONFLICT(article_id, cve_id) DO NOTHING`, articleID, cveID)
if err != nil {
return fmt.Errorf("link article %d cve %q: %w", articleID, cveID, err)
}
return nil
}
type CandidateArticle struct {
ID int64
SourceID int64
Title string
Time int64
}
type ClusterRow struct {
Key string
Members []int64
FirstSeen int64
LastSeen int64
}
func (s *Store) ClusterCandidates(since int64) ([]CandidateArticle, error) {
rows, err := s.db.Query(`
SELECT id, source_id, title, COALESCE(NULLIF(published_at, 0), fetched_at) AS t
FROM articles
WHERE COALESCE(NULLIF(published_at, 0), fetched_at) >= ?
ORDER BY id`, since)
if err != nil {
return nil, fmt.Errorf("cluster candidates: %w", err)
}
defer rows.Close()
var out []CandidateArticle
for rows.Next() {
var c CandidateArticle
if err := rows.Scan(&c.ID, &c.SourceID, &c.Title, &c.Time); err != nil {
return nil, fmt.Errorf("cluster candidates: scan: %w", err)
}
out = append(out, c)
}
return out, rows.Err()
}
func (s *Store) ArticleCVEMap() (map[int64][]string, error) {
rows, err := s.db.Query(`SELECT article_id, cve_id FROM article_cves`)
if err != nil {
return nil, fmt.Errorf("article cve map: %w", err)
}
defer rows.Close()
out := make(map[int64][]string)
for rows.Next() {
var articleID int64
var cveID string
if err := rows.Scan(&articleID, &cveID); err != nil {
return nil, fmt.Errorf("article cve map: scan: %w", err)
}
out[articleID] = append(out[articleID], cveID)
}
return out, rows.Err()
}
func (s *Store) ReplaceClusters(rows []ClusterRow) error {
tx, err := s.db.Begin()
if err != nil {
return fmt.Errorf("replace clusters: begin: %w", err)
}
defer func() { _ = tx.Rollback() }()
if _, err := tx.Exec(`DELETE FROM cluster_members`); err != nil {
return fmt.Errorf("replace clusters: clear members: %w", err)
}
if _, err := tx.Exec(`DELETE FROM clusters`); err != nil {
return fmt.Errorf("replace clusters: clear clusters: %w", err)
}
for _, r := range rows {
var clusterID int64
if err := tx.QueryRow(`
INSERT INTO clusters (cluster_key, first_seen, last_seen, size)
VALUES (?, ?, ?, ?) RETURNING id`,
r.Key, r.FirstSeen, r.LastSeen, len(r.Members),
).Scan(&clusterID); err != nil {
return fmt.Errorf("replace clusters: insert cluster %q: %w", r.Key, err)
}
for _, articleID := range r.Members {
if _, err := tx.Exec(`
INSERT INTO cluster_members (cluster_id, article_id) VALUES (?, ?)`,
clusterID, articleID,
); err != nil {
return fmt.Errorf("replace clusters: insert member %d: %w", articleID, err)
}
}
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("replace clusters: commit: %w", err)
}
return nil
}
func (s *Store) ListArticles(f ListFilter) ([]ArticleSummary, error) {
query := `
SELECT a.id, s.name, a.title, a.canonical_url, a.published_at
FROM articles a
JOIN sources s ON s.id = a.source_id
WHERE 1 = 1`
var args []any
if f.Source != "" {
query += ` AND s.name = ?`
args = append(args, f.Source)
}
if f.Since > 0 {
query += ` AND a.published_at >= ?`
args = append(args, f.Since)
}
if f.Keyword != "" {
query += ` AND (a.title LIKE ? OR a.summary LIKE ?)`
like := "%" + f.Keyword + "%"
args = append(args, like, like)
}
if f.MinCVSS > 0 {
query += ` AND EXISTS (
SELECT 1 FROM article_cves ac JOIN cves c ON c.id = ac.cve_id
WHERE ac.article_id = a.id AND c.cvss_score >= ?)`
args = append(args, f.MinCVSS)
}
if f.KEV {
query += ` AND EXISTS (
SELECT 1 FROM article_cves ac JOIN cves c ON c.id = ac.cve_id
WHERE ac.article_id = a.id AND c.is_kev = 1)`
}
query += ` ORDER BY a.published_at DESC`
if f.Limit > 0 {
query += ` LIMIT ?`
args = append(args, f.Limit)
}
rows, err := s.db.Query(query, args...)
if err != nil {
return nil, fmt.Errorf("list articles: %w", err)
}
defer rows.Close()
return scanArticleSummaries(rows)
}
func scanArticleSummaries(rows *sql.Rows) ([]ArticleSummary, error) {
var out []ArticleSummary
for rows.Next() {
var a ArticleSummary
if err := rows.Scan(&a.ID, &a.SourceName, &a.Title, &a.CanonicalURL, &a.PublishedAt); err != nil {
return nil, fmt.Errorf("scan article summary: %w", err)
}
out = append(out, a)
}
return out, rows.Err()
}
func boolToInt(b bool) int {
if b {
return 1
}
return 0
}

116
PROJECTS/intermediate/security-news-scraper/internal/store/store_test.go Normal file
View File

@ -0,0 +1,116 @@
// ©AngelaMos | 2026
// store_test.go
package store
import (
"errors"
"path/filepath"
"testing"
)
func openTemp(t *testing.T) *Store {
t.Helper()
s, err := Open(filepath.Join(t.TempDir(), "test.db"))
if err != nil {
t.Fatalf("Open: %v", err)
}
t.Cleanup(func() { _ = s.Close() })
return s
}
func TestMigrateAppliesLatest(t *testing.T) {
s := openTemp(t)
if s.Version() < 1 {
t.Fatalf("schema version = %d, want >= 1", s.Version())
}
var n int
if err := s.DB().QueryRow(
`SELECT COUNT(*) FROM sqlite_master WHERE type='table' AND name='articles'`,
).Scan(&n); err != nil {
t.Fatal(err)
}
if n != 1 {
t.Error("articles table not created by migration")
}
}
func TestMigrateIsIdempotent(t *testing.T) {
path := filepath.Join(t.TempDir(), "idem.db")
s1, err := Open(path)
if err != nil {
t.Fatal(err)
}
v1 := s1.Version()
_ = s1.Close()
s2, err := Open(path)
if err != nil {
t.Fatalf("reopen: %v", err)
}
defer s2.Close()
if s2.Version() != v1 {
t.Errorf("reopen version = %d, want %d", s2.Version(), v1)
}
}
func TestSourceRoundTrip(t *testing.T) {
s := openTemp(t)
id, err := s.UpsertSource(SourceInput{
Name: "krebs", Title: "Krebs", URL: "https://krebsonsecurity.com/feed/",
Type: "rss", Weight: 1.0, Tags: []string{"news"}, Enabled: true,
})
if err != nil {
t.Fatalf("UpsertSource: %v", err)
}
row, err := s.GetSourceByName("krebs")
if err != nil {
t.Fatalf("GetSourceByName: %v", err)
}
if row.ID != id || row.Weight != 1.0 || !row.Enabled || len(row.Tags) != 1 {
t.Errorf("round trip mismatch: %+v", row)
}
id2, err := s.UpsertSource(SourceInput{
Name: "krebs", Title: "Krebs Updated", URL: "https://krebsonsecurity.com/feed/",
Type: "rss", Weight: 0.9, Tags: []string{"news"}, Enabled: true,
})
if err != nil {
t.Fatalf("re-upsert: %v", err)
}
if id2 != id {
t.Errorf("upsert should keep id %d, got %d", id, id2)
}
row2, _ := s.GetSourceByName("krebs")
if row2.Title != "Krebs Updated" || row2.Weight != 0.9 {
t.Errorf("upsert did not update fields: %+v", row2)
}
}
func TestArticleUniqueConstraint(t *testing.T) {
s := openTemp(t)
srcID, err := s.UpsertSource(SourceInput{
Name: "krebs", URL: "https://krebsonsecurity.com/feed/", Type: "rss", Enabled: true,
})
if err != nil {
t.Fatal(err)
}
a := Article{
SourceID: srcID, CanonicalURL: "https://krebsonsecurity.com/post-1",
ContentHash: "hash-1", Title: "Post 1", PublishedAt: 100, FetchedAt: 200,
}
if _, err := s.InsertArticle(a); err != nil {
t.Fatalf("first insert: %v", err)
}
if _, err := s.InsertArticle(a); !errors.Is(err, ErrDuplicate) {
t.Errorf("duplicate insert: got %v, want ErrDuplicate", err)
}
n, err := s.CountArticles()
if err != nil {
t.Fatal(err)
}
if n != 1 {
t.Errorf("article count = %d, want 1 (dup rejected)", n)
}
}

76
PROJECTS/intermediate/security-news-scraper/internal/store/watch.go Normal file
View File

@ -0,0 +1,76 @@
// ©AngelaMos | 2026
// watch.go
package store
import (
"fmt"
"strings"
)
func (s *Store) NewlyFetchedClusters(sinceFetched int64) ([]DigestCluster, error) {
ids, err := s.clusterIDsFetchedSince(sinceFetched)
if err != nil {
return nil, err
}
if len(ids) == 0 {
return nil, nil
}
byID, order, err := s.clusterRowsByID(ids)
if err != nil {
return nil, err
}
if err := s.digestAttachArticles(byID); err != nil {
return nil, err
}
if err := s.digestAttachCVEs(byID); err != nil {
return nil, err
}
out := make([]DigestCluster, 0, len(order))
for _, id := range order {
out = append(out, *byID[id])
}
return out, nil
}
func (s *Store) clusterIDsFetchedSince(sinceFetched int64) ([]int64, error) {
rows, err := s.db.Query(`
SELECT DISTINCT cm.cluster_id
FROM cluster_members cm
JOIN articles a ON a.id = cm.article_id
WHERE a.fetched_at >= ?
ORDER BY cm.cluster_id`, sinceFetched)
if err != nil {
return nil, fmt.Errorf("newly fetched clusters: %w", err)
}
defer rows.Close()
var ids []int64
for rows.Next() {
var id int64
if err := rows.Scan(&id); err != nil {
return nil, fmt.Errorf("newly fetched clusters: scan: %w", err)
}
ids = append(ids, id)
}
return ids, rows.Err()
}
func (s *Store) clusterRowsByID(ids []int64) (map[int64]*DigestCluster, []int64, error) {
if len(ids) == 0 {
return map[int64]*DigestCluster{}, nil, nil
}
placeholders := make([]string, len(ids))
args := make([]any, len(ids))
for i, id := range ids {
placeholders[i] = "?"
args[i] = id
}
query := `SELECT id, cluster_key, size, first_seen, last_seen FROM clusters WHERE id IN (` +
strings.Join(placeholders, ",") + `) ORDER BY id`
rows, err := s.db.Query(query, args...)
if err != nil {
return nil, nil, fmt.Errorf("cluster rows by id: %w", err)
}
defer rows.Close()
return scanClusterRows(rows)
}

99
PROJECTS/intermediate/security-news-scraper/internal/store/watch_test.go Normal file
View File

@ -0,0 +1,99 @@
// ©AngelaMos | 2026
// watch_test.go
package store
import "testing"
func TestNewlyFetchedClustersFiltersByFetchTimeNotPublishTime(t *testing.T) {
s := openTemp(t)
src, _ := s.UpsertSource(SourceInput{Name: "a", URL: "https://a/f", Type: "rss", Weight: 1, Enabled: true})
aOld, _ := s.InsertArticle(Article{SourceID: src, CanonicalURL: "https://a/old", ContentHash: "cold", TitleHash: "told", Title: "old", PublishedAt: 100, FetchedAt: 100})
aRecent, _ := s.InsertArticle(Article{SourceID: src, CanonicalURL: "https://a/recent", ContentHash: "crecent", TitleHash: "trecent", Title: "recently published, freshly fetched", PublishedAt: 940, FetchedAt: 1000})
aFresh, _ := s.InsertArticle(Article{SourceID: src, CanonicalURL: "https://a/fresh", ContentHash: "cfresh", TitleHash: "tfresh", Title: "fresh", PublishedAt: 1000, FetchedAt: 1000})
if err := s.ReplaceClusters([]ClusterRow{
{Key: "old", Members: []int64{aOld}, FirstSeen: 100, LastSeen: 100},
{Key: "recent", Members: []int64{aRecent}, FirstSeen: 940, LastSeen: 940},
{Key: "fresh", Members: []int64{aFresh}, FirstSeen: 1000, LastSeen: 1000},
}); err != nil {
t.Fatal(err)
}
got, err := s.NewlyFetchedClusters(1000)
if err != nil {
t.Fatal(err)
}
seen := map[string]bool{}
for _, c := range got {
for _, a := range c.Articles {
seen[a.CanonicalURL] = true
}
}
if len(got) != 2 {
t.Fatalf("newly fetched clusters = %d, want 2 (recent + fresh)", len(got))
}
if !seen["https://a/recent"] {
t.Error("a story fetched this cycle whose publish time is just before the watermark must be surfaced by the fetch-time query")
}
if !seen["https://a/fresh"] {
t.Error("the fresh story must be surfaced")
}
if seen["https://a/old"] {
t.Error("a story fetched before the watermark must be excluded")
}
dig, err := s.DigestClusters(1000)
if err != nil {
t.Fatal(err)
}
for _, c := range dig {
for _, a := range c.Articles {
if a.CanonicalURL == "https://a/recent" {
t.Error("this is the whole point of the separate query: the publish-time DigestClusters filter drops the recent-but-just-fetched story, so watch must use the fetch-time query instead")
}
}
}
}
func TestNewlyFetchedClustersEmpty(t *testing.T) {
s := openTemp(t)
got, err := s.NewlyFetchedClusters(0)
if err != nil {
t.Fatal(err)
}
if len(got) != 0 {
t.Fatalf("expected no clusters on an empty store, got %d", len(got))
}
}
func TestNewlyFetchedClustersAttachesCVEs(t *testing.T) {
s := openTemp(t)
src, _ := s.UpsertSource(SourceInput{Name: "a", URL: "https://a/f", Type: "rss", Weight: 1, Enabled: true})
a1, _ := s.InsertArticle(Article{SourceID: src, CanonicalURL: "https://a/1", ContentHash: "c1", TitleHash: "t1", Title: "kev story", PublishedAt: 5000, FetchedAt: 5000})
if err := s.UpsertCVEStub("CVE-2026-9"); err != nil {
t.Fatal(err)
}
score := 9.8
if err := s.UpdateCVEEnrichment(CVE{ID: "CVE-2026-9", CVSSScore: &score, IsKEV: true, EnrichedAt: 1, EnrichStatus: EnrichStatusOK}); err != nil {
t.Fatal(err)
}
if err := s.LinkArticleCVE(a1, "CVE-2026-9"); err != nil {
t.Fatal(err)
}
if err := s.ReplaceClusters([]ClusterRow{{Key: "1", Members: []int64{a1}, FirstSeen: 5000, LastSeen: 5000}}); err != nil {
t.Fatal(err)
}
got, err := s.NewlyFetchedClusters(1000)
if err != nil {
t.Fatal(err)
}
if len(got) != 1 || len(got[0].CVEs) != 1 {
t.Fatalf("want 1 cluster with 1 CVE, got %d clusters", len(got))
}
if !got[0].CVEs[0].IsKEV {
t.Error("KEV flag should be attached to the newly fetched cluster")
}
}

35
PROJECTS/intermediate/security-news-scraper/internal/tui/browser.go Normal file
View File

@ -0,0 +1,35 @@
// ©AngelaMos | 2026
// browser.go
package tui
import (
"fmt"
"net/url"
"os/exec"
"runtime"
"github.com/CarterPerez-dev/nadezhda/internal/setup"
)
func openURL(target string) error {
u, err := url.Parse(target)
if err != nil || (u.Scheme != "http" && u.Scheme != "https") {
return fmt.Errorf("refusing to open non-http url: %q", target)
}
name, args := openerCommand(target)
cmd := exec.Command(name, args...)
cmd.Env = setup.NonSecretEnviron()
return cmd.Start()
}
func openerCommand(target string) (string, []string) {
switch runtime.GOOS {
case "windows":
return "rundll32", []string{"url.dll,FileProtocolHandler", target}
case "darwin":
return "open", []string{target}
default:
return "xdg-open", []string{target}
}
}

213
PROJECTS/intermediate/security-news-scraper/internal/tui/detail.go Normal file
View File

@ -0,0 +1,213 @@
// ©AngelaMos | 2026
// detail.go
package tui
import (
"fmt"
"sort"
"strings"
"github.com/charmbracelet/lipgloss"
"github.com/CarterPerez-dev/nadezhda/internal/rank"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const (
cvssMaxScore = 10.0
nvdBaseURL = "https://nvd.nist.gov/vuln/detail/"
detailIndent = 3
metaDivider = " · "
)
func (m Model) renderDetailBody() string {
s := m.selected()
c := s.Cluster
w := m.bodyWidth()
t := m.theme
indent := strings.Repeat(" ", detailIndent)
var b strings.Builder
b.WriteString(t.HeadlineSel.Width(w).Render(headlineOf(c)))
b.WriteString("\n")
b.WriteString(m.detailMeta(s))
b.WriteString("\n\n")
b.WriteString(m.sectionHeader("OUTLETS", w))
b.WriteString("\n")
for _, a := range sortedArticles(c) {
b.WriteString(m.renderArticle(a, w, indent))
}
cves := m.sortedDetailCVEs(c)
if len(cves) > 0 {
b.WriteString(m.sectionHeader("VULNERABILITIES", w))
b.WriteString("\n")
for _, v := range cves {
b.WriteString(m.renderCVE(v, w, indent))
}
}
if note, ok := m.notes[c.ClusterID]; ok {
b.WriteString(m.sectionHeader("AI IDEAS", w))
b.WriteString("\n")
b.WriteString(t.fg(colorMagenta).Bold(true).Render(strings.ToUpper(note.Format)) + "\n\n")
b.WriteString(m.wrapIndent(note.Summary, w, t.Text) + "\n\n")
b.WriteString(m.wrapIndent(note.Why, w, t.Muted) + "\n\n")
for i, a := range note.Angles {
b.WriteString(m.wrapIndent(fmt.Sprintf("%d. %s", i+1, a), w, t.Text) + "\n")
}
}
return strings.TrimRight(b.String(), "\n")
}
func (m Model) detailMeta(s rank.Scored) string {
t := m.theme
c := s.Cluster
parts := []string{
t.spectrumBar(s.Score, colScoreBarW) + " " + t.Meta.Bold(true).Render(fmt.Sprintf("%.2f", s.Score)),
t.Muted.Render(fmt.Sprintf("%d outlets", outletCount(c))),
}
if mx := clusterMaxCVSS(c); mx != nil {
b := cvssBand(mx)
parts = append(parts, t.Muted.Render("CVSS ")+t.bandFG(b).Bold(true).Render(cvssString(mx)))
}
if me := clusterMaxEPSS(c); me != nil {
b := epssBand(me)
parts = append(parts, t.Muted.Render("EPSS ")+t.bandFG(b).Bold(true).Render(epssString(me)))
}
parts = append(parts,
t.Muted.Render("first "+relativeAge(c.FirstSeen, m.now)),
t.Muted.Render("last "+relativeAge(c.LastSeen, m.now)),
)
return strings.Join(parts, t.Dim.Render(metaDivider))
}
func (m Model) sectionHeader(title string, w int) string {
t := m.theme
head := t.fg(colorViolet).Bold(true).Render(glyphSelected + " " + title + " ")
n := w - lipgloss.Width(head)
if n < 0 {
n = 0
}
return head + t.Dim.Render(strings.Repeat(ruleBody, n))
}
func (m Model) renderArticle(a store.DigestArticle, w int, indent string) string {
t := m.theme
head := t.fg(colorCyan).Bold(true).Render(a.SourceName) +
t.Dim.Render(metaDivider) + t.Muted.Render(relativeAge(a.PublishedAt, m.now))
title := m.wrapIndent(a.Title, w, t.Text)
url := indent + t.Link.Render(truncate(a.CanonicalURL, w-detailIndent))
return head + "\n" + title + "\n" + url + "\n\n"
}
func (m Model) renderCVE(v store.CVE, w int, indent string) string {
t := m.theme
bnd := cvssBand(v.CVSSScore)
var b strings.Builder
id := t.fg(colorCyan).Bold(true).Render(v.ID)
if v.IsKEV {
id += " " + t.chip("KEV", colorMagenta)
}
if v.KEVRansomware {
id += " " + t.chip("RANSOMWARE", colorMagenta)
}
id += " " + t.bandFG(bnd).Bold(true).Render(bandLabel(bnd))
b.WriteString(id + "\n")
if v.CVSSScore != nil {
line := indent + t.Label.Render("CVSS ") +
t.bandBar(bnd, *v.CVSSScore/cvssMaxScore, colScoreBarW) + " " +
t.bandFG(bnd).Bold(true).Render(cvssString(v.CVSSScore))
if v.CVSSSeverity != "" {
line += " " + t.Muted.Render(v.CVSSSeverity)
}
if v.CVSSVersion != "" {
line += t.Dim.Render(" (v" + v.CVSSVersion + ")")
}
b.WriteString(line + "\n")
}
if v.EPSS != nil {
eb := epssBand(v.EPSS)
line := indent + t.Label.Render("EPSS ") + t.bandFG(eb).Bold(true).Render(epssString(v.EPSS))
if v.EPSSPercentile != nil {
line += t.Dim.Render(" (percentile ") + t.Muted.Render(epssString(v.EPSSPercentile)) + t.Dim.Render(")")
}
b.WriteString(line + "\n")
}
var meta []string
if v.CWE != "" {
meta = append(meta, t.Label.Render("CWE ")+t.Muted.Render(v.CWE))
}
if v.CVSSVector != "" {
meta = append(meta, t.Dim.Render(v.CVSSVector))
}
if len(meta) > 0 {
b.WriteString(indent + strings.Join(meta, t.Dim.Render(metaDivider)) + "\n")
}
if strings.TrimSpace(v.Description) != "" {
b.WriteString(m.wrapIndent(v.Description, w, t.Muted) + "\n")
}
b.WriteString(indent + t.Link.Render(nvdBaseURL+v.ID) + "\n\n")
return b.String()
}
func (m Model) wrapIndent(s string, w int, style lipgloss.Style) string {
width := w - detailIndent
if width < 1 {
width = 1
}
wrapped := style.Width(width).Render(strings.TrimSpace(s))
pad := strings.Repeat(" ", detailIndent)
lines := strings.Split(wrapped, "\n")
for i := range lines {
lines[i] = pad + lines[i]
}
return strings.Join(lines, "\n")
}
func sortedArticles(c store.DigestCluster) []store.DigestArticle {
out := make([]store.DigestArticle, len(c.Articles))
copy(out, c.Articles)
sort.SliceStable(out, func(i, j int) bool {
if out[i].PublishedAt != out[j].PublishedAt {
return out[i].PublishedAt > out[j].PublishedAt
}
return out[i].ID < out[j].ID
})
return out
}
func (m Model) sortedDetailCVEs(c store.DigestCluster) []store.CVE {
out := make([]store.CVE, 0, len(c.CVEs))
for _, dc := range c.CVEs {
if full, ok := m.cveDetail[dc.ID]; ok {
out = append(out, full)
continue
}
out = append(out, store.CVE{ID: dc.ID, CVSSScore: dc.CVSSScore, EPSS: dc.EPSS, IsKEV: dc.IsKEV})
}
sort.SliceStable(out, func(i, j int) bool {
if out[i].IsKEV != out[j].IsKEV {
return out[i].IsKEV
}
if ci, cj := scoreOf(out[i].CVSSScore), scoreOf(out[j].CVSSScore); ci != cj {
return ci > cj
}
return out[i].ID < out[j].ID
})
return out
}
func scoreOf(p *float64) float64 {
if p == nil {
return -1
}
return *p
}

197
PROJECTS/intermediate/security-news-scraper/internal/tui/format.go Normal file
View File

@ -0,0 +1,197 @@
// ©AngelaMos | 2026
// format.go
package tui
import (
"fmt"
"strings"
"time"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
type band int
const (
bandNone band = iota
bandLow
bandMedium
bandHigh
bandCritical
)
const (
cvssCritical = 9.0
cvssHigh = 7.0
cvssMedium = 4.0
epssHot = 0.5
epssWarm = 0.1
secsPerMinute = 60
secsPerHour = 3600
secsPerDay = 86400
ellipsis = "…"
emptyMarker = "───"
naMarker = "—"
)
func clamp01(v float64) float64 {
if v < 0 {
return 0
}
if v > 1 {
return 1
}
return v
}
func cvssBand(score *float64) band {
if score == nil {
return bandNone
}
switch v := *score; {
case v >= cvssCritical:
return bandCritical
case v >= cvssHigh:
return bandHigh
case v >= cvssMedium:
return bandMedium
case v > 0:
return bandLow
default:
return bandNone
}
}
func bandLabel(b band) string {
switch b {
case bandCritical:
return "CRITICAL"
case bandHigh:
return "HIGH"
case bandMedium:
return "MEDIUM"
case bandLow:
return "LOW"
default:
return naMarker
}
}
func clusterHasKEV(c store.DigestCluster) bool {
for _, v := range c.CVEs {
if v.IsKEV {
return true
}
}
return false
}
func clusterMaxCVSS(c store.DigestCluster) *float64 {
var best *float64
for _, v := range c.CVEs {
if v.CVSSScore == nil {
continue
}
if best == nil || *v.CVSSScore > *best {
best = v.CVSSScore
}
}
return best
}
func clusterMaxEPSS(c store.DigestCluster) *float64 {
var best *float64
for _, v := range c.CVEs {
if v.EPSS == nil {
continue
}
if best == nil || *v.EPSS > *best {
best = v.EPSS
}
}
return best
}
func clusterBand(c store.DigestCluster) band {
if clusterHasKEV(c) {
return bandCritical
}
return cvssBand(clusterMaxCVSS(c))
}
func epssBand(p *float64) band {
if p == nil {
return bandNone
}
switch v := *p; {
case v >= epssHot:
return bandCritical
case v >= epssWarm:
return bandHigh
case v > 0:
return bandMedium
default:
return bandNone
}
}
func cvssString(score *float64) string {
if score == nil {
return naMarker
}
return fmt.Sprintf("%.1f", *score)
}
func epssString(p *float64) string {
if p == nil {
return naMarker
}
return fmt.Sprintf("%.1f%%", clamp01(*p)*100)
}
func relativeAge(unix int64, now time.Time) string {
if unix <= 0 {
return naMarker
}
secs := now.Unix() - unix
if secs < 0 {
secs = 0
}
switch {
case secs < secsPerMinute:
return "just now"
case secs < secsPerHour:
return fmt.Sprintf("%dm ago", secs/secsPerMinute)
case secs < secsPerDay:
return fmt.Sprintf("%dh ago", secs/secsPerHour)
default:
return fmt.Sprintf("%dd ago", secs/secsPerDay)
}
}
func truncate(s string, max int) string {
s = strings.Join(strings.Fields(s), " ")
r := []rune(s)
if max <= 0 {
return ""
}
if len(r) <= max {
return s
}
if max == 1 {
return ellipsis
}
return string(r[:max-1]) + ellipsis
}
func padRight(s string, width int) string {
w := len([]rune(s))
if w >= width {
return s
}
return s + strings.Repeat(" ", width-w)
}

173
PROJECTS/intermediate/security-news-scraper/internal/tui/format_test.go Normal file
View File

@ -0,0 +1,173 @@
// ©AngelaMos | 2026
// format_test.go
package tui
import (
"testing"
"time"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
func ptr(v float64) *float64 { return &v }
func TestCVSSBand(t *testing.T) {
cases := []struct {
in *float64
want band
}{
{nil, bandNone},
{ptr(0), bandNone},
{ptr(3.9), bandLow},
{ptr(4.0), bandMedium},
{ptr(6.9), bandMedium},
{ptr(7.0), bandHigh},
{ptr(8.9), bandHigh},
{ptr(9.0), bandCritical},
{ptr(10.0), bandCritical},
}
for _, c := range cases {
if got := cvssBand(c.in); got != c.want {
t.Errorf("cvssBand(%v) = %v, want %v", c.in, got, c.want)
}
}
}
func TestEPSSBand(t *testing.T) {
cases := []struct {
in *float64
want band
}{
{nil, bandNone},
{ptr(0), bandNone},
{ptr(0.05), bandMedium},
{ptr(0.1), bandHigh},
{ptr(0.49), bandHigh},
{ptr(0.5), bandCritical},
{ptr(0.97), bandCritical},
}
for _, c := range cases {
if got := epssBand(c.in); got != c.want {
t.Errorf("epssBand(%v) = %v, want %v", c.in, got, c.want)
}
}
}
func TestBandLabel(t *testing.T) {
cases := map[band]string{
bandCritical: "CRITICAL",
bandHigh: "HIGH",
bandMedium: "MEDIUM",
bandLow: "LOW",
bandNone: naMarker,
}
for b, want := range cases {
if got := bandLabel(b); got != want {
t.Errorf("bandLabel(%v) = %q, want %q", b, got, want)
}
}
}
func TestCVSSString(t *testing.T) {
if got := cvssString(nil); got != naMarker {
t.Errorf("cvssString(nil) = %q, want %q", got, naMarker)
}
if got := cvssString(ptr(9.8)); got != "9.8" {
t.Errorf("cvssString(9.8) = %q, want 9.8", got)
}
if got := cvssString(ptr(10)); got != "10.0" {
t.Errorf("cvssString(10) = %q, want 10.0", got)
}
}
func TestEPSSString(t *testing.T) {
if got := epssString(nil); got != naMarker {
t.Errorf("epssString(nil) = %q, want %q", got, naMarker)
}
if got := epssString(ptr(0.5)); got != "50.0%" {
t.Errorf("epssString(0.5) = %q, want 50.0%%", got)
}
if got := epssString(ptr(0.001)); got != "0.1%" {
t.Errorf("epssString(0.001) = %q, want 0.1%%", got)
}
}
func TestRelativeAge(t *testing.T) {
now := time.Unix(1_000_000, 0)
cases := []struct {
unix int64
want string
}{
{0, naMarker},
{now.Unix() + 100, "just now"},
{now.Unix() - 30, "just now"},
{now.Unix() - 120, "2m ago"},
{now.Unix() - 7200, "2h ago"},
{now.Unix() - 172800, "2d ago"},
}
for _, c := range cases {
if got := relativeAge(c.unix, now); got != c.want {
t.Errorf("relativeAge(%d) = %q, want %q", c.unix, got, c.want)
}
}
}
func TestTruncate(t *testing.T) {
cases := []struct {
in string
max int
want string
}{
{"hello world", 20, "hello world"},
{"hello world", 5, "hell" + ellipsis},
{" multi space ", 20, "multi space"},
{"abc", 1, ellipsis},
{"abc", 0, ""},
}
for _, c := range cases {
if got := truncate(c.in, c.max); got != c.want {
t.Errorf("truncate(%q, %d) = %q, want %q", c.in, c.max, got, c.want)
}
}
}
func TestClusterSignals(t *testing.T) {
c := store.DigestCluster{
CVEs: []store.DigestCVE{
{ID: "CVE-A", CVSSScore: ptr(7.5), EPSS: ptr(0.2)},
{ID: "CVE-B", CVSSScore: ptr(9.8), EPSS: ptr(0.9), IsKEV: true},
{ID: "CVE-C"},
},
}
if !clusterHasKEV(c) {
t.Error("clusterHasKEV = false, want true")
}
if got := clusterMaxCVSS(c); got == nil || *got != 9.8 {
t.Errorf("clusterMaxCVSS = %v, want 9.8", got)
}
if got := clusterMaxEPSS(c); got == nil || *got != 0.9 {
t.Errorf("clusterMaxEPSS = %v, want 0.9", got)
}
if got := clusterBand(c); got != bandCritical {
t.Errorf("clusterBand = %v, want bandCritical (KEV)", got)
}
noKev := store.DigestCluster{CVEs: []store.DigestCVE{{ID: "CVE-D", CVSSScore: ptr(7.5)}}}
if got := clusterBand(noKev); got != bandHigh {
t.Errorf("clusterBand(no kev, 7.5) = %v, want bandHigh", got)
}
empty := store.DigestCluster{}
if got := clusterBand(empty); got != bandNone {
t.Errorf("clusterBand(empty) = %v, want bandNone", got)
}
}
func TestOutletColor(t *testing.T) {
cases := map[int]string{1: colorDim, 2: colorBlue, 3: colorCyan, 4: colorViolet, 9: colorViolet}
for n, want := range cases {
if got := outletColor(n); got != want {
t.Errorf("outletColor(%d) = %q, want %q", n, got, want)
}
}
}

59
PROJECTS/intermediate/security-news-scraper/internal/tui/keys.go Normal file
View File

@ -0,0 +1,59 @@
// ©AngelaMos | 2026
// keys.go
package tui
import "github.com/charmbracelet/bubbles/key"
type keyMap struct {
Up key.Binding
Down key.Binding
Top key.Binding
Bottom key.Binding
Open key.Binding
Browser key.Binding
Ideate key.Binding
Back key.Binding
Quit key.Binding
}
func defaultKeyMap() keyMap {
return keyMap{
Up: key.NewBinding(
key.WithKeys("up", "k"),
key.WithHelp("↑/k", "up"),
),
Down: key.NewBinding(
key.WithKeys("down", "j"),
key.WithHelp("↓/j", "down"),
),
Top: key.NewBinding(
key.WithKeys("g", "home"),
key.WithHelp("g", "top"),
),
Bottom: key.NewBinding(
key.WithKeys("G", "end"),
key.WithHelp("G", "bottom"),
),
Open: key.NewBinding(
key.WithKeys("enter"),
key.WithHelp("⏎", "detail"),
),
Browser: key.NewBinding(
key.WithKeys("o"),
key.WithHelp("o", "open"),
),
Ideate: key.NewBinding(
key.WithKeys("i"),
key.WithHelp("i", "ideate"),
),
Back: key.NewBinding(
key.WithKeys("esc", "backspace"),
key.WithHelp("esc", "back"),
),
Quit: key.NewBinding(
key.WithKeys("q", "ctrl+c"),
key.WithHelp("q", "quit"),
),
}
}

220
PROJECTS/intermediate/security-news-scraper/internal/tui/list.go Normal file
View File

@ -0,0 +1,220 @@
// ©AngelaMos | 2026
// list.go
package tui
import (
"fmt"
"strings"
"github.com/charmbracelet/lipgloss"
"github.com/CarterPerez-dev/nadezhda/internal/rank"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
const (
colSelBarW = 1
colRankW = 2
colScoreBarW = 10
colScoreNumW = 4
colOutletW = 6
colCVEW = 5
colGaps = 6
minHeadlineW = 12
glyphSelected = "◆"
glyphHot = "▐"
glyphOutlet = "◉"
)
func (m Model) listView() string {
header := m.listChrome()
footer := m.listFooter()
capacity := m.height - lipgloss.Height(header) - lipgloss.Height(footer)
if capacity < 1 {
capacity = 1
}
body := m.listBody(capacity)
return lipgloss.JoinVertical(lipgloss.Left, header, body, footer)
}
func (m Model) listChrome() string {
right := m.theme.Meta.Render(fmt.Sprintf("%d stories", len(m.scored)))
return lipgloss.JoinVertical(lipgloss.Left,
m.spread(m.wordmark(), right),
m.theme.rule(m.width),
m.listColHeader(),
)
}
func (m Model) listColHeader() string {
t := m.theme.ColHead
left := strings.Join([]string{
" ",
fmt.Sprintf("%*s", colRankW, "#"),
padRight("STORY", m.headlineWidth()),
padRight("SCORE", colScoreBarW),
fmt.Sprintf("%*s", colScoreNumW, ""),
padRight("OUT", colOutletW),
}, " ")
return m.spread(t.Render(left), t.Render("CVE"))
}
func (m Model) listFooter() string {
keys := m.keyHints(m.keys.Up, m.keys.Down, m.keys.Open, m.keys.Browser, m.keys.Quit)
right := m.severityLegend()
if s := m.statusText(); s != "" {
right = s
}
return lipgloss.JoinVertical(lipgloss.Left,
m.theme.rule(m.width),
m.spread(keys, right),
)
}
func (m Model) listBody(capacity int) string {
if len(m.scored) == 0 {
return m.emptyBody(capacity)
}
first, last := windowRange(m.cursor, capacity, len(m.scored))
lines := make([]string, 0, capacity)
for i := first; i < last; i++ {
lines = append(lines, m.renderRow(i, m.scored[i], i == m.cursor))
}
for len(lines) < capacity {
lines = append(lines, "")
}
return strings.Join(lines, "\n")
}
func (m Model) emptyBody(capacity int) string {
t := m.theme
msg := t.Muted.Render("no stories in the store yet — run ") +
t.KeyGlyph.Render("nadezhda scrape")
return lipgloss.Place(m.width, capacity, lipgloss.Center, lipgloss.Center, msg)
}
func (m Model) renderRow(i int, s rank.Scored, selected bool) string {
t := m.theme
c := s.Cluster
b := clusterBand(c)
kev := clusterHasKEV(c)
barStyle := t.Dim
barGlyph := " "
switch {
case selected:
barGlyph, barStyle = glyphSelected, t.SelBar
case kev || b == bandCritical:
barGlyph, barStyle = glyphHot, t.bandFG(bandCritical)
case b != bandNone:
barGlyph, barStyle = glyphHot, t.bandFG(b)
}
rankStyle := t.Rank
headStyle := t.Headline
if selected {
rankStyle, headStyle = t.RankSel, t.HeadlineSel
}
hlW := m.headlineWidth()
left := strings.Join([]string{
barStyle.Render(barGlyph),
rankStyle.Render(fmt.Sprintf("%*d", colRankW, i+1)),
headStyle.Render(padRight(truncate(headlineOf(c), hlW), hlW)),
t.spectrumBar(s.Score, colScoreBarW),
t.Meta.Render(fmt.Sprintf("%*s", colScoreNumW, fmt.Sprintf("%.2f", s.Score))),
m.renderOutlets(c),
}, " ")
return m.spread(left, m.renderCVEChip(c, b, kev))
}
func windowRange(cursor, capacity, total int) (int, int) {
if capacity < 1 {
capacity = 1
}
first := 0
if cursor >= capacity {
first = cursor - capacity + 1
}
last := first + capacity
if last > total {
last = total
}
return first, last
}
func (m Model) renderOutlets(c store.DigestCluster) string {
n := outletCount(c)
dots := n
if dots > colOutletW {
dots = colOutletW
}
styled := m.theme.fg(outletColor(n)).Render(strings.Repeat(glyphOutlet, dots))
if pad := colOutletW - dots; pad > 0 {
styled += strings.Repeat(" ", pad)
}
return styled
}
func (m Model) renderCVEChip(c store.DigestCluster, b band, kev bool) string {
t := m.theme
if kev {
return t.chip("KEV", colorMagenta)
}
if max := clusterMaxCVSS(c); max != nil {
return t.bandFG(b).Bold(true).Render(cvssString(max))
}
if len(c.CVEs) > 0 {
return t.Dim.Render("cve")
}
return t.Dim.Render(emptyMarker)
}
func (m Model) headlineWidth() int {
fixed := colSelBarW + colRankW + colScoreBarW + colScoreNumW + colOutletW + colCVEW + colGaps
if w := m.width - fixed; w > minHeadlineW {
return w
}
return minHeadlineW
}
func headlineArticle(c store.DigestCluster) store.DigestArticle {
var best store.DigestArticle
for i, a := range c.Articles {
if i == 0 || a.PublishedAt > best.PublishedAt ||
(a.PublishedAt == best.PublishedAt && a.ID < best.ID) {
best = a
}
}
return best
}
func headlineOf(c store.DigestCluster) string {
if a := headlineArticle(c); strings.TrimSpace(a.Title) != "" {
return a.Title
}
return "(untitled cluster)"
}
func outletCount(c store.DigestCluster) int {
seen := make(map[string]struct{}, len(c.Articles))
for _, a := range c.Articles {
seen[a.SourceName] = struct{}{}
}
return len(seen)
}
func outletColor(n int) string {
switch {
case n >= 4:
return colorViolet
case n == 3:
return colorCyan
case n == 2:
return colorBlue
default:
return colorDim
}
}

314
PROJECTS/intermediate/security-news-scraper/internal/tui/model.go Normal file
View File

@ -0,0 +1,314 @@
// ©AngelaMos | 2026
// model.go
package tui
import (
"time"
"github.com/charmbracelet/bubbles/key"
"github.com/charmbracelet/bubbles/spinner"
"github.com/charmbracelet/bubbles/viewport"
tea "github.com/charmbracelet/bubbletea"
"github.com/CarterPerez-dev/nadezhda/internal/ai"
"github.com/CarterPerez-dev/nadezhda/internal/rank"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
type viewState int
const (
stateLoading viewState = iota
stateList
stateDetail
stateError
)
const (
defaultWidth = 100
defaultHeight = 32
detailChrome = 4
spinnerFPS = time.Second / 10
)
var raveSpinner = spinner.Spinner{
Frames: []string{"◇", "◈", "◆", "◈", "◇", "·"},
FPS: spinnerFPS,
}
type Data struct {
Scored []rank.Scored
CVEDetail map[string]store.CVE
Notes map[int64]ai.IdeationResult
}
type Loader func() (Data, error)
type dataMsg struct{ data Data }
type errMsg struct{ err error }
type openedMsg struct {
url string
err error
}
type Ideator func(store.DigestCluster) (ai.IdeationResult, error)
type ideatedMsg struct {
clusterID int64
result ai.IdeationResult
}
type ideateErrMsg struct{ err error }
type Model struct {
state viewState
loader Loader
now time.Time
theme Theme
keys keyMap
spinner spinner.Model
viewport viewport.Model
width int
height int
scored []rank.Scored
cveDetail map[string]store.CVE
cursor int
err error
opener func(string) error
status string
statusErr bool
ideator Ideator
generating bool
notes map[int64]ai.IdeationResult
}
func New(loader Loader, ideator Ideator, now time.Time) Model {
th := NewTheme()
sp := spinner.New(spinner.WithSpinner(raveSpinner), spinner.WithStyle(th.Spinner))
m := Model{
state: stateLoading,
loader: loader,
now: now,
theme: th,
keys: defaultKeyMap(),
spinner: sp,
viewport: viewport.New(defaultWidth, defaultHeight-detailChrome),
width: defaultWidth,
height: defaultHeight,
cveDetail: map[string]store.CVE{},
opener: openURL,
ideator: ideator,
notes: map[int64]ai.IdeationResult{},
}
return m
}
func (m Model) Init() tea.Cmd {
return tea.Batch(m.spinner.Tick, m.load())
}
func (m Model) load() tea.Cmd {
loader := m.loader
return func() tea.Msg {
data, err := loader()
if err != nil {
return errMsg{err}
}
return dataMsg{data}
}
}
func (m Model) Update(msg tea.Msg) (tea.Model, tea.Cmd) {
switch msg := msg.(type) {
case tea.WindowSizeMsg:
m.applySize(msg.Width, msg.Height)
return m, nil
case dataMsg:
m.scored = msg.data.Scored
m.cveDetail = msg.data.CVEDetail
if msg.data.Notes != nil {
m.notes = msg.data.Notes
}
m.state = stateList
return m, nil
case errMsg:
m.state = stateError
m.err = msg.err
return m, nil
case openedMsg:
if msg.err != nil {
m.status, m.statusErr = "open failed: "+msg.err.Error(), true
} else {
m.status, m.statusErr = "opened in browser", false
}
return m, nil
case ideatedMsg:
m.generating = false
m.notes[msg.clusterID] = msg.result
m.status, m.statusErr = "ideas ready", false
if m.state == stateDetail {
m.viewport.SetContent(m.renderDetailBody())
m.viewport.GotoBottom()
}
return m, nil
case ideateErrMsg:
m.generating = false
m.status, m.statusErr = "ideate failed: "+msg.err.Error(), true
return m, nil
case spinner.TickMsg:
if m.state != stateLoading && !m.generating {
return m, nil
}
var cmd tea.Cmd
m.spinner, cmd = m.spinner.Update(msg)
return m, cmd
case tea.KeyMsg:
return m.handleKey(msg)
}
if m.state == stateDetail {
var cmd tea.Cmd
m.viewport, cmd = m.viewport.Update(msg)
return m, cmd
}
return m, nil
}
func (m Model) handleKey(msg tea.KeyMsg) (tea.Model, tea.Cmd) {
m.status = ""
if key.Matches(msg, m.keys.Quit) {
return m, tea.Quit
}
if key.Matches(msg, m.keys.Browser) && (m.state == stateList || m.state == stateDetail) {
return m, m.openSelected()
}
if key.Matches(msg, m.keys.Ideate) && m.state == stateDetail {
return m.startIdeate()
}
switch m.state {
case stateList:
return m.handleListKey(msg)
case stateDetail:
return m.handleDetailKey(msg)
default:
return m, nil
}
}
func (m Model) openSelected() tea.Cmd {
target := headlineArticle(m.selected().Cluster).CanonicalURL
if target == "" {
return nil
}
open := m.opener
return func() tea.Msg {
return openedMsg{url: target, err: open(target)}
}
}
func (m Model) startIdeate() (tea.Model, tea.Cmd) {
if m.ideator == nil {
m.status, m.statusErr = "AI not set up — run: nadezhda ai", true
return m, nil
}
if m.generating || len(m.scored) == 0 {
return m, nil
}
m.generating = true
m.status, m.statusErr = "ideating "+headlineOf(m.selected().Cluster), false
return m, tea.Batch(m.ideateSelected(), m.spinner.Tick)
}
func (m Model) ideateSelected() tea.Cmd {
ideator := m.ideator
cluster := m.selected().Cluster
return func() tea.Msg {
res, err := ideator(cluster)
if err != nil {
return ideateErrMsg{err}
}
return ideatedMsg{clusterID: cluster.ClusterID, result: res}
}
}
func (m Model) handleListKey(msg tea.KeyMsg) (tea.Model, tea.Cmd) {
switch {
case key.Matches(msg, m.keys.Up):
if m.cursor > 0 {
m.cursor--
}
case key.Matches(msg, m.keys.Down):
if m.cursor < len(m.scored)-1 {
m.cursor++
}
case key.Matches(msg, m.keys.Top):
m.cursor = 0
case key.Matches(msg, m.keys.Bottom):
if len(m.scored) > 0 {
m.cursor = len(m.scored) - 1
}
case key.Matches(msg, m.keys.Open):
if len(m.scored) > 0 {
m.state = stateDetail
m.viewport.SetContent(m.renderDetailBody())
m.viewport.GotoTop()
}
}
return m, nil
}
func (m Model) handleDetailKey(msg tea.KeyMsg) (tea.Model, tea.Cmd) {
if key.Matches(msg, m.keys.Back) {
m.state = stateList
return m, nil
}
var cmd tea.Cmd
m.viewport, cmd = m.viewport.Update(msg)
return m, cmd
}
func (m *Model) applySize(w, h int) {
m.width = w
m.height = h
m.viewport.Width = m.bodyWidth()
m.viewport.Height = m.detailBodyHeight()
if m.state == stateDetail {
m.viewport.SetContent(m.renderDetailBody())
}
}
func (m Model) bodyWidth() int {
if m.width > 1 {
return m.width
}
return 1
}
func (m Model) detailBodyHeight() int {
if h := m.height - detailChrome; h > 1 {
return h
}
return 1
}
func (m Model) selected() rank.Scored {
if m.cursor < 0 || m.cursor >= len(m.scored) {
return rank.Scored{}
}
return m.scored[m.cursor]
}
func Run(loader Loader, ideator Ideator) error {
m := New(loader, ideator, time.Now())
_, err := tea.NewProgram(m, tea.WithAltScreen()).Run()
return err
}
var _ tea.Model = Model{}

405
PROJECTS/intermediate/security-news-scraper/internal/tui/model_test.go Normal file
View File

@ -0,0 +1,405 @@
// ©AngelaMos | 2026
// model_test.go
package tui
import (
"errors"
"reflect"
"strings"
"testing"
"time"
"github.com/charmbracelet/bubbles/spinner"
tea "github.com/charmbracelet/bubbletea"
"github.com/CarterPerez-dev/nadezhda/internal/ai"
"github.com/CarterPerez-dev/nadezhda/internal/rank"
"github.com/CarterPerez-dev/nadezhda/internal/store"
)
func testNow() time.Time { return time.Unix(1_720_000_000, 0) }
func runeKey(r rune) tea.KeyMsg { return tea.KeyMsg{Type: tea.KeyRunes, Runes: []rune{r}} }
func sampleData() Data {
now := testNow().Unix()
c1 := store.DigestCluster{
ClusterID: 1,
Key: "log4shell",
Size: 2,
FirstSeen: now - 3600,
LastSeen: now - 600,
Articles: []store.DigestArticle{
{ID: 1, SourceName: "krebs", SourceWeight: 1.0, Title: "Log4Shell RCE exploited in the wild", CanonicalURL: "https://krebsonsecurity.com/log4shell", PublishedAt: now - 600},
{ID: 2, SourceName: "theregister", SourceWeight: 0.9, Title: "Log4j flaw under active attack", CanonicalURL: "https://theregister.com/log4j", PublishedAt: now - 3600},
},
CVEs: []store.DigestCVE{
{ID: "CVE-2021-44228", CVSSScore: ptr(10.0), EPSS: ptr(0.97), IsKEV: true},
},
}
c2 := store.DigestCluster{
ClusterID: 2,
Key: "policy",
Size: 1,
FirstSeen: now - 7200,
LastSeen: now - 7200,
Articles: []store.DigestArticle{
{ID: 3, SourceName: "darkreading", SourceWeight: 0.7, Title: "New disclosure guidelines published", CanonicalURL: "https://darkreading.com/policy", PublishedAt: now - 7200},
},
}
scored := []rank.Scored{{Cluster: c1, Score: 0.94}, {Cluster: c2, Score: 0.42}}
detail := map[string]store.CVE{
"CVE-2021-44228": {
ID: "CVE-2021-44228",
Description: "Apache Log4j2 JNDI features do not protect against attacker controlled LDAP endpoints.",
CVSSScore: ptr(10.0),
CVSSVersion: "3.1",
CVSSSeverity: "CRITICAL",
CVSSVector: "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:C/C:H/I:H/A:H",
CWE: "CWE-502",
IsKEV: true,
EPSS: ptr(0.97),
},
}
return Data{Scored: scored, CVEDetail: detail}
}
func toModel(t *testing.T, tm tea.Model) Model {
t.Helper()
m, ok := tm.(Model)
if !ok {
t.Fatalf("Update returned %T, want tui.Model", tm)
}
return m
}
func step(t *testing.T, m Model, msg tea.Msg) Model {
t.Helper()
tm, _ := m.Update(msg)
return toModel(t, tm)
}
func loadedModel(t *testing.T) Model {
t.Helper()
return step(t, New(nil, nil, testNow()), dataMsg{sampleData()})
}
func TestInitialStateIsLoading(t *testing.T) {
if m := New(nil, nil, testNow()); m.state != stateLoading {
t.Fatalf("initial state = %v, want stateLoading", m.state)
}
}
func TestDataMsgTransitionsToList(t *testing.T) {
m := loadedModel(t)
if m.state != stateList {
t.Fatalf("state = %v, want stateList", m.state)
}
if len(m.scored) != 2 {
t.Fatalf("len(scored) = %d, want 2", len(m.scored))
}
}
func TestErrMsgTransitionsToError(t *testing.T) {
m := step(t, New(nil, nil, testNow()), errMsg{errors.New("wire down")})
if m.state != stateError {
t.Fatalf("state = %v, want stateError", m.state)
}
if m.err == nil {
t.Fatal("err = nil, want non-nil")
}
}
func TestListNavigationClamps(t *testing.T) {
m := loadedModel(t)
m = step(t, m, runeKey('j'))
if m.cursor != 1 {
t.Fatalf("after down, cursor = %d, want 1", m.cursor)
}
m = step(t, m, runeKey('j'))
if m.cursor != 1 {
t.Fatalf("after down past end, cursor = %d, want 1", m.cursor)
}
m = step(t, m, runeKey('k'))
if m.cursor != 0 {
t.Fatalf("after up, cursor = %d, want 0", m.cursor)
}
m = step(t, m, runeKey('k'))
if m.cursor != 0 {
t.Fatalf("after up past start, cursor = %d, want 0", m.cursor)
}
}
func TestListTopBottom(t *testing.T) {
m := loadedModel(t)
m = step(t, m, runeKey('G'))
if m.cursor != 1 {
t.Fatalf("after G, cursor = %d, want 1", m.cursor)
}
m = step(t, m, runeKey('g'))
if m.cursor != 0 {
t.Fatalf("after g, cursor = %d, want 0", m.cursor)
}
}
func TestOpenAndBack(t *testing.T) {
m := loadedModel(t)
m = step(t, m, tea.KeyMsg{Type: tea.KeyEnter})
if m.state != stateDetail {
t.Fatalf("after enter, state = %v, want stateDetail", m.state)
}
m = step(t, m, tea.KeyMsg{Type: tea.KeyEsc})
if m.state != stateList {
t.Fatalf("after esc, state = %v, want stateList", m.state)
}
}
func TestQuitReturnsQuitCmd(t *testing.T) {
m := loadedModel(t)
_, cmd := m.Update(runeKey('q'))
if cmd == nil {
t.Fatal("quit returned nil cmd")
}
if reflect.TypeOf(cmd()) != reflect.TypeOf(tea.Quit()) {
t.Fatalf("quit cmd yielded %T, want tea.QuitMsg", cmd())
}
}
func TestWindowSizeSizesViewport(t *testing.T) {
m := loadedModel(t)
m = step(t, m, tea.WindowSizeMsg{Width: 120, Height: 40})
if m.width != 120 || m.height != 40 {
t.Fatalf("size = %dx%d, want 120x40", m.width, m.height)
}
if m.viewport.Width != 120 {
t.Fatalf("viewport.Width = %d, want 120", m.viewport.Width)
}
if m.viewport.Height != 40-detailChrome {
t.Fatalf("viewport.Height = %d, want %d", m.viewport.Height, 40-detailChrome)
}
}
func TestSpinnerTickIgnoredOutsideLoading(t *testing.T) {
m := loadedModel(t)
m = step(t, m, spinner.TickMsg{})
if m.state != stateList {
t.Fatalf("spinner tick changed state to %v, want stateList", m.state)
}
}
func TestViewsRenderNonEmpty(t *testing.T) {
loading := New(nil, nil, testNow())
if strings.TrimSpace(loading.View()) == "" {
t.Error("loading view is empty")
}
list := loadedModel(t)
lv := list.View()
if !strings.Contains(lv, "NADEZHDA") {
t.Error("list view missing brand")
}
if !strings.Contains(lv, "Log4Shell") {
t.Error("list view missing headline")
}
detail := step(t, list, tea.KeyMsg{Type: tea.KeyEnter})
dv := detail.View()
if !strings.Contains(dv, "CVE-2021-44228") {
t.Error("detail view missing CVE id")
}
if !strings.Contains(dv, "CRITICAL") {
t.Error("detail view missing severity label")
}
errv := step(t, New(nil, nil, testNow()), errMsg{errors.New("boom")})
if strings.TrimSpace(errv.View()) == "" {
t.Error("error view is empty")
}
}
func TestEmptyStoreRendersHint(t *testing.T) {
m := step(t, New(nil, nil, testNow()), dataMsg{Data{}})
if m.state != stateList {
t.Fatalf("state = %v, want stateList", m.state)
}
if !strings.Contains(m.View(), "scrape") {
t.Error("empty list view missing scrape hint")
}
}
func TestTooSmallTerminal(t *testing.T) {
m := step(t, loadedModel(t), tea.WindowSizeMsg{Width: 20, Height: 5})
if !strings.Contains(m.View(), "too small") {
t.Error("tiny terminal did not render the too-small notice")
}
}
func TestWindowRange(t *testing.T) {
cases := []struct {
cursor, capacity, total int
first, last int
}{
{0, 5, 10, 0, 5},
{4, 5, 10, 0, 5},
{5, 5, 10, 1, 6},
{9, 5, 10, 5, 10},
{0, 5, 3, 0, 3},
{2, 5, 3, 0, 3},
{0, 0, 10, 0, 1},
{7, 3, 10, 5, 8},
}
for _, c := range cases {
first, last := windowRange(c.cursor, c.capacity, c.total)
if first != c.first || last != c.last {
t.Errorf("windowRange(%d, %d, %d) = (%d, %d), want (%d, %d)",
c.cursor, c.capacity, c.total, first, last, c.first, c.last)
}
if c.cursor < c.total && (c.cursor < first || c.cursor >= last) {
t.Errorf("windowRange(%d, %d, %d) window [%d,%d) excludes cursor",
c.cursor, c.capacity, c.total, first, last)
}
}
}
func TestHeadlineFreshest(t *testing.T) {
fresh := store.DigestCluster{Articles: []store.DigestArticle{
{ID: 1, Title: "older", PublishedAt: 100},
{ID: 2, Title: "newest", PublishedAt: 300},
{ID: 3, Title: "mid", PublishedAt: 200},
}}
if got := headlineOf(fresh); got != "newest" {
t.Errorf("headlineOf(fresh) = %q, want newest", got)
}
tie := store.DigestCluster{Articles: []store.DigestArticle{
{ID: 5, Title: "high-id", PublishedAt: 300},
{ID: 2, Title: "low-id", PublishedAt: 300},
}}
if got := headlineOf(tie); got != "low-id" {
t.Errorf("headlineOf(tie) = %q, want low-id (lowest id on equal time)", got)
}
if got := headlineOf(store.DigestCluster{}); got != "(untitled cluster)" {
t.Errorf("headlineOf(empty) = %q, want (untitled cluster)", got)
}
}
func TestOpenInBrowser(t *testing.T) {
m := loadedModel(t)
var opened string
m.opener = func(u string) error { opened = u; return nil }
tm, cmd := m.Update(runeKey('o'))
m = toModel(t, tm)
if cmd == nil {
t.Fatal("pressing o produced no command")
}
msg := cmd()
if opened != "https://krebsonsecurity.com/log4shell" {
t.Fatalf("opened %q, want the selected cluster's headline url", opened)
}
om, ok := msg.(openedMsg)
if !ok {
t.Fatalf("open cmd returned %T, want openedMsg", msg)
}
m = step(t, m, om)
if m.status == "" || m.statusErr {
t.Errorf("after open: status=%q err=%v, want a success status", m.status, m.statusErr)
}
if !strings.Contains(m.View(), m.status) {
t.Error("open status not shown in the view")
}
m = step(t, m, runeKey('j'))
if m.status != "" {
t.Error("status should clear on the next keypress")
}
}
func TestOpenURLRejectsNonHTTP(t *testing.T) {
for _, bad := range []string{"file:///etc/passwd", "javascript:alert(1)", "", "ftp://x/y"} {
if err := openURL(bad); err == nil {
t.Errorf("openURL(%q) = nil, want refusal", bad)
}
}
}
func TestIdeateDisabledShowsHint(t *testing.T) {
m := loadedModel(t)
m = step(t, m, tea.KeyMsg{Type: tea.KeyEnter})
m = step(t, m, runeKey('i'))
if !m.statusErr || !strings.Contains(m.status, "nadezhda ai") {
t.Errorf("i with nil ideator: status=%q err=%v, want a setup hint", m.status, m.statusErr)
}
if m.generating {
t.Error("generating should stay false when ideator is nil")
}
}
func TestIdeateIgnoredInListView(t *testing.T) {
m := loadedModel(t)
m.ideator = func(c store.DigestCluster) (ai.IdeationResult, error) {
return ai.IdeationResult{Summary: "s", Angles: []string{"a"}, Format: "blog"}, nil
}
tm, cmd := m.Update(runeKey('i'))
m = toModel(t, tm)
if m.generating || cmd != nil {
t.Errorf("i in the list must be a no-op: generating=%v cmd=%v", m.generating, cmd)
}
}
func TestIdeateFlowStoresAndRenders(t *testing.T) {
m := loadedModel(t)
m.ideator = func(c store.DigestCluster) (ai.IdeationResult, error) {
return ai.IdeationResult{Summary: "s", Why: "w", Angles: []string{"angle-one", "angle-two"}, Format: "video"}, nil
}
m = step(t, m, tea.KeyMsg{Type: tea.KeyEnter})
tm, cmd := m.Update(runeKey('i'))
m = toModel(t, tm)
if !m.generating || cmd == nil {
t.Fatalf("after i: generating=%v cmd=%v", m.generating, cmd)
}
msg := m.ideateSelected()()
im, ok := msg.(ideatedMsg)
if !ok {
t.Fatalf("ideateSelected produced %T, want ideatedMsg", msg)
}
if im.clusterID != 1 || im.result.Summary != "s" {
t.Fatalf("ideatedMsg = %+v", im)
}
m = step(t, m, im)
if m.generating {
t.Error("generating still true after ideatedMsg")
}
if m.notes[1].Summary != "s" || len(m.notes[1].Angles) != 2 {
t.Errorf("note not stored: %+v", m.notes[1])
}
m = step(t, m, tea.KeyMsg{Type: tea.KeyEnter})
dv := m.View()
if !strings.Contains(dv, "AI IDEAS") || !strings.Contains(dv, "angle-one") {
t.Error("detail view missing the ideation section")
}
}
func TestIdeateErrorSetsStatus(t *testing.T) {
m := loadedModel(t)
m.ideator = func(c store.DigestCluster) (ai.IdeationResult, error) {
return ai.IdeationResult{}, errors.New("boom")
}
m = step(t, m, tea.KeyMsg{Type: tea.KeyEnter})
m = step(t, m, runeKey('i'))
msg := m.ideateSelected()()
em, ok := msg.(ideateErrMsg)
if !ok {
t.Fatalf("ideateSelected produced %T, want ideateErrMsg", msg)
}
m = step(t, m, em)
if m.generating {
t.Error("generating still true after ideateErrMsg")
}
if !m.statusErr {
t.Error("statusErr not set after ideate failure")
}
}

151
PROJECTS/intermediate/security-news-scraper/internal/tui/theme.go Normal file
View File

@ -0,0 +1,151 @@
// ©AngelaMos | 2026
// theme.go
package tui
import (
"math"
"strings"
"github.com/charmbracelet/lipgloss"
)
const (
colorVoid = "#0B0620"
colorText = "#E7E0FF"
colorMuted = "#9A8AD0"
colorDim = "#574B7D"
colorMagenta = "#FF2E97"
colorAmber = "#FF8A3D"
colorYellow = "#FFE04D"
colorCyan = "#00E5FF"
colorViolet = "#A65CFF"
colorBlue = "#2D6BFF"
)
const (
barFull = "█"
barEmpty = "░"
ruleLead = "▓▒░"
ruleBody = "─"
)
func bandColor(b band) string {
switch b {
case bandCritical:
return colorMagenta
case bandHigh:
return colorAmber
case bandMedium:
return colorYellow
case bandLow:
return colorCyan
default:
return colorDim
}
}
type Theme struct {
Brand lipgloss.Style
BrandMark lipgloss.Style
Meta lipgloss.Style
ColHead lipgloss.Style
Text lipgloss.Style
Muted lipgloss.Style
Dim lipgloss.Style
Rank lipgloss.Style
RankSel lipgloss.Style
Headline lipgloss.Style
HeadlineSel lipgloss.Style
SelBar lipgloss.Style
KeyGlyph lipgloss.Style
KeyDesc lipgloss.Style
Spinner lipgloss.Style
PanelTitle lipgloss.Style
Label lipgloss.Style
Link lipgloss.Style
}
func NewTheme() Theme {
base := lipgloss.NewStyle().Foreground(lipgloss.Color(colorText))
return Theme{
Brand: lipgloss.NewStyle().Foreground(lipgloss.Color(colorViolet)).Bold(true),
BrandMark: lipgloss.NewStyle().Foreground(lipgloss.Color(colorMagenta)).Bold(true),
Meta: lipgloss.NewStyle().Foreground(lipgloss.Color(colorCyan)),
ColHead: lipgloss.NewStyle().Foreground(lipgloss.Color(colorMuted)).Bold(true),
Text: base,
Muted: lipgloss.NewStyle().Foreground(lipgloss.Color(colorMuted)),
Dim: lipgloss.NewStyle().Foreground(lipgloss.Color(colorDim)),
Rank: lipgloss.NewStyle().Foreground(lipgloss.Color(colorMuted)),
RankSel: lipgloss.NewStyle().Foreground(lipgloss.Color(colorMagenta)).Bold(true),
Headline: base,
HeadlineSel: lipgloss.NewStyle().Foreground(lipgloss.Color(colorText)).Bold(true),
SelBar: lipgloss.NewStyle().Foreground(lipgloss.Color(colorMagenta)).Bold(true),
KeyGlyph: lipgloss.NewStyle().Foreground(lipgloss.Color(colorCyan)).Bold(true),
KeyDesc: lipgloss.NewStyle().Foreground(lipgloss.Color(colorMuted)),
Spinner: lipgloss.NewStyle().Foreground(lipgloss.Color(colorMagenta)).Bold(true),
PanelTitle: lipgloss.NewStyle().Foreground(lipgloss.Color(colorViolet)).Bold(true),
Label: lipgloss.NewStyle().Foreground(lipgloss.Color(colorBlue)).Bold(true),
Link: lipgloss.NewStyle().Foreground(lipgloss.Color(colorCyan)).Underline(true),
}
}
func (t Theme) fg(color string) lipgloss.Style {
return lipgloss.NewStyle().Foreground(lipgloss.Color(color))
}
func (t Theme) bandFG(b band) lipgloss.Style {
return t.fg(bandColor(b))
}
func (t Theme) chip(text, color string) string {
return lipgloss.NewStyle().
Foreground(lipgloss.Color(colorVoid)).
Background(lipgloss.Color(color)).
Bold(true).
Render(" " + text + " ")
}
func (t Theme) spectrumBar(frac float64, width int) string {
if width <= 0 {
return ""
}
filled := int(math.Round(clamp01(frac) * float64(width)))
stops := []string{colorCyan, colorBlue, colorViolet, colorMagenta}
var b strings.Builder
for i := 0; i < width; i++ {
if i < filled {
b.WriteString(t.fg(stops[i*len(stops)/width]).Render(barFull))
} else {
b.WriteString(t.Dim.Render(barEmpty))
}
}
return b.String()
}
func (t Theme) bandBar(b band, frac float64, width int) string {
if width <= 0 {
return ""
}
filled := int(math.Round(clamp01(frac) * float64(width)))
color := bandColor(b)
var sb strings.Builder
for i := 0; i < width; i++ {
if i < filled {
sb.WriteString(t.fg(color).Render(barFull))
} else {
sb.WriteString(t.Dim.Render(barEmpty))
}
}
return sb.String()
}
func (t Theme) rule(width int) string {
lead := len([]rune(ruleLead)) + 1
n := width - lead
if n < 0 {
return t.fg(colorViolet).Render(ruleLead)
}
return t.fg(colorViolet).Render(ruleLead) + " " + t.Dim.Render(strings.Repeat(ruleBody, n))
}

162
PROJECTS/intermediate/security-news-scraper/internal/tui/view.go Normal file
View File

@ -0,0 +1,162 @@
// ©AngelaMos | 2026
// view.go
package tui
import (
"fmt"
"strings"
"github.com/charmbracelet/bubbles/key"
"github.com/charmbracelet/lipgloss"
)
const (
brandName = "NADEZHDA"
tagline = "THREAT.WIRE"
markOpen = "◤ "
markClose = " ◢"
markMid = "═◆═"
errWrapMax = 80
errWrapMin = 20
errWrapMargin = 6
minRenderWidth = 44
minRenderHeight = 10
)
func (m Model) View() string {
if m.width < minRenderWidth || m.height < minRenderHeight {
return m.tooSmallView()
}
switch m.state {
case stateLoading:
return m.loadingView()
case stateError:
return m.errorView()
case stateDetail:
return m.detailView()
default:
return m.listView()
}
}
func (m Model) tooSmallView() string {
w, h := m.width, m.height
if w < 1 {
w = 1
}
if h < 1 {
h = 1
}
msg := m.theme.Muted.Render(fmt.Sprintf("terminal too small — need at least %dx%d", minRenderWidth, minRenderHeight))
return lipgloss.Place(w, h, lipgloss.Center, lipgloss.Center, msg)
}
func (m Model) wordmark() string {
t := m.theme
return t.BrandMark.Render(markOpen) + t.Brand.Render(brandName) + t.BrandMark.Render(markClose) +
" " + t.fg(colorCyan).Render(markMid) + " " + t.fg(colorViolet).Render(tagline) + " " + t.fg(colorCyan).Render(markMid)
}
func (m Model) spread(left, right string) string {
gap := m.width - lipgloss.Width(left) - lipgloss.Width(right)
if gap < 1 {
gap = 1
}
return left + strings.Repeat(" ", gap) + right
}
func (m Model) keyHints(bindings ...key.Binding) string {
t := m.theme
parts := make([]string, 0, len(bindings))
for _, b := range bindings {
h := b.Help()
parts = append(parts, t.KeyGlyph.Render(h.Key)+" "+t.KeyDesc.Render(h.Desc))
}
return strings.Join(parts, t.Dim.Render(" · "))
}
func (m Model) severityLegend() string {
t := m.theme
item := func(color, label string) string {
return t.fg(color).Render("■") + " " + t.KeyDesc.Render(label)
}
return strings.Join([]string{
item(colorMagenta, "KEV/CRIT"),
item(colorAmber, "HIGH"),
item(colorYellow, "MED"),
item(colorCyan, "LOW"),
}, " ")
}
func (m Model) loadingView() string {
t := m.theme
brand := t.BrandMark.Render(markOpen) + t.Brand.Render(brandName) + t.BrandMark.Render(markClose)
tag := t.fg(colorViolet).Render(tagline)
line := m.spinner.View() + " " + t.Meta.Render("assembling threat wire") + " " + m.spinner.View()
block := lipgloss.JoinVertical(lipgloss.Center, brand, tag, "", line)
return lipgloss.Place(m.width, m.height, lipgloss.Center, lipgloss.Center, block)
}
func (m Model) errorView() string {
t := m.theme
title := t.fg(colorMagenta).Bold(true).Render("▚ WIRE FAULT ▞")
block := lipgloss.JoinVertical(lipgloss.Center,
title,
"",
m.errString(),
"",
t.KeyDesc.Render("press ")+t.KeyGlyph.Render("q")+t.KeyDesc.Render(" to quit"),
)
return lipgloss.Place(m.width, m.height, lipgloss.Center, lipgloss.Center, block)
}
func (m Model) errString() string {
if m.err == nil {
return m.theme.Text.Render("unknown fault")
}
w := m.width - errWrapMargin
if w > errWrapMax {
w = errWrapMax
}
if w < errWrapMin {
w = errWrapMin
}
return m.theme.Text.Width(w).Align(lipgloss.Center).Render(m.err.Error())
}
func (m Model) detailView() string {
t := m.theme
scroll := fmt.Sprintf("%3.0f%%", m.viewport.ScrollPercent()*100)
left := t.PanelTitle.Render(markOpen+"DOSSIER"+markClose) + " " +
t.Muted.Render(fmt.Sprintf("story %d of %d", m.cursor+1, len(m.scored)))
head := m.spread(left, t.Meta.Render(scroll))
foot := m.spread(
m.keyHints(m.keys.Back, m.keys.Browser, m.keys.Ideate, m.keys.Down, m.keys.Up, m.keys.Quit),
m.statusText(),
)
return lipgloss.JoinVertical(lipgloss.Left,
head,
t.rule(m.width),
m.viewport.View(),
t.rule(m.width),
foot,
)
}
func (m Model) statusText() string {
if m.generating {
return m.spinner.View() + " " + m.theme.fg(colorCyan).Render(m.status)
}
if m.status == "" {
return ""
}
color := colorCyan
if m.statusErr {
color = colorMagenta
}
return m.theme.fg(color).Render(m.status)
}

8
PROJECTS/intermediate/security-news-scraper/internal/version/version.go Normal file
View File

@ -0,0 +1,8 @@
// ©AngelaMos | 2026
// version.go
package version
const Name = "nadezhda"
var Version = "0.1.0-dev"

114
PROJECTS/intermediate/security-news-scraper/internal/watch/notify.go Normal file
View File

@ -0,0 +1,114 @@
// ©AngelaMos | 2026
// notify.go
package watch
import (
"bytes"
"context"
"encoding/json"
"fmt"
"net/http"
"strings"
"time"
)
const (
notifyMaxStatus = 300
defaultNotifyTimeout = 15 * time.Second
headerContentType = "Content-Type"
mimeJSON = "application/json"
)
type Notifier interface {
Notify(ctx context.Context, r Report) error
}
type WebhookNotifier struct {
URL string
Client *http.Client
}
type webhookPayload struct {
Text string `json:"text"`
Content string `json:"content"`
Items []webhookItem `json:"items"`
}
type webhookItem struct {
Title string `json:"title"`
URL string `json:"url"`
Score float64 `json:"score"`
MaxCVSS float64 `json:"max_cvss"`
IsKEV bool `json:"is_kev"`
CVEs []string `json:"cves,omitempty"`
Sources int `json:"sources"`
}
func (w WebhookNotifier) Notify(ctx context.Context, r Report) error {
if len(r.Notable) == 0 {
return nil
}
body, err := json.Marshal(buildPayload(r))
if err != nil {
return fmt.Errorf("notify: marshal: %w", err)
}
req, err := http.NewRequestWithContext(ctx, http.MethodPost, w.URL, bytes.NewReader(body))
if err != nil {
return fmt.Errorf("notify: new request: %w", err)
}
req.Header.Set(headerContentType, mimeJSON)
resp, err := w.client().Do(req)
if err != nil {
return fmt.Errorf("notify: post: %w", err)
}
defer resp.Body.Close()
if resp.StatusCode >= notifyMaxStatus {
return fmt.Errorf("notify: webhook returned %s", resp.Status)
}
return nil
}
func (w WebhookNotifier) client() *http.Client {
if w.Client != nil {
return w.Client
}
return &http.Client{Timeout: defaultNotifyTimeout}
}
func buildPayload(r Report) webhookPayload {
summary := summarize(r)
items := make([]webhookItem, len(r.Notable))
for i, n := range r.Notable {
items[i] = webhookItem{
Title: n.Title,
URL: n.URL,
Score: n.Score,
MaxCVSS: n.MaxCVSS,
IsKEV: n.IsKEV,
CVEs: n.CVEs,
Sources: n.Sources,
}
}
return webhookPayload{Text: summary, Content: summary, Items: items}
}
func summarize(r Report) string {
var sb strings.Builder
fmt.Fprintf(&sb, "nadezhda: %d notable %s\n", len(r.Notable), storyWord(len(r.Notable)))
for _, n := range r.Notable {
tag := ""
if n.IsKEV {
tag = " [KEV]"
}
fmt.Fprintf(&sb, "- %s%s (%s)\n", n.Title, tag, n.URL)
}
return strings.TrimRight(sb.String(), "\n")
}
func storyWord(n int) string {
if n == 1 {
return "story"
}
return "stories"
}

Some files were not shown because too many files have changed in this diff Show More