RomanNum3ral
/
Cybersecurity-Projects
mirror of https://github.com/CarterPerez-dev/Cybersecurity-Projects.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
Cybersecurity-Projects/PROJECTS/foundations/http-headers-scanner/README.md

11 KiB
Raw Permalink Blame History 

██╗  ██╗███████╗ █████╗ ██████╗ ███████╗██████╗ ███████╗
██║  ██║██╔════╝██╔══██╗██╔══██╗██╔════╝██╔══██╗██╔════╝
███████║█████╗  ███████║██║  ██║█████╗  ██████╔╝███████╗
██╔══██║██╔══╝  ██╔══██║██║  ██║██╔══╝  ██╔══██╗╚════██║
██║  ██║███████╗██║  ██║██████╔╝███████╗██║  ██║███████║
╚═╝  ╚═╝╚══════╝╚═╝  ╚═╝╚═════╝ ╚══════╝╚═╝  ╚═╝╚══════╝

Cybersecurity Projects Tier: Foundations Python 3.13 License: AGPLv3 Tests Lint HTTP Client

Fetch a URL once and grade its HTTP security headers A through F using the same weighted-rubric model as Mozilla Observatory.

This is a quick overview, security theory, architecture, and full walkthroughs are in the learn modules.

[!NOTE] Foundations tier, this project is built for someone who has never written Python before. The source code is heavily commented as a teaching aid, the learn/ folder explains every concept from zero, and the whole tool is one readable file. If you already know Python, the natural next step is PROJECTS/foundations/password-manager, the hardest foundations project, which adds Argon2id, AES-GCM, and on-disk state.

What It Does

  • Performs one polite HTTPS request to the URL you provide and inspects the response headers
  • Grades six security-critical headers against a weighted rubric (high = 30 pts, medium = 15 pts, low = 5 pts)
  • Reports each finding as ok, weak, or missing with a one-line explanation of why
  • Computes a 0 to 100 score and maps it to an A through F letter grade (90+ = A, 80+ = B, etc.)
  • Catches subtly broken values like Strict-Transport-Security: max-age=0 (header present, actively disabled) and flags them as weak, not ok
  • Follows redirects and grades the final URL, the one your browser would actually land on
  • Prints a colored Rich table plus a grade panel plus a recommendation list for every non-ok finding
  • Returns meaningful exit codes: 0 for A/B, 1 for C/D, 2 for F or network error, useful in CI pipelines

The Headers It Grades

Header Severity What it stops
Strict-Transport-Security high SSL stripping on coffee-shop wifi
Content-Security-Policy high XSS via injected <script> tags
X-Content-Type-Options medium MIME-sniffing of uploaded files
X-Frame-Options medium Clickjacking via hidden iframes
Referrer-Policy low Leaking secret tokens via the Referer header
Permissions-Policy low Compromised third-party scripts abusing camera/mic/etc.

Every header maps to a real attack class with a real history. The 01-CONCEPTS.md module walks through each one with concrete attack examples.

Quick Start

./install.sh
just run -- https://example.com
# Grade: B, Score: 85 / 100  (example.com is missing CSP and Permissions-Policy)

[!TIP] This project uses just as a command runner. Type just to see all available commands.

Install: curl -sSf https://just.systems/install.sh | bash -s -- --to ~/.local/bin

Demo URLs

Try these, each demonstrates a different grading path:

URL Expected grade Why
https://github.com A Comprehensive CSP, HSTS with includeSubDomains, almost every header set
https://web.dev A Google's own developer-docs site, full modern header set
https://mozilla.org A Mozilla practices what Observatory preaches
https://example.com B / C Has HSTS, but missing CSP, Permissions-Policy, and others
http://neverssl.com F Intentionally serves plain HTTP, no security headers at all 
just run -- https://github.com
just run -- https://example.com
just run -- https://web.dev --timeout 5
just run -- http://neverssl.com

[!IMPORTANT] Always include the http:// or https:// scheme. The scanner refuses bare hostnames like github.com because it cannot guess which scheme you meant, and guessing wrong is exactly the SSL-stripping problem HSTS exists to prevent.

Sample Output

                  Headers for https://github.com/ (HTTP 200)
┌─────────────────────────────┬─────────┬──────────┬─────────────────────────┐
│ header                      │ status  │ severity │ note                    │
├─────────────────────────────┼─────────┼──────────┼─────────────────────────┤
│ Strict-Transport-Security   │ ok      │ high     │ Present and contains... │
│ Content-Security-Policy     │ ok      │ high     │ Present                 │
│ X-Content-Type-Options      │ ok      │ medium   │ Present and contains... │
│ X-Frame-Options             │ ok      │ medium   │ Present                 │
│ Referrer-Policy             │ ok      │ low      │ Present                 │
│ Permissions-Policy          │ missing │ low      │ Header ... is not set   │
└─────────────────────────────┴─────────┴──────────┴─────────────────────────┘
╭─ Result ───────────────────╮
│ Grade: A                   │
│ Score: 95 / 100            │
╰────────────────────────────╯

Followed by a Recommendations: block for every non-ok finding, with the exact header value to add.

Exit Codes

The scanner returns shell-friendly exit codes so you can wire it into CI:

Grade Exit code Meaning
A, B 0 Green light, no action needed
C, D 1 Worth investigating, often acceptable depending on context
F or network error 2 Hard fail, must fix 
just run -- https://my-deployed-site.com
if [ $? -gt 1 ]; then exit 1; fi   # fail the build only on F or error

Tooling

just            # list available recipes
just test       # run pytest (11 tests, runs in under a second, network-mocked with respx)
just lint       # ruff + mypy --strict + pylint
just format     # yapf
just run -- <url>  # scan a URL

Requirements

  • Python 3.13+, the install script will check.
  • uv, modern Python package manager (auto-installed by ./install.sh).
  • just, command runner (auto-installed by ./install.sh).
  • A working internet connection at runtime (the scanner makes one real HTTPS request per scan, but the test suite mocks the network with respx and runs fully offline).

No compilers, no system libraries. The project is one Python file plus tests.

Learn

This project includes step-by-step learning materials covering security theory, architecture, and implementation, written for someone who has never touched Python before.

Module Topic
00 - Overview Quick start, prerequisites, expected output, common first-run problems
01 - Concepts What HTTP is, what a header is, each security header with the real attack it stops (SSL stripping, clickjacking, MIME sniffing, XSS, referer leakage)
02 - Architecture The four-stage pipeline, dataclasses as value objects, the I/O fence pattern (functional core / imperative shell)
03 - Implementation Function-by-function walkthrough, every Python feature explained when first encountered, plus test patterns and tooling
04 - Challenges Twelve extension ideas, from "add a seventh header rule" up through "wrap it in a FastAPI service with rate limiting"

Real-World Context

This scanner is a teaching-scale version of tools that do the same job at production scale:

Once you understand how this scanner makes decisions, those tools become readable instead of magical. The 04-CHALLENGES.md module includes ideas for growing this project toward what Observatory does.

See Also

License

AGPL 3.0