Cybersecurity-Projects/README.md at main

5.1 KiB

Raw Permalink Blame History

██████╗ ██╗     ██████╗       ███████╗ ██████╗ █████╗ ███╗   ██╗
██╔══██╗██║     ██╔══██╗      ██╔════╝██╔════╝██╔══██╗████╗  ██║
██║  ██║██║     ██████╔╝█████╗███████╗██║     ███████║██╔██╗ ██║
██║  ██║██║     ██╔═══╝ ╚════╝╚════██║██║     ██╔══██║██║╚██╗██║
██████╔╝███████╗██║           ███████║╚██████╗██║  ██║██║ ╚████║
╚═════╝ ╚══════╝╚═╝           ╚══════╝ ╚═════╝╚═╝  ╚═╝╚═╝  ╚═══╝

Data Loss Prevention scanner for files, databases, and network traffic.

This is a quick overview. Security theory, architecture, and full walkthroughs are in the learn modules.

What It Does

Scans files (PDF, DOCX, XLSX, CSV, JSON, XML, YAML, Parquet, Avro, archives, emails) for PII, credentials, financial data, and PHI
Scans databases (PostgreSQL, MySQL, MongoDB, SQLite) with schema introspection and sampling
Scans network captures (PCAP/PCAPNG) with protocol parsing, TCP reassembly, and DNS exfiltration detection
Confidence scoring pipeline: regex match, checksum validation (Luhn, Mod-97, Mod-11), context keyword proximity, entity co-occurrence
Maps findings to compliance frameworks (HIPAA, PCI-DSS, GDPR, CCPA, SOX, GLBA, FERPA)
Reports in console (Rich tables), JSON, SARIF 2.1.0, or CSV

Quick Start

bash install.sh
dlp-scan file ./data

[!TIP] This project uses just as a command runner. Type just to see all available commands.

Install: curl -sSf https://just.systems/install.sh | bash -s -- --to ~/.local/bin

Usage

dlp-scan file ./data/employees/              # scan a directory
dlp-scan file ./report.pdf -f json           # scan a file, JSON output
dlp-scan db postgres://user:pass@host/db     # scan PostgreSQL
dlp-scan db sqlite:///path/to/local.db       # scan SQLite
dlp-scan network capture.pcap               # scan network traffic
dlp-scan file ./data -f sarif -o results.sarif  # SARIF for CI/CD
dlp-scan report convert results.json -f csv  # convert report format
dlp-scan report summary results.json         # print summary stats

Global Options

--config, -c    Path to YAML config file
--verbose, -v   Enable debug logging
--version       Show version

Output Formats

Format	Flag	Use Case
Console	`-f console`	Interactive review with Rich tables
JSON	`-f json`	Structured analysis and archival
SARIF	`-f sarif`	GitHub code scanning, CI/CD integration
CSV	`-f csv`	Compliance team export, spreadsheet import

Stack

Language: Python 3.12+

CLI: Typer 0.15+ with Rich integration

Detection: Regex + checksum validators + Shannon entropy + context keyword scoring

File Formats: PyMuPDF, python-docx, openpyxl, xlrd, defusedxml, lxml, pyarrow, fastavro, extract-msg

Databases: asyncpg (PostgreSQL), aiomysql (MySQL), pymongo async (MongoDB), aiosqlite (SQLite)

Network: dpkt (PCAP parsing), TCP reassembly, DPI protocol identification, DNS exfiltration heuristics

Config: Pydantic 2.10+ models with YAML config loading (ruamel.yaml)

Quality: ruff, mypy (strict), yapf, pytest + hypothesis, structlog

Configuration

Copy .dlp-scanner.yml to your project root and customize. Key settings:

detection:
  min_confidence: 0.20        # minimum score to report
  enable_rules: ["*"]         # glob patterns for rule IDs
  allowlists:
    values: ["123-45-6789"]   # suppress known test values

output:
  format: "console"           # console, json, sarif, csv
  redaction_style: "partial"  # partial, full, none

Learn

This project includes step-by-step learning materials covering security theory, architecture, and implementation.

Module	Topic
00 - Overview	Prerequisites and quick start
01 - Concepts	DLP theory and real-world breaches
02 - Architecture	System design and data flow
03 - Implementation	Code walkthrough
04 - Challenges	Extension ideas and exercises

License

AGPLv3

5.1 KiB Raw Permalink Blame History