Cybersecurity-Projects/PROJECTS/intermediate/dlp-scanner/learn/04-CHALLENGES.md

04-CHALLENGES.md

Extension Challenges

You have built a DLP scanner with file, database, and network scanning, a confidence scoring pipeline, compliance mapping, and multi-format reporting. These challenges extend it into new territory.

Ordered by difficulty. The easy ones take an hour or two. The advanced ones are multi-day efforts that teach you skills used in production DLP systems.

Easy Challenges

Challenge 1: Add a New PII Rule (Date of Birth)

What to build: A detection rule for dates of birth in common formats: MM/DD/YYYY, YYYY-MM-DD, DD-Mon-YYYY.

Why it matters: Date of birth is classified as PHI under HIPAA's 18 identifiers and as personal data under GDPR. The 2015 Anthem breach exposed 78.8 million records including DOBs, and the combination of DOB + name + zip code is enough to uniquely identify 87% of the US population (Latanya Sweeney's research at Carnegie Mellon).

What you will learn:

• Writing regex patterns that match multiple date formats
• Adding a validation function that rejects impossible dates (month 13, day 32, Feb 30)
• Tuning base_score relative to false positive rate (dates appear everywhere)

Hints:

• Create the rule in a new file detectors/rules/pii_extended.py and add the rules list to ALL_RULES in registry.py
• Use a low base_score (0.10-0.15) because date strings are extremely common
• Context keywords like "date of birth", "dob", "birthday", "born on" should provide the majority of the signal
• The validator should parse the matched string into a real date and reject invalid ones
• Add the rule to RULE_FRAMEWORK_MAP in compliance.py with HIPAA and GDPR

Test it works: Create a text file with "Patient DOB: 03/15/1987" and "Order date: 03/15/1987". The first should score higher than the second due to context keywords.

Challenge 2: HTML Report Output

What to build: A new reporter that generates a standalone HTML file with a sortable findings table, severity color coding, and a summary chart.

Why it matters: Compliance teams often need to share scan results with non-technical stakeholders who do not have command-line tools. An HTML report that opens in a browser is more accessible than JSON or CSV.

What you will learn:

• Implementing the reporter pattern (match the existing protocol)
• HTML template generation in Python (string templates or Jinja2)
• Adding a new output format to the CLI without modifying existing code

Hints:

• Create reporters/html_report.py with a HtmlReporter class
• Add "html" to REPORTER_MAP in engine.py and VALID_FORMATS in commands/scan.py
• Use inline CSS so the report is a single self-contained HTML file with no external dependencies
• Color severity levels using the same scheme as the console reporter (red for critical, yellow for medium, green for low)
• Include a summary section at the top with counts by severity and framework

Test it works: Run dlp-scan file ./data -f html -o report.html and open the file in a browser. The table should be sortable by clicking column headers (add minimal JavaScript for this).

Challenge 3: Allowlist by File Path Pattern

What to build: Extend the allowlist system to suppress findings from files matching glob patterns. Currently, allowlists.file_patterns exists in the config but is not enforced during scanning.

Why it matters: Test fixtures, mock data, and seed files intentionally contain fake PII. Teams waste hours triaging findings from tests/fixtures/sample_data.csv that contain test credit card numbers. Path-based allowlisting eliminates this noise.

What you will learn:

• Connecting config to scan-time behavior
• Glob pattern matching with fnmatch
• The difference between value-level and file-level suppression

Hints:

• The AllowlistConfig.file_patterns field already exists in config.py
• Add the check in FileScanner._scan_file before running detection, or in _scan_directory before scanning the file
• Match against the relative path from the scan target, not the absolute path
• Patterns like test_*, *_fixture*, and mock_* should match filenames

Test it works: Create a file test_data.txt with a valid SSN. Scan with file_patterns: ["test_*"] in config. The SSN should not appear in results.

Intermediate Challenges

Challenge 4: Incremental Scanning with Hash Cache

What to build: A scan cache that stores SHA-256 hashes of scanned files and skips unchanged files on subsequent scans.

Why it matters: Large codebases and file shares contain millions of files. Re-scanning unchanged files wastes time. Symantec DLP and Microsoft Purview both use content hashing to skip unchanged files, reducing scan time by 60-90% on repeated scans.

What you will learn:

• Content-addressable caching strategies
• SQLite as an embedded metadata store
• Cache invalidation (the hardest problem in computer science, and one you actually have to solve)

Implementation approach:

1. Create cache.py with a ScanCache class backed by SQLite

   • Table: (file_path TEXT, content_hash TEXT, scan_time TEXT, finding_count INTEGER)
   • Hash computation: SHA-256 of file contents
   • Lookup: if the file exists in cache with the same hash, skip scanning and load cached finding count

2. Integrate with FileScanner

   • Before extracting text, check the cache
   • After scanning, store the hash and finding count
   • Add --no-cache flag to force full rescan

3. Handle invalidation edge cases:

   • What if detection rules change between scans? (The same file might produce different findings with new rules)
   • What if the config changes min_confidence? (Previously-suppressed findings might now be reportable)
   • What if a file is deleted? (Stale cache entries should not appear in results)

Hints:

• Store a hash of the active rule set and config in the cache. If either changes, invalidate the entire cache
• Use aiosqlite to match the async pattern of the database scanner, or use synchronous sqlite3 since file scanning is already synchronous
• The cache file should live next to the config: .dlp-scanner-cache.db

Extra credit: Add dlp-scan cache stats and dlp-scan cache clear subcommands.

Challenge 5: Severity Override by Compliance Framework

What to build: A config option that overrides severity based on compliance framework requirements. For example, any PCI-DSS finding should be at least "high" regardless of confidence score, because PCI-DSS does not have a concept of "low severity" unencrypted card data.

Why it matters: Different compliance frameworks have different severity thresholds. GDPR treats unencrypted email addresses as medium priority for remediation, but PCI-DSS treats any unencrypted PAN as a blocking finding. Production DLP tools let compliance teams configure per-framework severity floors.

What you will learn:

• Adding config-driven behavior to the scoring pipeline
• The tension between confidence-based and policy-based severity
• How production DLP tools balance detection accuracy with compliance requirements

Implementation approach:

1. Add to config:

   compliance:
     severity_overrides:
       PCI_DSS: "high"
       HIPAA: "medium"
   

2. Apply after scoring: In the _match_to_finding function (or equivalent), after computing severity from confidence, check if any of the finding's compliance frameworks have a severity floor, and upgrade if necessary

3. Preserve original confidence: The confidence score should not change. Only the severity classification changes. This lets analysts see that a finding scored 0.35 (normally "low") but was elevated to "high" because of PCI-DSS policy

Hints:

• Add severity_overrides: dict[str, str] to ComplianceConfig in config.py
• Use SEVERITY_ORDER from constants.py to compare severity levels numerically
• Log when a severity is overridden so analysts understand why a low-confidence finding shows up as high severity

Challenge 6: Database Column Name Heuristic Scoring

What to build: A pre-scan heuristic that boosts detection confidence for columns whose names suggest sensitive data (e.g., ssn, credit_card_number, patient_dob).

Why it matters: Database schema names are strong metadata signals. A column named ssn in a table named employees is almost certainly storing Social Security Numbers, even before you look at the data. The Capital One breach investigation found that the compromised S3 bucket contained CSV exports with column headers like SSN and AccountNumber, which would have been trivially detectable with column-name analysis.

What you will learn:

• Schema introspection as a detection signal
• Combining metadata and content signals
• How production DLP tools use schema analysis to prioritize scanning

Implementation approach:

1. Create a column name classifier with patterns mapping column names to rule IDs:

   *ssn*, *social_sec* -> PII_SSN
   *credit_card*, *card_num*, *pan* -> FIN_CREDIT_CARD
   *email*, *e_mail* -> PII_EMAIL
   *dob*, *date_of_birth*, *birthday* -> PII_DOB
   

2. Apply as a context boost in the database scanner: when a column name matches a pattern, add a pre-boost to the base score before running the normal detection pipeline

3. Carry through to findings: Add the column name match as additional evidence in the finding's metadata

Hints:

• Implement this in scanners/db_scanner.py before the detection loop
• Use fnmatch for column name pattern matching (same as rule filtering)
• A modest boost (+0.15 to +0.25) is appropriate. Column names are strong signals but not definitive (a column named ssn_backup_old might be empty or encrypted)

Advanced Challenges

Challenge 7: Custom Rule Language

What to build: A YAML-based rule definition format that lets users create detection rules without writing Python. Rules should support regex patterns, base scores, context keywords, and compliance framework tags.

Why it matters: Production DLP tools (Symantec DLP, Netskope) let compliance teams define custom rules through policy editors because not every regulated data type is covered by built-in rules. European IBANs, Brazilian CPFs, Indian Aadhaar numbers, and industry-specific identifiers all need custom patterns.

What you will learn:

• DSL design (keeping it simple enough to be useful, complex enough to be powerful)
• Safe regex compilation (preventing ReDoS)
• Hot reloading user-defined rules

Implementation approach:

1. Define the rule schema:

   rules:
     - id: CUSTOM_BR_CPF
       name: "Brazilian CPF Number"
       pattern: '\b\d{3}\.\d{3}\.\d{3}-\d{2}\b'
       base_score: 0.40
       context_keywords: ["cpf", "cadastro"]
       compliance: ["LGPD"]
       validator: "mod11"
   

2. Build a rule loader that reads YAML files from a rules/ directory, compiles regex patterns safely (with timeout protection against catastrophic backtracking), and creates DetectionRule objects

3. Register custom rules alongside built-in rules in the DetectorRegistry

4. Add built-in validator references (mod11, luhn, mod97) that users can reference by name instead of writing Python

Gotchas:

• Regex compilation must be safe: a user-provided pattern like (a+)+b causes catastrophic backtracking. Consider using the regex library with timeout, or validate patterns against known ReDoS patterns
• Custom rules should not be able to override or shadow built-in rules. Use ID prefixes (CUSTOM_) to namespace them
• Validator functions referenced by name need a registry of their own

Challenge 8: Real-Time File Monitoring

What to build: A watch mode that monitors directories for file changes using filesystem events and scans new or modified files automatically.

Why it matters: Batch scanning finds problems after the fact. Real-time monitoring catches sensitive data as soon as it hits disk. This is how endpoint DLP agents (CrowdStrike Falcon DLP, Digital Guardian) work: they hook filesystem events and scan in real time.

What you will learn:

• Filesystem event monitoring with watchdog or inotify
• Event debouncing (a single file save can trigger multiple events)
• Background scanning without blocking the event loop

Architecture changes:

┌──────────────────────────────┐
│  FileSystemEventHandler       │
│  (watchdog or inotify)        │
│                               │
│  on_modified -> debounce      │
│  on_created  -> scan_file     │
│  on_moved    -> scan_dest     │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│  ScanQueue (asyncio.Queue)    │
│                               │
│  Dedup by path               │
│  Rate limit scanning          │
└──────────────┬───────────────┘
               │
               ▼
┌──────────────────────────────┐
│  FileScanner.scan(file)       │
│  → Finding → Alert            │
└──────────────────────────────┘

Implementation steps:

1. Add watchdog as a dependency
2. Create commands/watch.py with a dlp-scan watch ./directory command
3. Implement a debouncer that batches filesystem events within a 500ms window
4. Use the existing FileScanner._scan_file for individual file scanning
5. Output findings to console in real time (stream mode, not batch)

Gotchas:

• Editor save operations often create temporary files, write to them, then rename. This generates create, modify, and rename events. You need to scan the final file, not the intermediate temp files
• Large file copies trigger on_modified repeatedly as data is written. Debounce by waiting until the file size stabilizes
• The watch mode should respect the same exclude patterns and extension filters as batch scanning

Challenge 9: SIEM Integration via Syslog

What to build: A reporter that sends findings to a SIEM (Splunk, Elastic, QRadar) via syslog (RFC 5424) or HTTP Event Collector (Splunk HEC).

Why it matters: DLP findings are useless if they sit in a JSON file that nobody reads. Production DLP deployments send alerts to SIEMs where SOC analysts triage them alongside firewall logs, EDR alerts, and authentication events. Correlating a DLP finding with a VPN login from an unusual location turns a medium-severity alert into an incident.

What you will learn:

• Syslog protocol formatting (RFC 5424 structured data)
• HTTP-based log shipping (Splunk HEC, Elastic Ingest)
• Alert fatigue management (batching, deduplication, severity filtering)

Implementation approach:

1. Create reporters/syslog_reporter.py that formats findings as RFC 5424 syslog messages:

   <134>1 2026-04-08T10:30:00Z scanner dlp-scan - -
   [finding@dlp rule_id="PII_SSN" severity="critical"
   confidence="0.92" uri="employees.csv"] SSN detected
   

2. Add Splunk HEC support as an alternative transport: POST JSON payloads to https://splunk:8088/services/collector/event with an HEC token

3. Add config:

   output:
     siem:
       type: "syslog"        # or "splunk_hec"
       host: "siem.corp.com"
       port: 514
       protocol: "tcp"       # or "udp"
       hec_token: ""         # for Splunk HEC
   

4. Implement batching: Send findings in batches of 50 with a 5-second flush interval to avoid overwhelming the SIEM

Expert Challenges

Challenge 10: Machine Learning False Positive Reduction

What to build: A feedback loop where analysts can mark findings as true positive or false positive, and a classifier learns to suppress likely false positives on future scans.

Why it matters: The single biggest complaint about DLP tools is false positive volume. Symantec DLP deployments commonly see 40-60% false positive rates on initial rollout. Analysts spend hours dismissing findings that match SSN patterns but are actually serial numbers, batch IDs, or zip+4 codes. A classifier trained on analyst feedback can reduce false positives by 70-80% while maintaining detection recall.

What you will learn:

• Feature engineering from detection signals (confidence, context keywords found, rule type, file type, surrounding text patterns)
• Online learning: updating a model as new feedback arrives without retraining from scratch
• The precision-recall tradeoff in security tooling (a false negative is a missed breach; a false positive is analyst fatigue)

Implementation phases:

Phase 1: Feedback Collection

• Add dlp-scan feedback <finding_id> --true-positive and --false-positive commands
• Store feedback in a SQLite database: (finding_id, rule_id, features_json, label, timestamp)
• Extract features: confidence, rule_id, file extension, context keywords matched, co-occurrence count, surrounding text entropy

Phase 2: Classifier

• Train a logistic regression or gradient boosted tree on accumulated feedback
• Features: one-hot encode rule_id, numeric confidence, boolean context_found, file_extension category
• Use scikit-learn with ONNX export for deployment without the full sklearn dependency

Phase 3: Integration

• After the detection pipeline produces matches, run the classifier as a post-filter
• Matches classified as likely false positives get demoted (severity lowered, or moved to a "suppressed" section)
• Never fully suppress a detection. Always show suppressed findings in a separate section so analysts can audit the classifier

Success criteria:

• Feedback collection works and stores features
• Classifier trains on 50+ labeled examples
• False positive rate drops by at least 30% on held-out test set
• No true positives are fully suppressed (only demoted)
• Model retrains automatically when feedback count crosses thresholds (100, 500, 1000)

Real-World Integration Challenges

Integrate with GitHub Code Scanning

The goal: Upload SARIF output to GitHub Code Scanning so DLP findings appear as annotations on pull requests.

What you will learn:

• GitHub Code Scanning API
• SARIF upload via GitHub Actions
• CI/CD pipeline integration for security tooling

Steps:

1. Create a GitHub Actions workflow that runs dlp-scan file . -f sarif -o results.sarif on pull requests
2. Upload the SARIF file using the github/codeql-action/upload-sarif action
3. Configure on: pull_request to scan only changed files (use git diff --name-only to get the list)
4. Set severity filtering so only high/critical findings block the PR

Scan AWS S3 Buckets

The goal: Add an S3 scanner that lists objects in a bucket, downloads them to a temp directory, and scans with the existing file scanner.

What you will learn:

• boto3 integration for S3 object listing and download
• Temporary file management for large object scanning
• Credential handling (IAM roles vs. access keys)

Steps:

1. Add dlp-scan s3 s3://bucket-name/prefix command
2. Use boto3 to list objects, filter by extension
3. Download each object to a temp directory (use tempfile.mkdtemp)
4. Scan with FileScanner and map findings back to S3 URIs
5. Clean up temp files after scanning

This directly addresses the Capital One breach scenario: unencrypted PII in S3 buckets that nobody knew existed.

Performance Challenge

Handle 1 Million Files

The goal: Make the file scanner handle a directory with 1 million files without running out of memory or taking more than an hour.

Current bottleneck: Path.rglob("*") generates a list of all files before scanning starts. With 1 million files, this consumes significant memory and delays the first scan result.

Optimization approaches:

Approach 1: Streaming directory walk

• Replace rglob with os.scandir recursive walk that yields files one at a time
• Process and discard each file before reading the next
• Memory stays constant regardless of directory size

Approach 2: Parallel extraction

• Use concurrent.futures.ThreadPoolExecutor for I/O-bound extraction (file reads, PDF parsing)
• Use concurrent.futures.ProcessPoolExecutor for CPU-bound detection (regex matching on large texts)
• Tune pool sizes based on profiling

Approach 3: Prioritized scanning

• Scan high-risk extensions first (.csv, .xlsx, .sql) before low-risk ones (.log, .txt)
• Report findings as they are discovered (streaming output) instead of waiting for the full scan to complete

Benchmark it:

time dlp-scan file /large-directory -f json -o results.json

Target: under 60 minutes for 1 million files with an average file size of 10KB.

Challenge Completion

Track your progress:

• Easy 1: Date of Birth Rule
• Easy 2: HTML Report Output
• Easy 3: Allowlist by File Path
• Intermediate 4: Incremental Scanning
• Intermediate 5: Severity Override
• Intermediate 6: Column Name Heuristic
• Advanced 7: Custom Rule Language
• Advanced 8: Real-Time Monitoring
• Advanced 9: SIEM Integration
• Expert 10: ML False Positive Reduction
• Integration: GitHub Code Scanning
• Integration: S3 Bucket Scanning
• Performance: 1 Million Files

Study Real Implementations

Compare your work to production DLP tools:

• Nightfall AI: Cloud-native DLP with ML-based detection. Open-sourced their detection patterns. Look at how they handle multi-format extraction
• truffleHog: Focuses on credential detection in git repos. Their entropy-based detection and regex patterns for API keys are similar to this project's credential rules
• detect-secrets: Yelp's secret scanner. Compare their plugin architecture to the detector registry pattern in this project
• Microsoft Purview: Enterprise DLP with 300+ built-in sensitive information types. Their documentation on exact data match (EDM) and trainable classifiers shows where the field is heading