docs: update learn/ for dlp-scanner audit changes
Reflect network_scanner.py rewrite, new scoring.py module, per-brand credit card rule IDs, NPI validator, FL/IL driver's license rules, Literal config types, and network exfil compliance mappings.
This commit is contained in:
CarterPerez-dev
parent
8d5a177bca
commit
2ba9985936
|
|
@ -165,13 +165,13 @@ Text Input
|
|||
|
||||
**Files:** `scanners/file_scanner.py`, `scanners/db_scanner.py`, `scanners/network_scanner.py`
|
||||
|
||||
All scanners follow the same `Scanner` protocol: a `scan(target: str) -> ScanResult` method. They share a common flow: iterate over targets, extract text, run detection, classify matches into findings with severity/compliance/remediation metadata, and aggregate into a `ScanResult`.
|
||||
All scanners follow the same `Scanner` protocol: a `scan(target: str) -> ScanResult` method. They share a common flow: iterate over targets, extract text, run detection, convert matches to findings via `match_to_finding` in `scoring.py` (which handles severity classification, compliance lookup, remediation, and redaction in one call), and aggregate into a `ScanResult`.
|
||||
|
||||
**FileScanner** walks a directory tree, applies extension and exclusion filters, dispatches each file to the appropriate extractor based on extension, and runs the detector on each `TextChunk`. The extension-to-extractor mapping is built once by `_build_extension_map`, which iterates over all extractor instances and indexes by their `supported_extensions`.
|
||||
|
||||
**DatabaseScanner** connects via URI scheme detection (postgres, mysql, mongodb, sqlite), introspects the schema to find text-type columns, samples rows using database-native sampling (TABLESAMPLE BERNOULLI for PostgreSQL, RAND() for MySQL, $sample for MongoDB), and scans column values.
|
||||
|
||||
**NetworkScanner** reads PCAP files using dpkt, extracts TCP/UDP payloads, decodes them to text, and runs detection. The companion modules in `network/` provide TCP stream reassembly, DNS query parsing, protocol identification via DPI, and DNS exfiltration detection.
|
||||
**NetworkScanner** reads PCAP files via `read_pcap`, feeds packets into a `FlowTracker` for TCP reassembly, and processes DNS traffic inline through `parse_dns` and `DnsExfilDetector`. Each packet payload is also checked by `detect_base64_payload` for encoded data. After packet iteration, the scanner reassembles TCP flows, identifies the application protocol via `identify_protocol`, extracts text with protocol awareness (`parse_http` for HTTP bodies and sensitive headers, skip encrypted TLS/SSH, UTF-8 decode for everything else), and runs detection on the extracted text.
|
||||
|
||||
### Extractors
|
||||
|
||||
|
|
@ -317,7 +317,7 @@ class DetectorMatch:
|
|||
└────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
Every configuration value has a constant default defined in `constants.py`. The Pydantic models in `config.py` use these constants as field defaults, so a completely empty config file produces a working scanner. The config loader uses `ruamel.yaml` (not PyYAML) because it preserves comments and handles YAML 1.2.
|
||||
Every configuration value has a constant default defined in `constants.py`. The Pydantic models in `config.py` use these constants as field defaults, so a completely empty config file produces a working scanner. Constrained-choice fields (`severity_threshold`, `format`, `redaction_style`) use `Literal` types defined in `constants.py` (e.g., `Literal["critical", "high", "medium", "low"]`), so Pydantic rejects invalid values at parse time rather than silently accepting a typo. The config loader uses `ruamel.yaml` (not PyYAML) because it preserves comments and handles YAML 1.2.
|
||||
|
||||
The YAML structure uses a `scan:` top-level key to group scanner-specific config, while `detection:`, `compliance:`, `output:`, and `logging:` sit at root level. This mirrors how users think about configuration: "how to scan" vs. "what to detect" vs. "how to report".
|
||||
|
||||
|
|
@ -334,7 +334,7 @@ Step-by-step walkthrough of `dlp-scan file ./data -f json`:
|
|||
(WARNING for machine-readable formats keeps stdout clean)
|
||||
|
||||
3. ScanEngine(config) constructs DetectorRegistry
|
||||
└─► Registry loads 28 rules from PII/Financial/Credential/Health modules
|
||||
└─► Registry loads 29 rules from PII/Financial/Credential/Health modules
|
||||
└─► Filters through enable_rules=["*"], disable_rules=[]
|
||||
|
||||
4. engine.scan_files("./data")
|
||||
|
|
@ -355,10 +355,11 @@ Step-by-step walkthrough of `dlp-scan file ./data -f json`:
|
|||
└─► EntropyDetector: high-entropy region detection
|
||||
|
||||
7. For each DetectorMatch above min_confidence:
|
||||
└─► score_to_severity(match.score) -> Severity
|
||||
└─► get_frameworks_for_rule(match.rule_id) -> compliance list
|
||||
└─► get_remediation_for_rule(match.rule_id) -> guidance string
|
||||
└─► redact(chunk.text, start, end, style="partial") -> snippet
|
||||
└─► match_to_finding(match, text, location, redaction_style)
|
||||
├─► score_to_severity(match.score) -> Severity
|
||||
├─► get_frameworks_for_rule(match.rule_id) -> compliance list
|
||||
├─► get_remediation_for_rule(match.rule_id) -> guidance string
|
||||
└─► redact(chunk.text, start, end, style) -> snippet
|
||||
└─► Append Finding to ScanResult
|
||||
|
||||
8. Back in _run_scan():
|
||||
|
|
@ -413,7 +414,9 @@ RULE_FRAMEWORK_MAP: rule_id -> [frameworks]
|
|||
RULE_REMEDIATION_MAP: rule_id -> guidance string
|
||||
```
|
||||
|
||||
When a `DetectorMatch` is converted to a `Finding` inside a scanner, the scanner calls `get_frameworks_for_rule` and `get_remediation_for_rule` to decorate the finding with compliance metadata. If the detection rule itself also carries `compliance_frameworks`, both sets are merged.
|
||||
Rule IDs match actual detection rules (e.g., `FIN_CREDIT_CARD_VISA`, `FIN_CREDIT_CARD_MC`, not a generic `FIN_CREDIT_CARD`). Network exfiltration indicators (`NET_DNS_EXFIL_*`, `NET_ENCODED_*`) are also mapped. Every rule has a remediation entry with specific guidance text; unknown rules fall back to a generic default.
|
||||
|
||||
When a `DetectorMatch` is converted to a `Finding` via `match_to_finding` in `scoring.py`, the function calls `get_frameworks_for_rule` and `get_remediation_for_rule` to decorate the finding with compliance metadata. If the detection rule itself also carries `compliance_frameworks`, both sets are merged.
|
||||
|
||||
This design keeps detection rules independent of compliance logic. The PII module does not need to know that HIPAA cares about SSNs. The compliance module owns that mapping, and it can be updated independently when regulations change.
|
||||
|
||||
|
|
@ -508,7 +511,7 @@ This "collect and continue" approach means a single corrupt PDF in a directory o
|
|||
|
||||
**File scanning** is I/O-bound. The scanner processes files sequentially to avoid overwhelming disk I/O. Text extraction for binary formats (PDF, Office) can be CPU-intensive, but these files are typically a small fraction of the total.
|
||||
|
||||
**Detection** scales linearly with text length times rule count. With 28 rules and an average text chunk of 500 lines, a single detection pass takes microseconds. The entropy detector is more expensive due to its sliding window, so it only runs when enabled and only against high-level text chunks (not individual regex matches).
|
||||
**Detection** scales linearly with text length times rule count. With 29 rules and an average text chunk of 500 lines, a single detection pass takes microseconds. The entropy detector is more expensive due to its sliding window, so it only runs when enabled and only against high-level text chunks (not individual regex matches).
|
||||
|
||||
**Memory** stays bounded through chunking. The plaintext extractor reads 500 lines at a time. Archive extraction enforces depth limits and zip bomb ratio checks.
|
||||
|
||||
|
|
@ -520,6 +523,7 @@ This "collect and continue" approach means a single corrupt PDF in a directory o
|
|||
- `constants.py` - All magic numbers, thresholds, type literals
|
||||
- `models.py` - Finding, Location, ScanResult, TextChunk
|
||||
- `compliance.py` - Rule-to-framework mapping, severity classification
|
||||
- `scoring.py` - Shared match-to-finding conversion for all scanners
|
||||
- `redaction.py` - Partial/full/none redaction strategies
|
||||
- `detectors/registry.py` - Rule loading, filtering, scoring pipeline
|
||||
- `detectors/pattern.py` - Regex matching with allowlist and checksum validation
|
||||
|
|
|
|||
|
|
@ -17,6 +17,7 @@ src/dlp_scanner/
|
|||
├── compliance.py # Rule-to-framework mapping
|
||||
├── redaction.py # Snippet masking
|
||||
├── log.py # structlog configuration
|
||||
├── scoring.py # Shared match-to-finding conversion
|
||||
├── commands/
|
||||
│ ├── scan.py # file, db, network commands
|
||||
│ └── report.py # convert, summary commands
|
||||
|
|
@ -174,6 +175,28 @@ def nhs_check(value: str) -> bool:
|
|||
|
||||
Multiply the first 9 digits by descending weights (10, 9, 8, ..., 2), sum them, compute `11 - (sum mod 11)`, and compare to the check digit. If the result is 10, the number is invalid (NHS never issues these). If the result is 11, the check digit is 0.
|
||||
|
||||
**Luhn-80840** for NPIs (in `detectors/rules/health.py`):
|
||||
|
||||
```python
|
||||
def _validate_npi(value: str) -> bool:
|
||||
digits = value.replace("-", "").replace(" ", "")
|
||||
if len(digits) != 10 or not digits.isdigit():
|
||||
return False
|
||||
|
||||
prefixed = "80840" + digits
|
||||
total = 0
|
||||
for i, d in enumerate(reversed(prefixed)):
|
||||
n = int(d)
|
||||
if i % 2 == 1:
|
||||
n *= 2
|
||||
if n > 9:
|
||||
n -= 9
|
||||
total += n
|
||||
return total % 10 == 0
|
||||
```
|
||||
|
||||
NPI (National Provider Identifier) validation is a Luhn variant. The trick is prepending `80840` (the healthcare industry prefix assigned by ANSI) before running the standard Luhn algorithm. This prefix is not part of the NPI itself, but the ISO standard requires it for check digit computation. A random 10-digit number has about a 10% chance of passing, making this check useful but not definitive. The base score of 0.10 reflects that NPI patterns match many unrelated 10-digit numbers, and context keywords like "provider" or "npi" are needed to push the score into actionable territory.
|
||||
|
||||
### Pattern Detection
|
||||
|
||||
The `PatternDetector` in `detectors/pattern.py` iterates over all active rules, runs each regex against the input text, filters through the allowlist, and applies checksum validation:
|
||||
|
|
@ -473,6 +496,79 @@ The `_get_full_suffix` function handles compound extensions like `.tar.gz` and `
|
|||
|
||||
## Network Analysis
|
||||
|
||||
### Scanner Orchestration
|
||||
|
||||
The `NetworkScanner` ties together the network modules into a multi-pass pipeline. The old implementation decoded raw packets as UTF-8 and ran detection directly. The rewrite is protocol-aware:
|
||||
|
||||
```python
|
||||
def _scan_pcap(self, path, result):
|
||||
tracker = FlowTracker()
|
||||
dns_detector = DnsExfilDetector(
|
||||
entropy_threshold=(
|
||||
self._net_config.dns_label_entropy_threshold
|
||||
),
|
||||
)
|
||||
packet_count = 0
|
||||
|
||||
for packet in read_pcap(
|
||||
path,
|
||||
max_packets=self._net_config.max_packets,
|
||||
):
|
||||
packet_count += 1
|
||||
tracker.add_packet(packet)
|
||||
|
||||
if (
|
||||
packet.protocol == "udp"
|
||||
and (
|
||||
packet.src_port == DNS_PORT
|
||||
or packet.dst_port == DNS_PORT
|
||||
)
|
||||
):
|
||||
self._process_dns_packet(
|
||||
packet.payload, packet.src_ip,
|
||||
packet.dst_ip, path, packet_count,
|
||||
dns_detector, result,
|
||||
)
|
||||
|
||||
if packet.payload:
|
||||
exfil_indicators = detect_base64_payload(
|
||||
packet.payload,
|
||||
src_ip=packet.src_ip,
|
||||
dst_ip=packet.dst_ip,
|
||||
)
|
||||
for indicator in exfil_indicators:
|
||||
finding = _indicator_to_finding(
|
||||
indicator, str(path), packet_count,
|
||||
)
|
||||
result.findings.append(finding)
|
||||
|
||||
txt_indicators = dns_detector.check_txt_volume()
|
||||
for indicator in txt_indicators:
|
||||
...
|
||||
|
||||
self._scan_reassembled_flows(tracker, path, result)
|
||||
```
|
||||
|
||||
Three things happen during the packet loop: every packet goes into the `FlowTracker` for later TCP reassembly, UDP packets on port 53 are parsed as DNS and fed to the `DnsExfilDetector`, and every payload is checked for base64/hex-encoded data by `detect_base64_payload`. After the loop, TXT query volume ratios are checked and TCP flows are reassembled for content scanning.
|
||||
|
||||
The reassembled flow scanning uses protocol-aware text extraction:
|
||||
|
||||
```python
|
||||
def _extract_scannable_text(self, stream, protocol):
|
||||
if protocol == "http":
|
||||
return self._extract_http_text(stream)
|
||||
if protocol in ("tls", "ssh"):
|
||||
return ""
|
||||
try:
|
||||
return stream.decode("utf-8", errors="replace")
|
||||
except Exception:
|
||||
return ""
|
||||
```
|
||||
|
||||
HTTP flows get parsed by `parse_http`, which extracts URIs, sensitive headers (`cookie`, `authorization`, `set-cookie`), and bodies. TLS and SSH flows are skipped entirely since the content is encrypted and cannot be scanned. Everything else falls through to a UTF-8 decode attempt.
|
||||
|
||||
DNS exfiltration indicators and encoded payload detections are converted to `Finding` objects through `_indicator_to_finding`, which maps indicator types to rule IDs via the `EXFIL_RULE_MAP` lookup table. Regex-based detections from reassembled flows go through `match_to_finding` like the other scanners.
|
||||
|
||||
### PCAP Parsing
|
||||
|
||||
The `read_pcap` function in `network/pcap.py` reads packets using dpkt and yields `PacketInfo` structs:
|
||||
|
|
@ -638,17 +734,77 @@ The `RULE_FRAMEWORK_MAP` in `compliance.py` is a static lookup table:
|
|||
```python
|
||||
RULE_FRAMEWORK_MAP = {
|
||||
"PII_SSN": ["HIPAA", "CCPA", "GLBA", "GDPR"],
|
||||
"FIN_CREDIT_CARD": ["PCI_DSS", "GLBA"],
|
||||
"PII_DRIVERS_LICENSE_FL": ["CCPA", "HIPAA"],
|
||||
"FIN_CREDIT_CARD_VISA": ["PCI_DSS", "GLBA"],
|
||||
"FIN_CREDIT_CARD_MC": ["PCI_DSS", "GLBA"],
|
||||
"FIN_IBAN": ["GDPR", "GLBA"],
|
||||
"HEALTH_MEDICAL_RECORD": ["HIPAA"],
|
||||
"HEALTH_NPI": ["HIPAA"],
|
||||
"NET_DNS_EXFIL_HIGH_ENTROPY": [],
|
||||
...
|
||||
}
|
||||
```
|
||||
|
||||
Each rule maps to the compliance frameworks that regulate that data type. SSNs trigger four frameworks because they are considered protected health information (HIPAA), personal information (CCPA), financial identifiers (GLBA), and personal data (GDPR). Credit card PANs only trigger PCI-DSS and GLBA because HIPAA and GDPR do not specifically regulate financial card numbers.
|
||||
Rule IDs match actual detection rules rather than using generic categories. Credit card rules are split by brand (`FIN_CREDIT_CARD_VISA`, `FIN_CREDIT_CARD_MC`, `FIN_CREDIT_CARD_AMEX`, `FIN_CREDIT_CARD_DISC`), each triggering PCI-DSS and GLBA. State-specific driver's license rules (`PII_DRIVERS_LICENSE_FL`, `PII_DRIVERS_LICENSE_IL`) map to CCPA and HIPAA alongside the generic CA pattern. Network exfiltration indicators (`NET_DNS_EXFIL_*`, `NET_ENCODED_*`) carry empty framework lists since DNS tunneling is a detection concern, not a regulatory data type.
|
||||
|
||||
SSNs trigger four frameworks because they are considered protected health information (HIPAA), personal information (CCPA), financial identifiers (GLBA), and personal data (GDPR). Every rule also has a corresponding entry in `RULE_REMEDIATION_MAP` with specific guidance text. Unknown rules fall back to a generic default.
|
||||
|
||||
The mapping is intentionally conservative. An SSN could trigger SOX if it appears in financial reporting data, but without business context the scanner cannot determine that. The listed frameworks are the ones where the mere presence of the data type creates a compliance obligation.
|
||||
|
||||
## Shared Scoring Module
|
||||
|
||||
The `match_to_finding` function in `scoring.py` centralizes the conversion from `DetectorMatch` to `Finding`. All three scanners import from this single location instead of duplicating the severity/compliance/redaction logic:
|
||||
|
||||
```python
|
||||
def match_to_finding(
|
||||
match: DetectorMatch,
|
||||
text: str,
|
||||
location: Location,
|
||||
redaction_style: RedactionStyle,
|
||||
) -> Finding:
|
||||
severity = score_to_severity(match.score)
|
||||
frameworks = get_frameworks_for_rule(match.rule_id)
|
||||
if match.compliance_frameworks:
|
||||
combined = (
|
||||
set(frameworks) | set(match.compliance_frameworks)
|
||||
)
|
||||
frameworks = sorted(combined)
|
||||
remediation = get_remediation_for_rule(match.rule_id)
|
||||
|
||||
snippet = redact(
|
||||
text, match.start, match.end,
|
||||
style=redaction_style,
|
||||
)
|
||||
|
||||
return Finding(
|
||||
rule_id=match.rule_id,
|
||||
rule_name=match.rule_name,
|
||||
severity=severity,
|
||||
confidence=match.score,
|
||||
location=location,
|
||||
redacted_snippet=snippet,
|
||||
compliance_frameworks=frameworks,
|
||||
remediation=remediation,
|
||||
)
|
||||
```
|
||||
|
||||
The function chains severity classification, compliance framework lookup, remediation guidance, and redaction in one call. The framework merging logic handles the case where a detection rule carries its own `compliance_frameworks` list: those are merged with the frameworks from the compliance module, deduplicated, and sorted for deterministic output.
|
||||
|
||||
Each scanner calls this in its match loop:
|
||||
|
||||
```python
|
||||
for match in matches:
|
||||
if match.score < min_confidence:
|
||||
continue
|
||||
|
||||
finding = match_to_finding(
|
||||
match, chunk.text, chunk.location,
|
||||
self._redaction_style,
|
||||
)
|
||||
result.findings.append(finding)
|
||||
```
|
||||
|
||||
Adding a new compliance framework or changing severity thresholds affects all three scanners uniformly without touching scanner code.
|
||||
|
||||
## Redaction
|
||||
|
||||
The `redact` function in `redaction.py` builds a snippet with masked content:
|
||||
|
|
|
|||
Loading…
Reference in New Issue