metadata-scrubber-tool:
added comprehensive report coomand("veriiy") and tests,
added docs
This commit is contained in:
HERITAGE-XION
parent
a10d6b566b
commit
35c7988e67
145
README.md
145
README.md
|
|
@ -11,6 +11,7 @@ A privacy-focused CLI tool that removes sensitive metadata from files. Supports
|
|||
- **Multi-format support** - Images (JPEG, PNG), PDFs, and Office docs (Word, Excel, PowerPoint)
|
||||
- **Concurrent processing** - Process 1000+ files efficiently with ThreadPoolExecutor
|
||||
- **Dry-run mode** - Preview what would be scrubbed without making changes
|
||||
- **Verification reports** - Before/after comparison to confirm removal
|
||||
- **Smart format detection** - Uses library-level format detection, not just file extensions
|
||||
- **Beautiful CLI** - Rich progress bars and formatted output
|
||||
- **Privacy-first** - Removes GPS coordinates, author info, timestamps, camera data
|
||||
|
|
@ -50,39 +51,133 @@ mst scrub photo.jpg --output ./cleaned
|
|||
|
||||
# Batch process entire folder
|
||||
mst scrub ./documents -r -ext docx --output ./cleaned
|
||||
|
||||
# Verify removal
|
||||
mst verify original.jpg ./cleaned/processed_original.jpg
|
||||
```
|
||||
|
||||
## 📖 Commands
|
||||
|
||||
### `mst read` - View Metadata
|
||||
|
||||
Extract and display all embedded metadata from a file.
|
||||
|
||||
```bash
|
||||
mst read photo.jpg # Single file
|
||||
mst read report.pdf # PDF file
|
||||
mst read ./docs -r -ext docx # All Word docs recursively
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
```
|
||||
╭────────────────── Metadata Report ──────────────────╮
|
||||
│ ╭────────────────────┬────────────────────────────╮ │
|
||||
│ │ Property │ Value │ │
|
||||
│ ├────────────────────┼────────────────────────────┤ │
|
||||
│ │ 📷 Camera │ │ │
|
||||
│ │ Make │ Canon │ │
|
||||
│ │ Model │ Canon EOS 80D │ │
|
||||
│ │ Software │ Adobe Photoshop │ │
|
||||
│ ├────────────────────┼────────────────────────────┤ │
|
||||
│ │ 📍 GPS │ │ │
|
||||
│ │ GPSLatitude │ 40.7128 │ │
|
||||
│ │ GPSLongitude │ -74.0060 │ │
|
||||
│ ├────────────────────┼────────────────────────────┤ │
|
||||
│ │ 📅 Dates │ │ │
|
||||
│ │ DateTimeOriginal │ 2024:01:15 14:30:00 │ │
|
||||
│ │ created │ 2024-01-15 14:30:00 │ │
|
||||
│ ╰────────────────────┴────────────────────────────╯ │
|
||||
╰─────────────────────────────────────────────────────╯
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `mst scrub` - Remove Metadata
|
||||
|
||||
Remove sensitive metadata from files and save cleaned copies.
|
||||
|
||||
```bash
|
||||
mst scrub photo.jpg --output ./out # Single file
|
||||
mst scrub ./photos -r -ext jpg -o ./out # All JPEGs in directory
|
||||
mst scrub ./docs -r -ext pdf --dry-run # Preview PDF scrubbing
|
||||
mst scrub ./docs -r -ext pdf --dry-run # Preview without changes
|
||||
mst scrub ./files -r -ext xlsx -w 8 # 8 concurrent workers
|
||||
```
|
||||
|
||||
### CLI Options
|
||||
**Example output:**
|
||||
```
|
||||
Processing 42 files with 4 workers...
|
||||
|
||||
⠸ Scrubbing metadata... ━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 42/42 0:00:12
|
||||
|
||||
╭───────────────────── Summary ─────────────────────╮
|
||||
│ ✅ Processed: 42 │
|
||||
│ ❌ Failed: 0 │
|
||||
│ 📁 Output: C:\Users\...\cleaned │
|
||||
╰───────────────────────────────────────────────────╯
|
||||
```
|
||||
|
||||
**Dry-run example:**
|
||||
```bash
|
||||
mst scrub ./photos -r -ext jpg --dry-run
|
||||
```
|
||||
```
|
||||
🔍 DRY-RUN MODE - No files will be modified
|
||||
|
||||
Would process 15 files:
|
||||
• photo1.jpg → processed_photo1.jpg
|
||||
• photo2.jpg → processed_photo2.jpg
|
||||
• vacation/beach.jpg → processed_beach.jpg
|
||||
...
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `mst verify` - Verify Metadata Removal
|
||||
|
||||
Compare original and processed files to confirm sensitive data was removed.
|
||||
|
||||
```bash
|
||||
mst verify original.jpg ./out/processed_original.jpg
|
||||
```
|
||||
|
||||
**Example output:**
|
||||
```
|
||||
Comparing: test_canon.jpg → processed_test_canon.jpg
|
||||
|
||||
Verification Report
|
||||
┏━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┓
|
||||
┃ Property ┃ Before ┃ After ┃
|
||||
┡━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━┩
|
||||
│ Make │ Canon │ ✅ Removed │
|
||||
│ Model │ Canon EOS 80D │ ✅ Removed │
|
||||
│ Software │ Adobe Photoshop │ ✅ Removed │
|
||||
│ GPSLatitude │ 40.7128 │ ✅ Removed │
|
||||
│ GPSLongitude │ -74.0060 │ ✅ Removed │
|
||||
│ Artist │ John Smith │ ✅ Removed │
|
||||
│ Copyright │ © 2024 John Smith │ ✅ Removed │
|
||||
│ DateTimeOriginal │ 2024:01:15 14:30:00 │ ⚪ Preserved │
|
||||
└─────────────────────────┴──────────────────────────┴────────────────┘
|
||||
|
||||
✅ Status: CLEAN - All sensitive metadata removed
|
||||
Removed: 38 | Preserved: 2
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ CLI Options
|
||||
|
||||
| Option | Description |
|
||||
|--------|-------------|
|
||||
| `-r`, `--recursive` | Process directories recursively |
|
||||
| `-ext`, `--extension` | Filter by file extension |
|
||||
| `-ext`, `--extension` | Filter by file extension (jpg, png, pdf, docx, xlsx, pptx) |
|
||||
| `-o`, `--output` | Output directory for cleaned files |
|
||||
| `-d`, `--dry-run` | Preview without making changes |
|
||||
| `-w`, `--workers` | Number of concurrent workers |
|
||||
| `-w`, `--workers` | Number of concurrent workers (default: 4, max: 16) |
|
||||
| `-V`, `--verbose` | Show detailed debug logs |
|
||||
| `-v`, `--version` | Show version |
|
||||
|
||||
---
|
||||
|
||||
## 🛠️ Development
|
||||
|
||||
### Setup
|
||||
|
|
@ -108,32 +203,50 @@ mypy src
|
|||
|
||||
```
|
||||
src/
|
||||
├── main.py # CLI entry point (Typer app)
|
||||
├── main.py # CLI entry point (Typer app)
|
||||
├── commands/
|
||||
│ ├── read.py # Read metadata command
|
||||
│ └── scrub.py # Scrub metadata command
|
||||
│ ├── read.py # Read metadata command
|
||||
│ ├── scrub.py # Scrub metadata command
|
||||
│ └── verify.py # Verify removal command
|
||||
├── services/
|
||||
│ ├── metadata_factory.py # Factory for creating handlers
|
||||
│ ├── metadata_handler.py # Abstract base class
|
||||
│ ├── image_handler.py # JPEG/PNG handler
|
||||
│ ├── pdf_handler.py # PDF handler
|
||||
│ ├── excel_handler.py # Excel handler
|
||||
│ ├── metadata_factory.py # Factory for creating handlers
|
||||
│ ├── metadata_handler.py # Abstract base class
|
||||
│ ├── image_handler.py # JPEG/PNG handler
|
||||
│ ├── pdf_handler.py # PDF handler
|
||||
│ ├── excel_handler.py # Excel handler
|
||||
│ ├── powerpoint_handler.py # PowerPoint handler
|
||||
│ ├── worddoc_handler.py # Word document handler
|
||||
│ └── batch_processor.py # Concurrent batch processing
|
||||
│ ├── worddoc_handler.py # Word document handler
|
||||
│ ├── report_generator.py # Verification reports
|
||||
│ └── batch_processor.py # Concurrent batch processing
|
||||
└── core/
|
||||
├── jpeg_metadata.py # JPEG EXIF processor
|
||||
└── png_metadata.py # PNG metadata processor
|
||||
├── jpeg_metadata.py # JPEG EXIF processor
|
||||
└── png_metadata.py # PNG metadata processor
|
||||
|
||||
docs/
|
||||
├── metadata-risks.md # Privacy risks documentation
|
||||
└── best-practices.md # Secure file sharing guide
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
- **[Metadata Risks](docs/metadata-risks.md)** - Why metadata matters for privacy
|
||||
- **[Best Practices](docs/best-practices.md)** - Guidelines for secure file sharing
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ Security Considerations
|
||||
|
||||
- **Original files are never modified** - processed copies are created
|
||||
- **Use `--dry-run`** to preview changes before committing
|
||||
- **Use `mst verify`** to confirm sensitive data was removed
|
||||
- **GPS coordinates** are completely stripped for privacy
|
||||
- **Author information** is removed from all supported formats
|
||||
- **Always backup files** before scrubbing in production
|
||||
|
||||
---
|
||||
|
||||
## 📄 License
|
||||
|
||||
MIT License - See [LICENSE](LICENSE) for details.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,139 @@
|
|||
# Best Practices for Secure File Sharing
|
||||
|
||||
Guidelines for protecting your privacy when sharing files.
|
||||
|
||||
## Before Sharing Files
|
||||
|
||||
### 1. Always Check Metadata First
|
||||
|
||||
```bash
|
||||
mst read yourfile.pdf
|
||||
```
|
||||
|
||||
Review the output for any sensitive information.
|
||||
|
||||
### 2. Scrub Before Sharing
|
||||
|
||||
```bash
|
||||
mst scrub yourfile.pdf --output ./clean
|
||||
```
|
||||
|
||||
Always use the cleaned version for sharing.
|
||||
|
||||
### 3. Verify Removal
|
||||
|
||||
```bash
|
||||
mst verify yourfile.pdf ./clean/processed_yourfile.pdf
|
||||
```
|
||||
|
||||
Confirm sensitive data was actually removed.
|
||||
|
||||
---
|
||||
|
||||
## File Type Specific Guidance
|
||||
|
||||
### 📸 Images
|
||||
|
||||
**High Risk:** Photos from smartphones contain GPS coordinates by default.
|
||||
|
||||
```bash
|
||||
# Batch process all photos before uploading
|
||||
mst scrub ./photos -r -ext jpg --output ./clean
|
||||
```
|
||||
|
||||
**Tip:** Disable location services for your camera app to prevent GPS embedding.
|
||||
|
||||
### 📄 PDF Documents
|
||||
|
||||
**High Risk:** PDFs often contain author name, creator software, and sometimes revision history.
|
||||
|
||||
```bash
|
||||
mst scrub report.pdf --output ./clean
|
||||
```
|
||||
|
||||
**Note:** This tool removes document properties. For redacting visible content, use a dedicated PDF editor.
|
||||
|
||||
### 📊 Office Documents (Word, Excel, PowerPoint)
|
||||
|
||||
**High Risk:** Office files store author, company, and modification history.
|
||||
|
||||
```bash
|
||||
# Process all Word documents
|
||||
mst scrub ./documents -r -ext docx --output ./clean
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Workflow Integration
|
||||
|
||||
### For Regular File Sharing
|
||||
|
||||
1. Create a "clean" folder for processed files
|
||||
2. Before sharing, always scrub to that folder
|
||||
3. Share only from the clean folder
|
||||
|
||||
### For Automated Pipelines
|
||||
|
||||
```bash
|
||||
# In CI/CD or scripts
|
||||
mst scrub ./output -r -ext pdf --output ./publish
|
||||
```
|
||||
|
||||
### For Verification
|
||||
|
||||
```bash
|
||||
# Verify before publishing
|
||||
mst verify original.pdf ./publish/processed_original.pdf
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What's Preserved (And Why)
|
||||
|
||||
| Property | Reason |
|
||||
|----------|--------|
|
||||
| Created Date | File organization, audit trails |
|
||||
| Modified Date | Version tracking |
|
||||
| Language | Accessibility, screen readers |
|
||||
|
||||
These properties are generally not privacy-sensitive and are often needed for file management.
|
||||
|
||||
---
|
||||
|
||||
## Common Mistakes to Avoid
|
||||
|
||||
1. **Sharing the original instead of the processed file**
|
||||
- Always verify you're sharing from the output folder
|
||||
|
||||
2. **Not checking the output**
|
||||
- Use `mst verify` to confirm removal
|
||||
|
||||
3. **Forgetting about PDFs**
|
||||
- PDF author metadata is often overlooked
|
||||
|
||||
4. **Batch processing without verification**
|
||||
- Spot-check a few files after batch processing
|
||||
|
||||
5. **Assuming social media strips metadata**
|
||||
- Many platforms compress but don't fully remove metadata
|
||||
|
||||
---
|
||||
|
||||
## Quick Reference
|
||||
|
||||
```bash
|
||||
# Read metadata
|
||||
mst read file.jpg
|
||||
|
||||
# Scrub single file
|
||||
mst scrub file.jpg --output ./clean
|
||||
|
||||
# Scrub directory
|
||||
mst scrub ./folder -r -ext jpg --output ./clean
|
||||
|
||||
# Preview without changes
|
||||
mst scrub ./folder -r -ext jpg --dry-run
|
||||
|
||||
# Verify removal
|
||||
mst verify original.jpg ./clean/processed_original.jpg
|
||||
```
|
||||
|
|
@ -0,0 +1,103 @@
|
|||
# Metadata Privacy Risks
|
||||
|
||||
Understanding why metadata removal matters for privacy and security.
|
||||
|
||||
## What is Metadata?
|
||||
|
||||
Metadata is "data about data" - hidden information embedded in files that describes when, where, how, and by whom a file was created or modified.
|
||||
|
||||
## Common Metadata Types
|
||||
|
||||
### 📸 Images (JPEG, PNG)
|
||||
|
||||
| Metadata Type | Privacy Risk |
|
||||
|---------------|--------------|
|
||||
| **GPS Coordinates** | Reveals exact location where photo was taken |
|
||||
| **Camera Model** | Can identify specific device, linked to owner |
|
||||
| **Date/Time** | Reveals when photo was taken |
|
||||
| **Software** | Shows editing tools used |
|
||||
| **Thumbnail** | May contain original uncropped image |
|
||||
|
||||
### 📄 Documents (Word, PDF)
|
||||
|
||||
| Metadata Type | Privacy Risk |
|
||||
|---------------|--------------|
|
||||
| **Author** | Reveals creator's name/username |
|
||||
| **Company** | Exposes employer information |
|
||||
| **Last Modified By** | Shows who edited the document |
|
||||
| **Comments** | May contain internal notes |
|
||||
| **Revision History** | Can expose previous versions |
|
||||
|
||||
### 📊 Spreadsheets (Excel)
|
||||
|
||||
| Metadata Type | Privacy Risk |
|
||||
|---------------|--------------|
|
||||
| **Author** | Reveals creator identity |
|
||||
| **Company** | Exposes organization |
|
||||
| **Keywords** | May reveal document purpose |
|
||||
| **Custom Properties** | Internal tracking data |
|
||||
|
||||
### 📽️ Presentations (PowerPoint)
|
||||
|
||||
| Metadata Type | Privacy Risk |
|
||||
|---------------|--------------|
|
||||
| **Author** | Creator identity |
|
||||
| **Title/Subject** | May reveal confidential topics |
|
||||
| **Comments** | Presenter notes, internal feedback |
|
||||
|
||||
---
|
||||
|
||||
## Real-World Examples
|
||||
|
||||
### Case 1: Location Leak via Photo
|
||||
A journalist shared a photo online. The embedded GPS coordinates revealed their safe house location, putting sources at risk.
|
||||
|
||||
### Case 2: Internal Document Exposed
|
||||
A company released a redacted PDF. The author metadata still showed an employee's full name and the legal department.
|
||||
|
||||
### Case 3: Anonymous Submission Failed
|
||||
A whistleblower submitted documents "anonymously" but the Word metadata contained their username and computer name.
|
||||
|
||||
---
|
||||
|
||||
## High-Risk Scenarios
|
||||
|
||||
1. **Sharing photos on social media** - GPS can reveal home address
|
||||
2. **Sending documents to clients** - Author reveals internal usernames
|
||||
3. **Publishing reports externally** - Revision history may expose drafts
|
||||
4. **Submitting anonymous tips** - Metadata can identify the source
|
||||
5. **Legal discovery** - Metadata is often requested in litigation
|
||||
|
||||
---
|
||||
|
||||
## What This Tool Removes
|
||||
|
||||
| Category | Removed Properties |
|
||||
|----------|-------------------|
|
||||
| **Identity** | Author, Last Modified By, Creator |
|
||||
| **Location** | GPS Latitude, Longitude, Altitude |
|
||||
| **Device** | Camera Model, Software, Device ID |
|
||||
| **Tracking** | Comments, Keywords, Custom Properties |
|
||||
|
||||
### Preserved (Intentionally)
|
||||
|
||||
- **Created Date** - Often needed for file organization
|
||||
- **Modified Date** - Required for version tracking
|
||||
- **Language** - Accessibility purposes
|
||||
|
||||
---
|
||||
|
||||
## Best Practice
|
||||
|
||||
**Always scrub metadata before sharing files externally.**
|
||||
|
||||
```bash
|
||||
# Check what metadata exists
|
||||
mst read document.pdf
|
||||
|
||||
# Remove metadata
|
||||
mst scrub document.pdf --output ./clean
|
||||
|
||||
# Verify removal
|
||||
mst verify document.pdf ./clean/processed_document.pdf
|
||||
```
|
||||
|
|
@ -1,6 +1,6 @@
|
|||
[project]
|
||||
name = "metadata-scrubber"
|
||||
version = "0.2.0"
|
||||
version = "0.3.0"
|
||||
description = "A privacy-focused CLI tool that removes sensitive metadata from image files"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
|
|
@ -16,12 +16,7 @@ dependencies = [
|
|||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=9.0.0",
|
||||
"pytest-cov>=6.0.0",
|
||||
"mypy>=1.19.0",
|
||||
"ruff>=0.14.0",
|
||||
]
|
||||
dev = []
|
||||
|
||||
[project.scripts]
|
||||
mst = "src.main:app"
|
||||
|
|
@ -96,3 +91,12 @@ ignore_errors = true
|
|||
module = "src.services.worddoc_handler"
|
||||
ignore_errors = true
|
||||
|
||||
[dependency-groups]
|
||||
dev = [
|
||||
"mypy>=1.19.1",
|
||||
"pytest>=9.0.2",
|
||||
"pytest-cov>=7.0.0",
|
||||
"ruff>=0.14.11",
|
||||
"yapf>=0.43.0",
|
||||
]
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,81 @@
|
|||
"""
|
||||
Verify command for comparing metadata between original and processed files.
|
||||
|
||||
This command reads metadata from both files and generates a comparison report
|
||||
showing what was removed, preserved, or unchanged.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
import typer
|
||||
from rich.console import Console
|
||||
|
||||
from src.services.metadata_factory import MetadataFactory
|
||||
from src.services.report_generator import ReportGenerator
|
||||
|
||||
console = Console()
|
||||
log = logging.getLogger("metadata-scrubber")
|
||||
|
||||
|
||||
def verify(
|
||||
original_path: Path = typer.Argument(
|
||||
exists=True,
|
||||
file_okay=True,
|
||||
dir_okay=False,
|
||||
readable=True,
|
||||
resolve_path=True,
|
||||
help="Path to the original file (before scrubbing)",
|
||||
),
|
||||
processed_path: Path = typer.Argument(
|
||||
exists=True,
|
||||
file_okay=True,
|
||||
dir_okay=False,
|
||||
readable=True,
|
||||
resolve_path=True,
|
||||
help="Path to the processed file (after scrubbing)",
|
||||
),
|
||||
) -> None:
|
||||
"""
|
||||
Verify that metadata was properly removed from a scrubbed file.
|
||||
|
||||
Compares the original file with the processed version and shows
|
||||
a detailed report of what was removed, preserved, or still present.
|
||||
"""
|
||||
if log.isEnabledFor(logging.DEBUG):
|
||||
log.info(f"Original: {original_path}")
|
||||
log.info(f"Processed: {processed_path}")
|
||||
|
||||
try:
|
||||
# Read metadata from original file
|
||||
original_handler = MetadataFactory.get_handler(str(original_path))
|
||||
before_metadata = original_handler.read()
|
||||
|
||||
# Read metadata from processed file
|
||||
processed_handler = MetadataFactory.get_handler(str(processed_path))
|
||||
after_metadata = processed_handler.read()
|
||||
|
||||
# Generate comparison report
|
||||
generator = ReportGenerator()
|
||||
report = generator.generate_report(
|
||||
original_file=str(original_path),
|
||||
processed_file=str(processed_path),
|
||||
before_metadata=before_metadata,
|
||||
after_metadata=after_metadata,
|
||||
)
|
||||
|
||||
# Render the report
|
||||
console.print(
|
||||
f"[bold]Comparing:[/bold] {original_path.name} → {processed_path.name}"
|
||||
)
|
||||
console.print()
|
||||
generator.render_table(report)
|
||||
|
||||
# # Exit with appropriate code
|
||||
# if report.status == VerificationStatus.WARNING:
|
||||
# raise typer.Exit(code=1)
|
||||
|
||||
except Exception as e:
|
||||
console.print(f"[red]Error during verification:[/red] {e}")
|
||||
if log.isEnabledFor(logging.DEBUG):
|
||||
console.print_exception()
|
||||
|
|
@ -7,6 +7,7 @@ It initializes the Typer app, registers commands, and configures logging.
|
|||
Commands:
|
||||
read: Display metadata from files.
|
||||
scrub: Remove metadata from files.
|
||||
verify: Compare original and processed files to verify metadata removal.
|
||||
"""
|
||||
|
||||
import logging
|
||||
|
|
@ -15,6 +16,7 @@ import typer
|
|||
|
||||
from src.commands.read import read
|
||||
from src.commands.scrub import scrub
|
||||
from src.commands.verify import verify
|
||||
from src.utils.logger import setup_logging
|
||||
|
||||
# Initialize the Typer app with helpful defaults
|
||||
|
|
@ -22,7 +24,7 @@ app = typer.Typer(no_args_is_help=True, pretty_exceptions_show_locals=False)
|
|||
log = logging.getLogger("metadata-scrubber")
|
||||
|
||||
|
||||
__version__ = "0.2.0"
|
||||
__version__ = "0.3.0"
|
||||
|
||||
|
||||
# ---------------------------------------------------------
|
||||
|
|
@ -59,7 +61,7 @@ def main(
|
|||
),
|
||||
):
|
||||
"""
|
||||
Metadata Scrubber Tool - Clean your images personal identifying data. eg: author name, camera model, GPS coordinates, etc.
|
||||
Metadata Scrubber Tool - Clean your files' personal identifying data. eg: author name, camera model, GPS coordinates, etc.
|
||||
"""
|
||||
# Initialize the logger based on the user's flag
|
||||
setup_logging(verbose)
|
||||
|
|
@ -71,6 +73,7 @@ def main(
|
|||
# register commands
|
||||
app.command(name="read")(read)
|
||||
app.command(name="scrub")(scrub)
|
||||
app.command(name="verify")(verify)
|
||||
|
||||
# run app
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -0,0 +1,304 @@
|
|||
"""
|
||||
Report generator for metadata comparison and verification.
|
||||
|
||||
This module provides the ReportGenerator class which compares before/after
|
||||
metadata states and generates rich table output showing what was removed,
|
||||
preserved, or unchanged.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from enum import Enum
|
||||
from typing import Any
|
||||
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
|
||||
class PropertyStatus(Enum):
|
||||
"""Status of a metadata property after scrubbing."""
|
||||
|
||||
REMOVED = "removed" # Was present, now None/empty
|
||||
PRESERVED = "preserved" # Intentionally kept (created, modified, etc.)
|
||||
UNCHANGED = "unchanged" # Had no value before and after
|
||||
WARNING = "warning" # Should have been removed but wasn't
|
||||
|
||||
|
||||
class VerificationStatus(Enum):
|
||||
"""Overall verification status of a scrubbed file."""
|
||||
|
||||
CLEAN = "clean" # All sensitive metadata removed
|
||||
WARNING = "warning" # Some metadata may still be present
|
||||
ERROR = "error" # Verification failed
|
||||
|
||||
|
||||
@dataclass
|
||||
class PropertyComparison:
|
||||
"""Comparison result for a single metadata property."""
|
||||
|
||||
name: str
|
||||
before_value: Any
|
||||
after_value: Any
|
||||
status: PropertyStatus
|
||||
is_sensitive: bool = True # Whether this property is considered sensitive
|
||||
|
||||
|
||||
@dataclass
|
||||
class ComparisonReport:
|
||||
"""Complete comparison report between original and processed files."""
|
||||
|
||||
original_file: str
|
||||
processed_file: str
|
||||
comparisons: list[PropertyComparison] = field(default_factory=list)
|
||||
status: VerificationStatus = VerificationStatus.CLEAN
|
||||
removed_count: int = 0
|
||||
preserved_count: int = 0
|
||||
warning_count: int = 0
|
||||
|
||||
|
||||
class ReportGenerator:
|
||||
"""
|
||||
Generates before/after metadata comparison reports.
|
||||
|
||||
Compares metadata from original and processed files to verify
|
||||
that sensitive information has been properly removed.
|
||||
"""
|
||||
|
||||
# Properties that are intentionally preserved
|
||||
PRESERVED_PROPERTIES = {
|
||||
"created",
|
||||
"modified",
|
||||
"language",
|
||||
"last_printed",
|
||||
"revision",
|
||||
"JPEGInterchangeFormat",
|
||||
"JPEGInterchangeFormatLength",
|
||||
}
|
||||
|
||||
def __init__(self):
|
||||
"""Initialize the report generator."""
|
||||
self.console = Console()
|
||||
|
||||
def compare(
|
||||
self,
|
||||
before: dict[str, Any],
|
||||
after: dict[str, Any],
|
||||
preserved_keys: set[str] | None = None,
|
||||
) -> ComparisonReport:
|
||||
"""
|
||||
Compare before and after metadata states.
|
||||
|
||||
Args:
|
||||
before: Metadata dictionary from original file.
|
||||
after: Metadata dictionary from processed file.
|
||||
preserved_keys: Set of property names that should be preserved.
|
||||
|
||||
Returns:
|
||||
ComparisonReport with detailed comparison results.
|
||||
"""
|
||||
if preserved_keys is None:
|
||||
preserved_keys = self.PRESERVED_PROPERTIES
|
||||
|
||||
report = ComparisonReport(
|
||||
original_file="",
|
||||
processed_file="",
|
||||
)
|
||||
|
||||
# Get all unique keys from both dictionaries
|
||||
all_keys = set(before.keys()) | set(after.keys())
|
||||
|
||||
for key in sorted(all_keys):
|
||||
before_value = before.get(key)
|
||||
after_value = after.get(key)
|
||||
|
||||
# Determine property status
|
||||
status = self._determine_status(
|
||||
key, before_value, after_value, preserved_keys
|
||||
)
|
||||
|
||||
is_sensitive = key not in preserved_keys
|
||||
|
||||
comparison = PropertyComparison(
|
||||
name=key,
|
||||
before_value=before_value,
|
||||
after_value=after_value,
|
||||
status=status,
|
||||
is_sensitive=is_sensitive,
|
||||
)
|
||||
report.comparisons.append(comparison)
|
||||
|
||||
# Update counts
|
||||
if status == PropertyStatus.REMOVED:
|
||||
report.removed_count += 1
|
||||
elif status == PropertyStatus.PRESERVED:
|
||||
report.preserved_count += 1
|
||||
elif status == PropertyStatus.WARNING:
|
||||
report.warning_count += 1
|
||||
|
||||
# Determine overall status
|
||||
if report.warning_count > 0:
|
||||
report.status = VerificationStatus.WARNING
|
||||
else:
|
||||
report.status = VerificationStatus.CLEAN
|
||||
|
||||
return report
|
||||
|
||||
def _determine_status(
|
||||
self,
|
||||
key: str,
|
||||
before_value: Any,
|
||||
after_value: Any,
|
||||
preserved_keys: set[str],
|
||||
) -> PropertyStatus:
|
||||
"""
|
||||
Determine the status of a property based on before/after values.
|
||||
|
||||
Args:
|
||||
key: Property name.
|
||||
before_value: Value before processing.
|
||||
after_value: Value after processing.
|
||||
preserved_keys: Set of keys that should be preserved.
|
||||
|
||||
Returns:
|
||||
PropertyStatus indicating what happened to this property.
|
||||
"""
|
||||
# Check if this is a preserved property
|
||||
if key in preserved_keys:
|
||||
return PropertyStatus.PRESERVED
|
||||
|
||||
# Check if value was removed
|
||||
before_has_value = self._has_value(before_value)
|
||||
after_has_value = self._has_value(after_value)
|
||||
|
||||
if before_has_value and not after_has_value:
|
||||
return PropertyStatus.REMOVED
|
||||
elif not before_has_value and not after_has_value:
|
||||
return PropertyStatus.UNCHANGED
|
||||
elif before_has_value and after_has_value:
|
||||
# Sensitive property still has value - warning
|
||||
return PropertyStatus.WARNING
|
||||
else:
|
||||
return PropertyStatus.UNCHANGED
|
||||
|
||||
def _has_value(self, value: Any) -> bool:
|
||||
"""
|
||||
Check if a value is considered "present" (not None/empty).
|
||||
|
||||
Args:
|
||||
value: Value to check.
|
||||
|
||||
Returns:
|
||||
True if value is present and meaningful.
|
||||
"""
|
||||
if value is None:
|
||||
return False
|
||||
if isinstance(value, str) and value.strip() in ("", "-"):
|
||||
return False
|
||||
if value == 0 and not isinstance(value, bool):
|
||||
# 0 is typically not meaningful for metadata
|
||||
return False
|
||||
return True
|
||||
|
||||
def render_table(self, report: ComparisonReport) -> None:
|
||||
"""
|
||||
Render a comparison report as a rich table.
|
||||
|
||||
Args:
|
||||
report: ComparisonReport to render.
|
||||
"""
|
||||
# Create status emoji mapping
|
||||
status_icons = {
|
||||
PropertyStatus.REMOVED: "[green]✅ Removed[/green]",
|
||||
PropertyStatus.PRESERVED: "[dim]⚪ Preserved[/dim]",
|
||||
PropertyStatus.UNCHANGED: "[dim]— No change[/dim]",
|
||||
PropertyStatus.WARNING: "[yellow]⚠️ Still present[/yellow]",
|
||||
}
|
||||
|
||||
table = Table(
|
||||
title="[bold]Verification Report[/bold]",
|
||||
show_header=True,
|
||||
header_style="bold",
|
||||
)
|
||||
|
||||
table.add_column("Property", style="cyan")
|
||||
table.add_column("Before", style="dim")
|
||||
table.add_column("After", justify="left")
|
||||
|
||||
for comp in report.comparisons:
|
||||
before_str = self._format_value(comp.before_value)
|
||||
after_str = status_icons.get(comp.status, str(comp.after_value))
|
||||
|
||||
table.add_row(comp.name, before_str, after_str)
|
||||
|
||||
self.console.print(table)
|
||||
|
||||
# Print summary
|
||||
self._print_summary(report)
|
||||
|
||||
def _format_value(self, value: Any) -> str:
|
||||
"""
|
||||
Format a metadata value for display.
|
||||
|
||||
Args:
|
||||
value: Value to format.
|
||||
|
||||
Returns:
|
||||
Formatted string representation.
|
||||
"""
|
||||
if value is None:
|
||||
return "[dim]None[/dim]"
|
||||
if isinstance(value, str):
|
||||
if len(value) > 30:
|
||||
return value[:27] + "..."
|
||||
return value if value.strip() else "[dim]-[/dim]"
|
||||
return str(value)
|
||||
|
||||
def _print_summary(self, report: ComparisonReport) -> None:
|
||||
"""
|
||||
Print summary statistics for the report.
|
||||
|
||||
Args:
|
||||
report: ComparisonReport to summarize.
|
||||
"""
|
||||
self.console.print()
|
||||
|
||||
if report.status == VerificationStatus.CLEAN:
|
||||
self.console.print(
|
||||
"[bold green]✅ Status: CLEAN[/bold green] - "
|
||||
"All sensitive metadata removed"
|
||||
)
|
||||
elif report.status == VerificationStatus.WARNING:
|
||||
self.console.print(
|
||||
f"[bold yellow]⚠️ Status: WARNING[/bold yellow] - "
|
||||
f"{report.warning_count} properties may still contain data"
|
||||
)
|
||||
else:
|
||||
self.console.print("[bold red]❌ Status: ERROR[/bold red]")
|
||||
|
||||
self.console.print(
|
||||
f"[dim]Removed: {report.removed_count} | "
|
||||
f"Preserved: {report.preserved_count}[/dim]"
|
||||
)
|
||||
|
||||
def generate_report(
|
||||
self,
|
||||
original_file: str,
|
||||
processed_file: str,
|
||||
before_metadata: dict[str, Any],
|
||||
after_metadata: dict[str, Any],
|
||||
) -> ComparisonReport:
|
||||
"""
|
||||
Generate a full comparison report between two files.
|
||||
|
||||
Args:
|
||||
original_file: Path to original file.
|
||||
processed_file: Path to processed file.
|
||||
before_metadata: Metadata from original file.
|
||||
after_metadata: Metadata from processed file.
|
||||
|
||||
Returns:
|
||||
ComparisonReport with file paths and comparison data.
|
||||
"""
|
||||
report = self.compare(before_metadata, after_metadata)
|
||||
report.original_file = original_file
|
||||
report.processed_file = processed_file
|
||||
return report
|
||||
|
|
@ -0,0 +1,258 @@
|
|||
"""
|
||||
End-to-end tests for the verify command.
|
||||
|
||||
Tests the complete CLI workflow for the verify command, including
|
||||
comparing original and processed files and verifying output format.
|
||||
"""
|
||||
|
||||
import shutil
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from typer.testing import CliRunner
|
||||
|
||||
from src.main import app
|
||||
|
||||
runner = CliRunner()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def output_dir(tmp_path):
|
||||
"""Create a temporary output directory for processed files."""
|
||||
output = tmp_path / "output"
|
||||
output.mkdir()
|
||||
yield output
|
||||
if output.exists():
|
||||
shutil.rmtree(output)
|
||||
|
||||
|
||||
# ============== Image Tests ==============
|
||||
|
||||
|
||||
def test_verify_command_jpeg_clean_status(output_dir):
|
||||
"""Test verify command shows CLEAN status for properly scrubbed JPEG."""
|
||||
from tests.conftest import get_jpg_test_file
|
||||
|
||||
JPG_TEST_FILE = get_jpg_test_file()
|
||||
|
||||
# First scrub the file
|
||||
scrub_result = runner.invoke(
|
||||
app, ["scrub", JPG_TEST_FILE, "--output", str(output_dir)]
|
||||
)
|
||||
assert scrub_result.exit_code == 0
|
||||
|
||||
# Get the processed file path
|
||||
processed_file = output_dir / f"processed_{Path(JPG_TEST_FILE).name}"
|
||||
assert processed_file.exists()
|
||||
|
||||
# Now verify
|
||||
result = runner.invoke(app, ["verify", JPG_TEST_FILE, str(processed_file)])
|
||||
|
||||
assert result.exit_code == 0, f"Failed with: {result.stdout}"
|
||||
assert "Comparing" in result.stdout
|
||||
assert "CLEAN" in result.stdout or "Removed" in result.stdout
|
||||
|
||||
|
||||
# this test can be ran if a png with metadata is found and added to the assests folder.
|
||||
# def test_verify_command_png_clean_status(output_dir):
|
||||
# """Test verify command shows CLEAN status for properly scrubbed PNG."""
|
||||
# from tests.conftest import get_png_test_file
|
||||
|
||||
# # PNG_TEST_FILE = get_png_test_file()
|
||||
|
||||
# # First scrub the file
|
||||
# scrub_result = runner.invoke(
|
||||
# app, ["scrub", PNG_TEST_FILE, "--output", str(output_dir)]
|
||||
# )
|
||||
|
||||
# # Get the processed file path
|
||||
# processed_file = output_dir / f"processed_{Path(PNG_TEST_FILE).name}"
|
||||
|
||||
# # Skip if scrub failed (PNG might have no metadata)
|
||||
# if scrub_result.exit_code != 0 or not processed_file.exists():
|
||||
# pytest.skip("PNG file had no metadata to scrub")
|
||||
|
||||
# # Now verify
|
||||
# result = runner.invoke(app, ["verify", PNG_TEST_FILE, str(processed_file)])
|
||||
|
||||
# assert result.exit_code == 0, f"Failed with: {result.stdout}"
|
||||
# assert "Comparing" in result.stdout
|
||||
|
||||
|
||||
# ============== PDF Tests ==============
|
||||
|
||||
|
||||
def test_verify_command_pdf_clean_status(output_dir):
|
||||
"""Test verify command shows CLEAN status for properly scrubbed PDF."""
|
||||
from tests.conftest import get_pdf_test_file
|
||||
|
||||
PDF_TEST_FILE = get_pdf_test_file()
|
||||
|
||||
# First scrub the file
|
||||
scrub_result = runner.invoke(
|
||||
app, ["scrub", PDF_TEST_FILE, "--output", str(output_dir)]
|
||||
)
|
||||
assert scrub_result.exit_code == 0
|
||||
|
||||
# Get the processed file path
|
||||
processed_file = output_dir / f"processed_{Path(PDF_TEST_FILE).name}"
|
||||
assert processed_file.exists()
|
||||
|
||||
# Now verify
|
||||
result = runner.invoke(app, ["verify", PDF_TEST_FILE, str(processed_file)])
|
||||
|
||||
assert result.exit_code == 0, f"Failed with: {result.stdout}"
|
||||
assert "Comparing" in result.stdout
|
||||
|
||||
|
||||
# ============== Excel Tests ==============
|
||||
|
||||
|
||||
def test_verify_command_xlsx_clean_status(output_dir):
|
||||
"""Test verify command shows CLEAN status for properly scrubbed Excel."""
|
||||
from tests.conftest import get_xlsx_test_file
|
||||
|
||||
XLSX_TEST_FILE = get_xlsx_test_file()
|
||||
|
||||
# First scrub the file
|
||||
scrub_result = runner.invoke(
|
||||
app, ["scrub", XLSX_TEST_FILE, "--output", str(output_dir)]
|
||||
)
|
||||
assert scrub_result.exit_code == 0
|
||||
|
||||
# Get the processed file path
|
||||
processed_file = output_dir / f"processed_{Path(XLSX_TEST_FILE).name}"
|
||||
assert processed_file.exists()
|
||||
|
||||
# Now verify
|
||||
result = runner.invoke(app, ["verify", XLSX_TEST_FILE, str(processed_file)])
|
||||
|
||||
assert result.exit_code == 0, f"Failed with: {result.stdout}"
|
||||
assert "Comparing" in result.stdout
|
||||
|
||||
|
||||
# ============== PowerPoint Tests ==============
|
||||
|
||||
|
||||
def test_verify_command_pptx_clean_status(output_dir):
|
||||
"""Test verify command shows CLEAN status for properly scrubbed PowerPoint."""
|
||||
from tests.conftest import get_pptx_test_file
|
||||
|
||||
PPTX_TEST_FILE = get_pptx_test_file()
|
||||
|
||||
# First scrub the file
|
||||
scrub_result = runner.invoke(
|
||||
app, ["scrub", PPTX_TEST_FILE, "--output", str(output_dir)]
|
||||
)
|
||||
assert scrub_result.exit_code == 0
|
||||
|
||||
# Get the processed file path
|
||||
processed_file = output_dir / f"processed_{Path(PPTX_TEST_FILE).name}"
|
||||
assert processed_file.exists()
|
||||
|
||||
# Now verify
|
||||
result = runner.invoke(app, ["verify", PPTX_TEST_FILE, str(processed_file)])
|
||||
|
||||
assert result.exit_code == 0, f"Failed with: {result.stdout}"
|
||||
assert "Comparing" in result.stdout
|
||||
|
||||
|
||||
# ============== Word Document Tests ==============
|
||||
|
||||
|
||||
def test_verify_command_docx_clean_status(output_dir):
|
||||
"""Test verify command shows CLEAN status for properly scrubbed Word doc."""
|
||||
from tests.conftest import get_docx_test_file
|
||||
|
||||
DOCX_TEST_FILE = get_docx_test_file()
|
||||
|
||||
# First scrub the file
|
||||
scrub_result = runner.invoke(
|
||||
app, ["scrub", DOCX_TEST_FILE, "--output", str(output_dir)]
|
||||
)
|
||||
assert scrub_result.exit_code == 0
|
||||
|
||||
# Get the processed file path
|
||||
processed_file = output_dir / f"processed_{Path(DOCX_TEST_FILE).name}"
|
||||
assert processed_file.exists()
|
||||
|
||||
# Now verify
|
||||
result = runner.invoke(app, ["verify", DOCX_TEST_FILE, str(processed_file)])
|
||||
|
||||
assert result.exit_code == 0, f"Failed with: {result.stdout}"
|
||||
assert "Comparing" in result.stdout
|
||||
|
||||
|
||||
# ============== Error Tests ==============
|
||||
|
||||
|
||||
def test_verify_command_original_file_not_found():
|
||||
"""Test verify command handles missing original file."""
|
||||
result = runner.invoke(
|
||||
app, ["verify", "nonexistent_file.jpg", "also_nonexistent.jpg"]
|
||||
)
|
||||
|
||||
assert result.exit_code != 0
|
||||
|
||||
|
||||
def test_verify_command_processed_file_not_found():
|
||||
"""Test verify command handles missing processed file."""
|
||||
from tests.conftest import get_jpg_test_file
|
||||
|
||||
JPG_TEST_FILE = get_jpg_test_file()
|
||||
|
||||
result = runner.invoke(app, ["verify", JPG_TEST_FILE, "nonexistent_processed.jpg"])
|
||||
|
||||
assert result.exit_code != 0
|
||||
|
||||
|
||||
# ============== Output Format Tests ==============
|
||||
|
||||
|
||||
def test_verify_command_shows_removed_count(output_dir):
|
||||
"""Test that verify command shows removed count in output."""
|
||||
from tests.conftest import get_jpg_test_file
|
||||
|
||||
JPG_TEST_FILE = get_jpg_test_file()
|
||||
|
||||
# Scrub the file
|
||||
runner.invoke(app, ["scrub", JPG_TEST_FILE, "--output", str(output_dir)])
|
||||
processed_file = output_dir / f"processed_{Path(JPG_TEST_FILE).name}"
|
||||
|
||||
# Verify
|
||||
result = runner.invoke(app, ["verify", JPG_TEST_FILE, str(processed_file)])
|
||||
|
||||
assert "Removed:" in result.stdout
|
||||
|
||||
|
||||
def test_verify_command_shows_preserved_count(output_dir):
|
||||
"""Test that verify command shows preserved count in output."""
|
||||
from tests.conftest import get_jpg_test_file
|
||||
|
||||
JPG_TEST_FILE = get_jpg_test_file()
|
||||
|
||||
# Scrub the file
|
||||
runner.invoke(app, ["scrub", JPG_TEST_FILE, "--output", str(output_dir)])
|
||||
processed_file = output_dir / f"processed_{Path(JPG_TEST_FILE).name}"
|
||||
|
||||
# Verify
|
||||
result = runner.invoke(app, ["verify", JPG_TEST_FILE, str(processed_file)])
|
||||
|
||||
assert "Preserved:" in result.stdout
|
||||
|
||||
|
||||
def test_verify_command_shows_comparison_header(output_dir):
|
||||
"""Test that verify command shows comparison header."""
|
||||
from tests.conftest import get_jpg_test_file
|
||||
|
||||
JPG_TEST_FILE = get_jpg_test_file()
|
||||
|
||||
# Scrub the file
|
||||
runner.invoke(app, ["scrub", JPG_TEST_FILE, "--output", str(output_dir)])
|
||||
processed_file = output_dir / f"processed_{Path(JPG_TEST_FILE).name}"
|
||||
|
||||
# Verify
|
||||
result = runner.invoke(app, ["verify", JPG_TEST_FILE, str(processed_file)])
|
||||
|
||||
assert "Comparing:" in result.stdout
|
||||
assert Path(JPG_TEST_FILE).name in result.stdout
|
||||
Loading…
Reference in New Issue