dedup

Find and remove duplicate files between two directories using SHA-256 hashing with parallel workers.

Requirements

• Python 3.8+
• No external dependencies (standard library only)

Usage

# Compare two directories (read-only, generates a report)
python3 dedup.py /path/to/backup /path/to/original

# Move duplicates to a __duplicates__/ quarantine folder
python3 dedup.py --move /path/to/backup /path/to/original

# Permanently delete duplicates
python3 dedup.py --delete /path/to/backup /path/to/original

# Control parallelism and checkpointing
python3 dedup.py --workers 8 --checkpoint-every 500 /large/backup /original

# Restrict to specific extensions
python3 dedup.py --ext .jpg,.png,.cr2 /path/to/backup /original

# Top-level files only (no recursion)
python3 dedup.py --no-subdirs /path/to/backup /original

# Remove checkpoint data and start fresh
python3 dedup.py --clean /path/to/backup /original

Options

Flag	Description
--move	Move duplicates to __duplicates__/ folder (safe, reversible)
--delete	Permanently delete duplicates (irreversible)
--no-subdirs	Only compare top-level files
--ext .jpg,.png	Restrict to specific extensions (default: all media)
--report PATH	Write report to PATH (default: dirA/duplicates_report.txt)
--workers N	Number of parallel hash workers (default: CPU count)
--checkpoint-every N	Files between checkpoints, 0 to disable (default: 1000)
--clean	Remove checkpoint data and exit
--version	Show version

How It Works

1. Phase 1 — Hash all files in the reference directory (dir B)
2. Phase 2 — Hash all files in the source directory (dir A)
3. Phase 3 — Compare hashes and generate a report

Files are classified as:

• Exact matches — same filename and same content hash
• Hash matches — same content, different filename
• Same name, different content — same filename, different content (not duplicates)
• Unique to dirA — no matching file in dirB

Checkpointing

For large directories, progress is saved every --checkpoint-every files (default: 1000) to dirA/.dedup_checkpoint/. If the process is interrupted, re-run the same command to resume from where it left off.

Checkpoints are automatically cleaned after a successful run. Use --clean to manually discard them.

Output

A text report is written with sections for each match category, showing file paths and SHA-256 hashes.

License

MIT