MessyData
Resilient Multi-Source ETL Pipeline & Identity Resolution Engine
1. THE PROBLEM
Organizations collecting user data across multiple legacy databases end up with fragmented, duplicate, and corrupted records that skew business metrics.
2. TECHNICAL APPROACH & DECISIONS
1RapidFuzz String Distance Matching
Combined Jaro-Winkler string distance scoring with Token Sort ratio algorithms in RapidFuzz to cluster fuzzy duplicate candidate names and addresses.
Exact SQL LIKE matching
Exact SQL matching misses common human typos (e.g. 'Anirudh S' vs 'Anirud S'). RapidFuzz provided C++ accelerated fuzzy matching across thousands of rows.
3. TRADE-OFFS & HONEST REFLECTION
Fuzzy matching threshold tuning requires balancing false positives against missed duplicates. Set confidence cutoff to 88% to prioritize data safety.
4. CONCRETE OUTCOME & METRICS
Created a containerized ETL pipeline capable of deduplicating and cleaning multi-source customer datasets efficiently.