Patient data from FHIR R4, legacy Oracle EHR, lab SFTP drops, and EDI 837/835 reconciled into a master patient index with probabilistic matching, Safe-Harbor masking, and HIPAA audit.
Patient identifiers fragment across hospital systems — the same person exists multiple times under slightly different names, dates of birth or addresses. Analytical and operational workloads require a single identity reconciled across sources, with PHI handled correctly under HIPAA. This is the consolidation pattern: fuzzy identity matching, code validation, masking for analytics paths, claim and remittance reconciliation, all glued to an immutable audit log.
FHIR R4 server (OAuth client-credentials), legacy Oracle EHR (CDC), lab CSVs landed via SFTP, and EDI 837/835 claim files. A common ingest schema collapses the source-format differences before any processing.
Weighted Jaro-Winkler (via rapidfuzz) across name (0.35), identifier (0.30), DOB (0.25) and address (0.10). Scores ≥ 0.85 auto-merge into the MPI; 0.65–0.85 route to a steward queue; below 0.65 stay separate.
18 PHI identifiers masked before anything lands in the Elasticsearch analytics store or Delta lake. Salted SHA-256 hashes preserve join-by-identity across masked stores without ever exposing the raw value.
A lightweight X12 segment parser handles the CLM/CLP segments that account for 90% of reconciliation value. Production deployments swap in pyx12 for full segment coverage at the same interface.
Every read of unmasked PHI writes an audit row (who, what, when, why). The audit table is recommended for S3 Object Lock / ADLS immutability — operators cannot delete history even with DB credentials.
Jaro-Winkler handles transpositions and prefixes (nicknames, married names, initials) far better than plain Levenshtein. Materially better recall on the hard cases without sacrificing precision.
Salted SHA-256 of the identifier lets you join across masked stores without ever holding raw PHI in the analytics layer. The salt is environment-scoped — losing the analytics store doesn't compromise patient IDs.
Joins are by exact hash equality; fuzzy matches still need to happen upstream of masking. Acceptable because matching is the prior step.
Auto-merging in the grey zone causes silent data quality bugs. A human queue at the right threshold catches edge cases and feeds back into matching weight tuning.
Steward backlog grows with patient volume. Mitigated by tuning weights down toward 0.65 over time as the matching improves.
Full X12 837/835 is enormous; CLM (claim) and CLP (payment) cover the reconciliation use case. Tight scope first, escape hatch (pyx12) second.