Record Linkage & Deduplication

This codebase implements record linkage and deduplication using Splink, a Python library for probabilistic record linkage. It leverages techniques such as blocking, comparison, and clustering.

Blocking Rules

Blocking rules in Splink are specified as SQL expressions, defining the subset of record comparisons to be generated for processing. For example, to compare records with matching first names, the blocking rule “l.first_name = r.first_name” can be used. (tutorials/03_Blocking.ipynb)

Comparison Settings

Splink’s comparison settings determine how records are compared, which can be configured through the comparisons parameter. The comparison_template_library (CTL) and comparison_library (CL) provide predefined comparison functions.

Examples include:

• ctl.name_comparison for comparing names, allowing adjustments for term frequency. (examples/sqlite/deduplicate_50k_synthetic.ipynb)
• cl.damerau_levenshtein_at_thresholds for comparing strings with Damerau-Levenshtein distance at specified thresholds. (examples/sqlite/deduplicate_50k_synthetic.ipynb)
• cl.exact_match for comparing exact matches. (examples/sqlite/deduplicate_50k_synthetic.ipynb)

Estimating Probabilities

The estimate_probability_two_random_records_match function calculates the probability of two random records matching based on the provided deterministic rules. The output shows the expected number of matching pairs. (examples/sqlite/deduplicate_50k_synthetic.ipynb)

Model Parameter Estimation

The em_convergence and max_iterations parameters are used to control the Expectation-Maximization (EM) algorithm, which estimates the model parameters. (examples/sqlite/deduplicate_50k_synthetic.ipynb)

Deterministic Linkage

Splink supports both probabilistic and deterministic record linkage. Deterministic linkage uses rules-based approaches for linking. The blocking_rules_to_generate_predictions parameter specifies rules to generate predictions. (examples/duckdb/deterministic_dedupe.ipynb)

Linking and Deduplication

The link_type setting controls the type of linkage operation:

• dedupe_only: For deduplication, identifying records referring to the same entity.
• linkage: For linking records across different datasets.

Example Notebooks

The repository contains various interactive notebooks demonstrating Splink functionalities, including:

• deduplicate_50k_synthetic.ipynb: This notebook showcases deduplication on a synthetic dataset.
• deterministic_dedupe.ipynb: This notebook illustrates deterministic deduplication using rules.
• deduplicate_1k_synthetic.ipynb: This notebook demonstrates deduplication on a synthetic dataset using Spark.
• deduplicate_50k_synthetic.ipynb: This notebook illustrates deduplication using Amazon Athena.

Additional Resources

The Splink documentation and project homepage provide comprehensive information about the library:

• Splink Documentation
• Splink Homepage

          ## Top-Level Directory Explanations
          
          <a class='local-link directory-link' data-ref="data/" href="#data/">data/</a> - This directory likely contains data used by the project. The specific contents of this directory may vary.
          
          <a class='local-link directory-link' data-ref="examples/" href="#examples/">examples/</a> - This directory likely contains examples or sample code for using the project's components.
          
          <a class='local-link directory-link' data-ref="examples/athena/" href="#examples/athena/">examples/athena/</a> - This subdirectory may contain examples using Amazon Athena, an interactive query service for analyzing data in Amazon S3 using standard SQL.
          
          <a class='local-link directory-link' data-ref="examples/athena/dashboards/" href="#examples/athena/dashboards/">examples/athena/dashboards/</a> - This subdirectory may contain Athena dashboard files.
          
          <a class='local-link directory-link' data-ref="examples/duckdb/" href="#examples/duckdb/">examples/duckdb/</a> - This subdirectory may contain examples using DuckDB, an open-source in-memory analytic database.
          
          <a class='local-link directory-link' data-ref="examples/duckdb/dashboards/" href="#examples/duckdb/dashboards/">examples/duckdb/dashboards/</a> - This subdirectory may contain DuckDB dashboard files.
          
          <a class='local-link directory-link' data-ref="examples/sqlite/" href="#examples/sqlite/">examples/sqlite/</a> - This subdirectory may contain examples using SQLite, a popular open-source database management system.
          
          <a class='local-link directory-link' data-ref="examples/sqlite/dashboards/" href="#examples/sqlite/dashboards/">examples/sqlite/dashboards/</a> - This subdirectory may contain SQLite dashboard files.
          
          <a class='local-link directory-link' data-ref="tutorials/" href="#tutorials/">tutorials/</a> - This directory may contain tutorials or guides for using the project.