moj-analytical-services / splink_demos

This documentation is focused on the configuration options available for the development environment of the moj-analytical-services/splink_demos. It aims to provide detailed and step-by-step instructions and code examples to assist expert developers in setting up and customizing their development environment effectively.

1. Kernel Specification

When working in Jupyter notebooks, it’s crucial to configure the kernel correctly. The following code block specifies the kernel settings for the notebooks:

{
  "kernelspec": {
    "display_name": "splink_demos",
    "language": "python",
    "name": "splink_demos"
  },
  "language_info": {
    "codemirror_mode": {
      "name": "ipython",
      "version": 3
    },
    "name": "python",
    "version": "3.10"
  }
}

Note: Ensure that the correct version of Python is installed to match the configurations in your environment.

2. Data Cleaning and Standardization

Proper configuration of data cleaning is essential for optimal performance in Splink. Below are some recommended practices for standardizing your dataset:

  • Trim Whitespace: Remove leading and trailing whitespace on string values.

    df['name'] = df['name'].str.strip()

  • Remove Special Characters: Clean the data by removing special characters:

    df['name'] = df['name'].str.replace(r"[^a-zA-Z0-9 ]", "", regex=True)

  • Standardize Date Formats: Format dates consistently across the dataset.

    df['date_of_birth'] = pd.to_datetime(df['date_of_birth']).dt.strftime('%Y-%m-%d')

  • Uniform Abbreviations: Replace abbreviations with full words for accuracy.

    df['address'] = df['address'].str.replace("St.", "Street")

3. Defining Comparisons

When configuring the linkage model, defining comparisons is vital. Below is an example of how to encompass various comparisons in your model structure:

from splink import Comparison

# Defining comparisons
comparison_list = [
    Comparison("date_of_birth", ["Exact Match", "One Character Difference"]),
    Comparison("name", ["Exact Match", "Fuzzy Match"]),
    Comparison("location", ["Exact Match", "Substring Match"])
]

# Nest comparisons into the model
linkage_model = {
    "Model": "Data Linking Model",
    "Comparisons": comparison_list
}

Each Comparison can have multiple ComparisonLevels that signify how similarity is evaluated for various fields.

4. Estimating Model Parameters

To accurately gauge model performance, it is essential to estimate various parameters like probability_two_random_records_match, u, and m. This can be accomplished using the following code snippet:

# Estimating parameters
probability_match = linker_simple.estimate_u_using_random_sampling(max_pairs=1e7)

# Estimating m and u parameters
session = linker_simple.estimate_parameters_using_expectation_maximisation(
    "l.given_name = r.given_name"
)

5. Visualization of Comparisons

Once the comparisons are established, visualizing them helps in understanding the linkage model better. For visualization of comparison parameters, the following code can be utilized:

linker_simple.parameter_estimate_comparisons_chart()

This will generate a chart that illustrates the estimated parameters for the defined comparisons.

Conclusion

Configuration of the development environment is a critical step to ensure smooth operation of the moj-analytical-services/splink_demos. The information above provides a comprehensive guide on kernel specifications, data cleaning, defining comparisons, estimating model parameters, and visualizations. When executed correctly, these configurations will enhance the performance and accuracy of your data linkage processes.

Sources: