moj-analytical-services / splink_demos

Introduction

This section provides an in-depth look at how the moj-analytical-services/splink_demos project is monitored in a production environment, specifically focusing on quality assurance of prediction results as outlined in tutorials/07_Quality_assurance.ipynb.

1. Understanding Quality Assurance Metrics

In Splink, quality assurance is vital for ensuring that the predictions being made are accurate and reliable. Key metrics you will monitor in production include True Positives (TP), True Negatives (TN), False Positives (FP), False Negatives (FN), Precision, Recall, and the F1 score. These are defined quantitatively and can be represented in your monitoring dashboards.

1.1 Metrics Definitions

  • TP (True Positives): The number of correct positive predictions.
  • TN (True Negatives): The number of correct negative predictions.
  • FP (False Positives): The number of incorrect positive predictions.
  • FN (False Negatives): The number of incorrect negative predictions.
  • Precision: The ratio of correctly predicted positive observations to the total predicted positives. Formula: [ Precision = \frac{TP}{TP + FP} ]
  • Recall: The ratio of correctly predicted positive observations to the all observations in actual class. Formula: [ Recall = \frac{TP}{TP + FN} ]
  • F1 Score: The weighted average of Precision and Recall. Formula: [ F1 = 2 \times \frac{Precision \times Recall}{Precision + Recall} ]

2. Implementing the Monitoring

2.1 Code Snippet for Calculating Metrics

You can implement the calculations for these metrics in your production monitoring code as follows:

# Example Code for Calculating Metrics

def calculate_metrics(tp, tn, fp, fn):
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
    
    return {
        'TP': tp,
        'TN': tn,
        'FP': fp,
        'FN': fn,
        'Precision': precision,
        'Recall': recall,
        'F1 Score': f1,
    }

2.2 Example Usage

Assuming you have collected data about true positives, false positives, true negatives, and false negatives, you can call the function like so:

# Example values for monitoring
tp = 1145.0
tn = 80.0
fp = 80.0
fn = 0.0

metrics = calculate_metrics(tp, tn, fp, fn)
print(metrics)

3. Visualizing the Metrics

Visual representation helps in understanding the quality assurance metrics better. Use a visualization library to plot precision and recall metrics against the false positive rate (FP_rate) and true positive rate (TP_rate).

3.1 Code Snippet for Visualization

import matplotlib.pyplot as plt

def plot_metrics(fp_rate, tp_rate, precision, recall):
    plt.figure(figsize=(10, 5))
    
    plt.plot(fp_rate, tp_rate, label='TP Rate', marker='o')
    plt.plot(fp_rate, precision, label='Precision', marker='x')
    plt.plot(fp_rate, recall, label='Recall', marker='^')

    plt.title('Quality Assurance Metrics')
    plt.xlabel('False Positive Rate')
    plt.ylabel('Rate')
    plt.legend()
    plt.grid()
    plt.show()

# Example data
fp_rate = [0.0, 0.065306, 0.934694, 1.0]
tp_rate = [1.0, 0.000000, 1.000000, 0.0]
precision = [0.134498, 0.865502, 0.0, 0.074697]

plot_metrics(fp_rate, tp_rate, precision, recall)

4. Summary

Monitoring the production performance of the Splink demos involves calculating and visualizing significant quality assurance metrics such as TP, TN, FP, FN, Precision, Recall, and F1 Score. Using these metrics enables the identification of model performance trends and can inform adjustments to improve accuracy.

Understanding how to gather, compute, and visualize these metrics is crucial for maintaining the integrity of the predictions produced by the system.

Reference: