moj-analytical-services / splink_demos

This documentation outlines the procedures and code implementations used for managing and storing secrets in production within the moj-analytical-services/splink_demos project.

Introduction

Effective secret management is critical for maintaining security and confidentiality in production environments. This documentation covers the step-by-step processes and code examples to illustrate how secrets are handled.

Procedures

1. Secret Storage

Secrets such as API keys, database credentials, and tokens should never be hardcoded directly in the application code. Instead, they should be stored securely and retrieved at runtime. Here’s an example of managing secrets using environment variables.

Setting Environment Variables:

In a shell environment, you can define your secrets:

export DATABASE_URL="your-database-url"
export API_KEY="your-api-key"

Accessing Environment Variables in Code:

In your application code, you can access these environment variables like this:

import os

database_url = os.getenv('DATABASE_URL')
api_key = os.getenv('API_KEY')

print(f"Connecting to database at {database_url} with API key: {api_key}")

2. Using Configuration Files

When additional flexibility is needed, consider using configuration files that are not included in version control. For example, use a config.json file to store non-sensitive configuration and load it in your application.

Example config.json:

{
    "app_name": "Splink Demos",
    "version": "1.0",
    "api_endpoint": "https://api.yourservice.com"
}

Loading Configurations:

import json

with open('config.json') as config_file:
    config = json.load(config_file)

print(config["app_name"])

3. Secret Management Tools

For organizations that require more robust solutions, leveraging secret management tools like HashiCorp Vault or AWS Secrets Manager is recommended. These tools allow secure storage, versioning, and access control of secrets.

Example with HashiCorp Vault:

First, set up your Vault server and store your secrets:

vault kv put secret/myapp api_key=your-api-key

Retrieving Secrets:

import hvac

client = hvac.Client(url='http://127.0.0.1:8200', token='your-token')
secret = client.secrets.kv.read_secret(path='secret/myapp')

api_key = secret['data']['data']['api_key']
print(api_key)

4. Minimizing Secret Exposure

To minimize secret exposure, ensure to:

  • Limit the scope of secrets (only use what’s necessary).
  • Use role-based access control (RBAC) for sensitive operations.
  • Regularly rotate secrets and revoke compromised ones.

5. Auditing and Monitoring

Implement logging and monitoring of access to secrets. This can be achieved through built-in features of secret management systems or custom logging practices.

Example Logging:

import logging

logging.basicConfig(level=logging.INFO)

def access_secret(secret_name):
    logging.info(f"Accessed secret: {secret_name}")
    # Code to access the secret goes here

access_secret('API_KEY')

Conclusion

Managing secrets effectively is crucial for the security of your application. The methods described here provide a robust framework for handling secrets in production to ensure confidentiality and security compliance.

References