Django Settings In the Cloud
At Instawork we have multiple different environments (test, development, staging, production) and we have quite a few different Django based services that all need to be configured with their own settings. There are a lot of ways to manage these settings with dedicated packages and practices/patterns you can employ that are documented on the web.
Our requirements for a solution included:
- Security: there are can be sensitive secret keys that are exposed as Django settings. These should be encrypted and inaccessible from most developers.
- Ease of Use: committing settings to version control is ok when they are shared across many services, but when they are service-unique this becomes a nuisance to maintain.
- Auditing: we needed a way to track what changes were made and when.
- Debugging: settings should be easy to find and examine across different services; in addition it should be possible to analyze settings within a single system.
- Integration: should integrate with our existing cloud infrastructure.
None of the out-of-the-box solutions we found fulfilled all of the requirements, so we decided to build on tools provided by our cloud provider, Amazon Web Services.
Parameter Store
Under the Systems Manager service within AWS, you can find the parameter store. At first inspection, this is a simple key-value store, but looking deeper it has a few convenient features:
- Changes to parameters are versioned and stored in a history for auditing or debugging purposes.
- Values can be encrypted using another one of AWS’s services, the Key Management Service (KMS).
- It doesn’t require managing the underlying implementation and instead gives us an interface to create/modify/search for values.
- We can pass data from the parameter store conveniently into ECS (Elastic Container Service).
The Parameter Store is by no means perfect: the console interface is clunky and searching for keys is painful, but overall it fulfills a lot of requirements we had for our system. In the rest of this blog post, we’ll cover how to use the Parameter Store to store Django settings securely.
For our infrastructure, we deploy docker containers using AWS’s ECS. For the purposes of this blog post, you can use any means of deploying your infrastructure, but to make this more concrete, we’ll assume ECS.
Setting up an Encryption Key
First, let’s create an encryption key we can use to secure our settings. Using AWS’s CLI tool:
> aws kms create-key
{
"KeyMetadata": {
"AWSAccountId": "<hidden>",
"KeyId": "<hidden>",
"Arn": "arn:aws:kms:us-west-2:<hidden>:key/<hidden>",
"CreationDate": 1549328129.379,
"Enabled": true,
"Description": "",
"KeyUsage": "ENCRYPT_DECRYPT",
"KeyState": "Enabled",
"Origin": "AWS_KMS",
"KeyManager": "CUSTOMER"
}
}Something like this should be returned. That’s it, you now have an encryption key. Keep track of the KeyId we’ll need to reference it later.
Creating Parameters in SSM
We’ll use AWS’s CLI for creating our first parameter. We’ll use the encryption key we made in the previous section. If storing sensitive values, you can alternatively pass arguments as a json file (see documentation for details).
aws ssm put-parameter \
--name "/Dev/WebServer/TEST_PARAMETER" \
--value "hello world" \
--type "SecureString" \
--key-id "<hidden>"The output in this case will be the version of the parameter which is 1.
A word about key naming. AWS recommends using slash separate namespaces for keys, for example:
- /Dev/WebServer/TEST_PARAMETER_FOO
- /Dev/WebServer/TEST_PARAMETER_BAR
- /Dev/ApiServer/TEST_PARAMETER_BAZ
- /Staging/TaskProcessor/TEST_PARAMETER_BAC
This allows us to query for all development web server keys by using the string “/Dev/WebServer”. This will come in handy later as we are using these parameters from within Django.
IAM Roles and Permissions for ECS
Since we are going to be bootstrapping our configuration from AWS itself, we can’t use Django configuration for storing AWS access keys. Ideally the instance/task that is running your Django service has been assigned a role. If not, you can make one to grant it access to
- Read from the parameter store
- Read the key from KMS
- Decrypt from parameter store using the key
Django Configuration
Now that we have everything squared away on the AWS side, let’s jump into how we would move one of our settings into the cloud.
Django settings are not a flat file, but are executed as python so we can use the Boto 3 package to connect to the SSM and extract parameters.
import boto3
ssm = boto3.client('ssm')def _get_ssm_key(name):
key = ssm.get_parameter(Name=name, WithDecryption=True)
return key['Parameter']['Value']SECRET_KEY = _get_ssm_key('/Production/ApiServer/SECRET_KEY')
DEBUG = False
....
More Complex Configuration
What if we want to store arrays or dictionaries? We can store these values as JSON and decode them when we extract them from the parameter. To make things easier, it’s best to just make sure that ALL values are stored as JSON.
The code above can be modified to interpret these JSON values accordingly:
import boto3
import json
ssm = boto3.client('ssm')def _get_ssm_key(name):
key = ssm.get_parameter(Name=name, WithDecryption=True)
try:
return json.loads(key['Parameter']['Value'])
except ValueError:
# Assume value is a simple string
return key['Parameter']['Value']DEBUG = False
SECRET_KEY = _get_ssm_key('/Production/ApiServer/SECRET_KEY')
DATABASES = _get_ssm_key('/Production/ApiServer/DATABASES')
....
How to use Namespacing to your advantage
Since we are namespacing configuration in a way that lets us differentiate between servers easily, we can pass this information during startup to extract values exclusively for the service in question.
AWS_DJANGO_SETTINGS_PATH = ‘/Production/ApiServer’
We can pass the environment variable through to docker using the -e keyword manually or through ECS’s task configuration. We can get this value when the settings load and get the keys we need.
import boto3
import json
import os
ssm = boto3.client('ssm')NAMESPACE = os.environ.get('AWS_DJANGO_SETTINGS_PATH')def _get_ssm_key(name):
key = ssm.get_parameter(Name='{}/{}'.format(NAMESPACE, name), WithDecryption=True)
try:
return json.loads(key['Parameter']['Value'])
except ValueError:
# Assume value is a simple string
return key['Parameter']['Value']DEBUG = False
SECRET_KEY = _get_ssm_key('SECRET_KEY')
DATABASES = _get_ssm_key('DATABASES')
....
With this, we can make sure that as long as the environment variable is set correctly, each server will start with the correct secret key and database configuration.
Dynamic Settings
Now let’s expand to a situation where we don’t know the names of the settings we’re interested in up front. Let’s say a developer wants to add a setting for a new 3rd party API, THIRD_PARTY_API_SECRET. How can we automatically discover this settings value without having to edit our settings.py file? We can use Boto 3 to search for all keys under a specific namespace and load them all into our settings.
import boto3
import json
import os
ssm = boto3.client('ssm')NAMESPACE = os.environ.get('AWS_DJANGO_SETTINGS_PATH')def _get_ssm_keys(path, next_token=None):
aws_params = {
'Path': path,
'WithDecryption': True
}
if next_token:
aws_params['NextToken'] = next_token
res = client.get_parameters_by_path(**aws_params)
_process_keys(keys['Parameters'])
return keys['NextToken']
def _process_keys(keys):
for key in keys:
try:
value = json.loads(key['Value'])
except ValueError:
value = key['Value']
globals()[key['Name'].split('/')[-1]] = value
def load_settings():
next_token = _get_ssm_keys(NAMESPACE)
while next_token:
next_token = _get_ssm_keys(NAMESPACE, next_token)
DEBUG = False
load_settings() // Will load all available settings in the namespace
There are a few things going on. The first of which is making sure that we paginate the parameters resource in case there is more than one page of settings. This is what the next_token and the while loop is about.
The more interesting thing here is how we are storing the retrieved values into our settings. Here we are using the global keyword which returns to us the global namespace. We can modify this namespace and insert new values dynamically passing a string name as a parameter and setting its value.
Conclusions
We went through the steps of setting up AWS parameter store with Django settings and then loading those settings back into our Django applications. We covered how to dynamically load non-string values in your application so you can freely add settings without having to explicitly add them to your python settings file. Just adding them to the parameter store means that the next time settings are read, the value will be available!
There are some limitations to this approach:
- You cannot serialize python code to JSON if your settings require any sort of calculation or are derived in a non-trivial way.
- If there is a network issue communicating to SSM, your settings will not load and your application may not function.
- Django still needs to be reloaded if your settings change
In spite of these minor limitations, we have found the flexibility, visibility, and improved security to be indispensable as our codebase grows. A central location where we can monitor, audit, and modify our configuration allows us to spend less time managing our server and more time building a great product.
The parameter store isn’t the easiest to work with using AWS’s console so we wrote a script to make it easier.
Have you found other patterns or methods to keep your settings in check? Comment below.
