One of the most frustrating challenges in application development today is environment parity. While recent years have shown that virtualization and containerization tools like Vagrant and Docker (and ActiveState’s own State Tool) can ensure that the operating systems and dependencies that power an application’s infrastructure are consistent between environments, external dependencies are all-but-impossible to replicate in a non-production environment.
In a world where third-party dependencies have become irreversibly intertwined with the core business logic of nearly every software project, it is becoming increasingly difficult to keep development, staging, and production environments consistent with one another. While some products with self-hosted lookalikes – such as Minio to Amazon S3 – ease the pain of managing multiple deployments of a single service, it does little to ease the challenge of configuration management across environments.
When Production, Staging, Development, Bob’s Local Machine, and the QE Environment all require a different deployment of a given service, passing configuration files back and forth, or relying on complex conditionals to determine which deployment a given environment should speak to, becomes increasingly impractical. The need for flexibility also highlights the difficulty of these solutions in a large-scale environment. What if, for example, Bob’s Local Machine needs to temporarily speak to a service that is reserved for the Staging environment?
Environment Variables in Python
Cross-environment configuration is a pain, but thankfully the problems outlined above have become common enough in recent years that they are all-but-solved. Thanks to the support of the open source community, and the evangelism of best-practices like the 12 Factor Application, there has been a shift in recent years from file-based application configuration management to environment variable-based configuration management.
Available in every major OS, environment variables are – just as their name implies – variables that are implemented at the environment level to define the way applications running underneath it should behave. In simpler terms, they are application-agnostic variables that are managed outside the context of a given process. The primary advantage to using environment variables is that this allows developers to define how an application should run without changing a line of code.
For example, if Bob’s Local Machine needs to speak to the CDN service that is reserved for the Staging environment, he may change the CDN_URL environment variable to reflect what is defined in Staging without having to touch any managed code. More importantly, this allows Bob to define mock or internal services for use with unit and integration tests, all without having to write a single line of extra code.
Defining Environment Variables
While the act of defining environment variables is generally OS-dependent, the vast majority of programming languages have abstracted away these differences through the use of development packages like Python’s dotenv project. For instance, rather than having to define an API_USER environment variable at the OS level, a locally gitignored .env file can maintain environment variables across dev environments. This allows developers to utilize a locally-managed config file for setting environment variables, while at the same time enabling configuration via “true” environment variables in non-development environments.
As an example, here’s a simple .env file that a developer may use in their local environment:
APP_ENVIRONMENT=local APP_NAME=localhost QUEUE_DRIVER=sync API_USER=bob
On the flip side, the Production environment may define its environment variables within a Dockerfile, both of which are valid methods for maintaining environment variables.
Retrieving Environment Variables
Regardless of how environment variables are defined, they can always be retrieved in Python using the os.getenv() method:
import os
# Get environment variables
USER = os.getenv('API_USER')
Take note that, in the event that the environment variable is undefined, the value will default to None.
Getting Started with Secrets
Now, while environment variables are an excellent solution for managing disparate configurations across environments, they aren’t a silver bullet. In today’s development climate, security must be a top priority, and sensitive data must be kept in a secure manner.
Unfortunately, environment variables on their own are not secure. While they do a great job of storing configuration data, the way in which we define more sensitive data like passwords, API keys, and encryption keys should require more care. This is where secrets come into play. Encrypted at rest, secrets should only be retrieved in a single runtime as needed in order to reduce the odds of a data breach. In this way, even if your hosting provider gets compromised, you can rest assured that your sensitive secrets are locked up tight.
Creating & Managing Secrets with the State Tool
So, how do we create and manage secrets? While there are a number of different ways to tackle this problem, ActiveState’s State Tool is an excellent solution for the Python language. Similar to virtualenv or pipenv, the State Tool is a virtual environment management interface that will prevent cross-contamination of Python installations and configurations between projects. What sets it apart from other virtual environment management tools is its integration with the ActiveState platform, allowing for a central interface for managing environment configurations and, yes, secrets.
Before we can take advantage of the State Tool’s secrets management capabilities, we first need to use the State Tool to set up a virtual environment within our project directory. To do this, first identify the ActiveState project you’ll be working with (for this tutorial, you can use my project zachflower/envs-vs-secrets-demo as long as you have a free ActiveState Platform account, or you can create your own). Then, execute the state activate command for your given project:
$ state activate zachflower/envs-vs-secrets-demo Where would you like to checkout zachflower/envs-vs-secrets-demo? /home/zach/Projects/miscellaneous/activestate-variables/zachflower/envs-vs-secrets-demo Activating state: zachflower/envs-vs-secrets-demo The State Tool is currently in beta, we are actively changing and adding features based on developer feedback. Downloading required artifacts Downloading 1 / 1 Installing 0 / 1 [-----------------------------------------] 0 % You are now in an 'activated state', this will give you a virtual environment to work in that doesn't affect the rest of your system. Your 'activated state' allows you to define scripts, events and constants via the activestate.yaml file at the root of your project directory. To expand your language and/or package selection, or to define client-side encrypted secrets, please visit https://platform.www.activestate.com/zachflower/envs-vs-secrets-demo. To try out scripts with client-side encrypted secrets we've created a simple script for you in your activestate.yaml, try it out by running 'helloWorld'
As you can see, the activate command sets up the virtual environment as defined in your ActiveState project. In the event that the project has not yet been configured, a simple project will be seeded with default parameters to provide you with an example of how everything should go together. This configuration is stored in a file at the root of your project directory called activestate.yaml.
Configuration File for Secrets & More
The activestate.yaml file defines a development runtime under which your application will run. For example, the default one defines a simple script that utilizes a secret, as well as a few event listeners that perform actions whenever a defined event occurs:
project: https://platform.www.activestate.com/zachflower/envs-vs-secrets-demo
scripts:
# This script uses a secret. Note that you can define your own secrets at
# https://platform.www.activestate.com/zachflower/envs-vs-secrets-demo/scripts
- name: helloWorld
value: echo ${secrets.user.world}
events:
# This is the ACTIVATE event, it will run whenever a new virtual
environment is created (eg. by running `state activate`)
# On Linux and macOS this will be ran as part of your shell's rc
file, so you can use it to set up aliases, functions, environment
variables, etc.
- name: ACTIVATE
constraints:
os: macos,linux
value: |
echo "You are now in an 'activated state', this will give you a
virtual environment to work in that doesn't affect the rest of your
system."
echo ""
echo "Your 'activated state' allows you to define scripts, events
and constants via the activestate.yaml file at the root of your
project directory."
echo ""
echo "To expand your language and/or package selection, or to
define client-side encrypted secrets, please visit
https://platform.www.activestate.com/zachflower/envs-vs-secrets-demo."
echo ""
echo "To try out scripts with client-side encrypted secrets we've
created a simple script for you in your activestate.yaml, try it out
by running 'helloWorld'"
While there’s nothing particularly groundbreaking here, the potential power of this file should be immediately obvious. For example, the following script provides a basic example of how to utilize secrets in the configuration file:
scripts:
# This script uses a secret. Note that you can define your own secrets at
# https://platform.www.activestate.com/zachflower/envs-vs-secrets-demo/scripts
- name: helloWorld
value: echo ${secrets.user.world}
When the helloWorld command is executed (from within an activated state), it will echo the value of the user.world secret and, in the event that secret is not yet defined, will prompt for a value:
$ helloWorld The action you are taking uses a secret that has not been given a value yet. Name: world Description: - (This secret has no description, you can set one via the web dashboard) Scope: user (Only you can access the value) Please enter a value for secret "world": ******
Using Secrets
Although the example is relatively simplistic, the true value of secrets comes from the activation events. Instead of retrieving a secret value within the context of a script, environment variables can be defined using retrieved secrets without ever exposing those secrets outside of this secure environment:
project: https://platform.www.activestate.com/zachflower/envs-vs-secrets-demo?commitID=5fd1c161-c5a4-480c-8aba-29d8ab361b42
events:
- name: ACTIVATE
constraints:
os: macos,linux
value: |
export WORLD=${secrets.user.world}
Now, if the user.world secret is defined, then the WORLD environment variable will be defined and can be retrieved like any other environment variable:
$ echo $WORLD world!
However, if it is not defined, then the user will be prompted to define it upon activation of the ActiveState virtual environment:
The action you are taking uses a secret that has not been given a value yet. Name: hello Description: - (This secret has no description, you can set one via the web dashboard) Scope: user (Only you can access the value) Please enter a value for secret "world": ******
Pretty cool, right?
Taking it Further
Configuration management in application development is, for the most part, a solved issue, but secrets management just isn’t yet. The number of projects that have checked sensitive data into a version control repository is staggering, and is something even highly respected companies have done, but through the use of proper security hygiene and products like ActiveState’s State Tool, keeping sensitive configuration data safe and secure is becoming easier by the day.
- Try it out for yourself by creating a free ActiveState Platform account and downloading the State Tool to simplify secrets management.
- You can also watch our webinar about How to Manage Shared Secrets using the State Tool
Related Blogs:
Share Secrets Quickly and Easily without Sacrificing Security
