Update 2021: We’ve compiled the results of the State of Enterprise CI/CD survey in a downloadable datasheet so that you can gain a better understanding of where your CI/CD implementation stands compared to other survey respondents. After reading the survey results, learn how to optimize your CI/CD implementation with our resources.
A CI/CD Pipeline for Python that eliminates “works on my machine” and doesn’t create a schism between dev and test environments? Read on.
Working in a professional setting as a developer these days means coordinating your development activities with other members of your team. Even though you might have the freedom to use your favorite editor, and agree with your team members to use your favorite language to develop your code in, you’ll still need standardized tooling to communicate and synchronize your code with the team. One of the primary standard tools will be a shared source code control system, which allows developers to maintain their productivity while reducing the burden on support teams (if any).
In the open source world, there’s another area where not using standardized tooling has resulted in a growing pain: the runtime environment. It’s trivially easy for developers to customize their development environment, especially for interpreted languages, just by adding a new import statement. However, that import likely pulls in dependencies, and now you’ve got a unique environment that may be difficult to replicate on another developer’s machine, or in production at deployment time. The difficulty of merging the “runtime environment,” which includes not just the OS/version but also all binary and/or source code dependencies, becomes a burden on development.
There are a number of solutions that try to ease this pain, including:
- Language specific virtual environments (e.g., pipenv, virtualenv and the like)
- Specialized package manager environments (like conda)
- Containers (like Docker)
- Full Virtual Machines (VMs)
These solutions all attempt to standardize the runtime environment across dev, test and production — wherever the code will be developed and executed. Each of these solutions are trying to solve the problem of a reproducible environment in order to complete the puzzle of synchronized development. But all of them end up increasing complexity in the process.
The midpoint between development and production, where the synchronized source code is executed and validated, is the test environment. In modern development practices, this phase is automated using Continuous Integration/Continuous Delivery (CI/CD) tools. CI/CD setup is critically important because it’s here that:
- Source code from all developers needs to be merged
- All dependencies need to be accounted for
- Environment setup needs to be as close as possible to a production environment
DevOps professionals usually have a bag of tricks to make all this work. But those tricks are usually so specific that it’s impossible to generalize them as a solution for individual, local development environments. As a result, a schism is created between dev and test environments causing the “works on my machine” syndrome.
ActiveState is coming up with a new way to solve this problem by making the runtime environment perfectly reproducible not only on individual developer machines, but on CI/CD and production machines as well, with a single automatable command.
This tutorial focuses on how to use the ActiveState solution in conjunction with one popular CI/CD tool on the market, Travis CI. We’ll be using a sample application written in Python and hosted on GitHub (note that you can also use your favorite Perl project instead). Travis CI will grab the source code from GitHub and the language runtime environment from ActiveState Platform, and then build and run its tests for a successful round of development iteration.
- The project’s code base is hosted on Github.
- The runtime environment for this project is located on the ActiveState Platform.
We’ll use the standard git client to sync the code from GitHub, and the ActiveState Platform CLI (a.k.a. the State Tool) to sync the Python runtime environment.
There are also other methods to set up the environment (e.g. using pip, conda, Docker containers, etc), but in this case all that’s required is to sync the code and the runtime. There’s no need to pip install or conda install dependencies since the ActiveState Platform automatically retrieves and resolves all dependencies for you, up front. All the required packages and their dependencies will be installed to the CI/CD environment via the State Tool with a single command, which means the application will just work.
Without further ado, let’s dive into the details for a couple of popular cloud-based CI/CD solutions.
Travis CI
Travis is one of the most popular CI tools used by the open source community, so we’ve chosen to use it for this Linux build example.
The first step to using Travis is to create an account and sign in. To simplify things, you can use your GitHub account to sign in, which will make your repositories available to and buildable with Travis. Once you’ve logged in, you can just fork our learn-python repo into your own account and enable it in Travis by going to your profile page and clicking the slider under the Repositories tab.
The next step is to go to the project page by clicking on the “learn-python“ project name. Before we can build the project, we’ll need to link it to the ActiveState Platform by providing a valid API key. In the Travis settings click on More Options->Settings.
And add a new Environment Variable named ACTIVESTATE_API_KEY.
To get the API Key, you’ll need to install the State Tool locally on your machine:
sh <(curl -q https://platform.www.activestate.com/dl/cli/install.sh)
Authenticate with your username and password:
state auth --username <yourname> --password <yourpassword>
And then run the following command:
curl -X POST "https://platform.www.activestate.com/api/v1/apikeys" -H "accept: application/json" -H "Content-Type: application/json" -H "Authorization: Bearer `state export jwt`" -d "{ \"name\": \"APIKeyForCI\"}"
The JSON response contains your API key in the “token” field (not “tokenID”). Copy this value (don’t forget to exclude the quotation marks around the text) into the ActiveState API Key Environment Variable you created in Travis and you’re good to go.
Now all you need to do is just go to the project page by clicking the “Current” tab in order to start the build.
You’ll notice from the log that the code is synced using git. Ignore the mention of Ruby, which is just the default language execution environment for Travis. We’ll be setting up our own execution environment.
In order to understand what’s happening further, let’s have a look at the configuration file travis.yml:
# Install dependencies. install: - sh <(curl -q https://platform.www.activestate.com/dl/cli/install.sh) -n before_script: # Run linting and tests. script: - state run lints - state run tests # Turn email notifications off. notifications: email: false
As you can see, the configuration is pretty barebones. The install section is just a one liner that installs the State Tool, and the script execution part has only two lines to run the linting and the tests. Both these scripts are actually executed by the State Tool as defined in the activestate.yaml file, along with a link to the ActiveState Platform project configuration:
project: https://platform.www.activestate.com/shnewto/learn-python?commitID=2d0ab259-6bfa-40ac-8e8d-823fa9a2afeb languages: - name: python scripts: - name: noop value: rundll32 - name: tests value: | pytest - name: lints value: | pylint src flake8 src --statistics --count
The project URL is where the runtime environment (a version of Python + required packages + all dependencies) for the “Learn Python” project has been configured (you can paste it in a browser to view it through the ActiveState Platform Web UI). The State Tool downloads and installs it into the CI environment. The “state run lints” script triggers the download and install of the runtime environment automatically (called “activation”), and then “state run tests” reuses the “activated” environment to execute the tests in. If everything has been set up properly, you should see a successful build result.
Setting up CI/CD environments: Conclusion
The setup of CI/CD environments are more difficult than they need to be. A lot of the difficulty lies in trying to triangulate a CI/CD environment that can not only merge multiple different development environments but also output a production-ready solution. Put simply: organizations cannot trace a production workload back to its original packages and commits.
The ActiveState Platform, in conjunction with the State Tool, simplifies CI/CD setup by providing a consistent, reproducible environment deployable with a single command to developer desktops, test instances and production systems. Additionally, since the ActiveState Platform builds all runtime packages and dependencies from vetted source code, binary artifacts can be traced back to their original source, helping to resolve the build provenance issue.
In these ways, the ActiveState Platform effectively eliminates the “works on my machine” problem, and simplifies the setup of a more secure, consistent, up-to-date CI/CD pipeline.
- If you’d like to try it out, sign up for a free ActiveState Platform account where you can create your own runtime environment and download the State Tool.
- How does your enterprise’s practice of CI/CD compare to other enterprise’s CI/CD practice? Our State of Enterprise CI/CD 2020 survey has concluded and you can see the results here. Find our CI/CD resources here.
Related Blogs:
