Pip Tools
Pip tools are a set of third party command line tools designed to help with Python 3 dependency management by keeping dependencies up to date, even when you have them pinned.
You can install it by running the following command:
python -m pip install pip-toolsYou can then run the following command to automatically create a requirements.txt file based on your setup.py or requirements.in file:
python -m piptools compileIf your existing requirements.txt file contains the complete and proper set of dependencies, no changes will be made, even if updates are available. However, you can force pip-compile to update all packages in an existing requirements.txt by running the following command:
pip-compile --upgradeYou can even update specific packages to the latest version (or any specific version) by running the following command:
pip-compile --upgrade-packageSo far so good, but pip tools (like pip itself) does not do Python dependency resolution, which means that upgrading is a crapshoot: you may pull in dependencies or transitive dependencies (ie., dependencies of dependencies) that don’t work together, thereby corrupting your environment.
Advanced Dependency Management
Unfortunately, most real world Python projects are more complicated than those supported by pip tools. For example, pip tools currently provides no way to:
- Distinguish packages from the dependencies they pull in
- Create a set dependencies and versions that are known to be compatible
- Update a virtual environment efficiently with a single command
- Support multiple configurations of dependencies that aren’t just subsets of the original. For example, one configuration for each environment: development, test/CI/CD, production, etc.
If you require any of these features, consider the ActiveState Platform, which offers advanced dependency management for Python projects.
Distinguish Packages from Dependencies
Packages refer to those Python libraries that are required to run your application. Dependencies are libraries that the packages require to be able to run. There are other dependencies, as well, including transitive dependencies (which are dependencies of dependencies) and Operating System or OS-level dependencies (which are dependencies required to run the application on the specified OS).
For each Python project the ActiveState Platform provides:
- A complete Bill of Materials (BoM) view showing:
- The Python version
- Packages
- Direct dependencies, as well as transitive dependencies
- OS-level dependencies
- Shared dependencies (ie., OpenSSL)
As a result, you can easily distinguish between top level packages you explicitly included in your project versus their dependencies that were pulled in automatically.
Dependency Resolution
Python dependency resolution refers to the ability for a package manager (like pip) to be able to not only pull in all the dependencies required by the packages in your project, but also be able to ensure they are all compatible with each other. Unfortunately, neither pip nor pip tools are able to accomplish this.
By way of comparison, the ActiveState Platform will not only resolve all dependencies to ensure they work together but will also flag unresolvable conflicts and even suggest a manual workaround to the problem.
By following the instructions and selecting a more recent version of pandas will resolve the conflict. Note that when upgrading a package, the ActiveState Platform will show you all the cascading configuration changes so you can know the ramifications of any change to your environment BEFORE you commit to the update.
Virtual Environment Creation & Updating
As most Python software engineers know, the best practice is to install each of your projects in a virtual environment in order to isolate dependencies and prevent conflicts.
The ActiveState Platform provides natives support for virtual environments, which means that whenever you install a project using our command line interface, the State Tool will create a virtual environment (similar to venv, virtualenv or pipenv) on your local system and populate it with the runtime environment, including:
- The version of Python required by your project
- All the packages and dependencies required to run your application
The runtime environment is pre-compiled from source code by the ActiveState Platform (including linked C libraries) in a reproducible manner. As a result, deployment happens much faster than if every dependency was independently retrieved from a repository (like PyPI or GitHub) in a serial manner.
Updates to the local environment happen in the same: a single command downloads and installs all updated dependencies.
Multiple Python Environment Configurations
A separate configuration of the Python runtime environment is often required for each environment the application is to be developed, tested and run in. The most common way to do this is to create a default configuration (typically for the development environment) and have the other environments simply inherit the dependencies with minor differences.
However, problems arise when the configuration is targeted at a different operating system or platform, such as Python 2, which might arise when doing migration work. Simple inheritance is not enough.
The ActiveState Platform provides the flexibility to create multiple child configurations from a parent configuration and:
- Inherit all, some or no updates from the parent configuration
- Completely change the child configuration, including OS, platform and packages/dependencies, as required
Ready to see for yourself? You can try the ActiveState Platform by signing up for a free account using your email or GitHub credentials. Or sign up for a free demo and let us show you how you can better manage dependencies.
Watch this video to learn how to use the ActiveState Platform to create a Python 3.9 environment, and then use the Platform’s CLI (State Tool) to install and manage it.
Frequently Asked Questions
What are some benefits of PIP-tools?
Pip tools are a set of third party command line tools designed to help with Python dependency management by keeping dependencies up to date, even when you have them pinned. Pip tools can:
- Automatically generate a requirements.txt for you from your requirement.ini or setup.py file
- Automatically update all the dependencies in your requirements.txt file with a single command
However, advanced dependency management functionality, such as dependency resolution and dependency conflict management are lacking. As a result, upgrading may or may not corrupt your environment.
To avoid Python environment corruption, use a package management solution like the ActiveState Platform, which not only resolves dependencies but flags and provides manual solutions for dependency conflicts. Try it yourself by signing up for a free account.
How do I use PIP-tools?
You can use pip-tools to help with dependency management in multiple ways, including:
- Create a requirements.txt file by running:
python -m piptools compile- Update all dependencies in a requirements.txt file by running:
pip-compile --upgrade- Update a specific dependency to the latest version (or any specific version) by running:
pip-compile --upgrade-packageUnfortunately, pip-tools will not resolve dependencies on upgrade, potentially corrupting your environment. Instead, try using the ActiveState Platform’s command line interface the State Tool, which will:
- Automatically create virtual environments for each of your projects, and help you manage them
- Install, update and resolve dependencies
You can download the State Tool for free and start working with it by referencing our User Guide.
What can I use instead of PIP?
Pip is the default package manager for Python, and is included with every version of Python you install. However, their a number of alternative package managers available for Python, including:
- Conda – the package manager for Anaconda’s Python ecosystem
- Poetry – a third-party package manager best known for its ability to perform dependency resolution – a feature that is still missing in pip.
- ActiveState Platform – a package management solution that includes a number of advanced package management features, including:
- Automated builds from source code, including linked C libraries
- Automated dependency resolution plus dependency conflict handling
- Vulnerability remediation
Read more about alternatives to pip in our Python Package Management Guide for Enterprise Developers white paper.
What does Pip-sync do?
Pip-sync is a feature of pip-tools, which allows you to update a virtual environment by installing, upgrading, or uninstalling all dependencies necessary to match the contents of your requirements.txt file.
You can run pip-sync with the following command from within your virtual environment:
python -m piptools syncSince environments can often get out of sync with the requirements.txt, this a good way to realign it. However, because pip-tools does not perform dependency resolution, you may be left with:
- Orphaned dependencies
- Conflicting/unresolved dependencies
Instead, consider using the ActiveState Platform, which lets you sync your local virtual environment with the current project configuration by simply running “state pull.” But the ActiveState Platform provides you with the flexibility to update your virtual environment with:
- A branch of project, such as that for the test or production environment
- Any previous configuration of the project since all versions are saved
Ready to see for yourself? You can try the ActiveState Platform by signing up for a free account using your email or GitHub credentials. Or sign up for a free demo and let us show you how you can better manage environment sync.
Tame the complexities of your open source
Chat with a member of our product team today.
Tame the complexities of your open source
Chat with a member of our product team today.
