The single most exciting change that Python 3.11 introduces is a change to the way the interpreter processes general bytecode. Much like Java in its infancy, Python has been characterized as “slow.” But much like the Java Virtual Machine (JVM) before it, the Python interpreter is a Just In Time (JIT) compiler that needs to be optimized in order to be performant. Speed has never been a priority to date, but Microsoft has been helping to fund the Python Software Foundation (PSF) with the express goal of speeding things up. As a result, Python 3.11 will introduce an optimization in the interpreter that “predicts” the types and values it is currently operating on in order to process them quicker.
In other words, when the interpreter encounters a general binary operation (multiply, add, divide, etc) it can replace it with the specific operation for integers, floats, strings, etc, significantly speeding up processing. There are a number of other tweaks that will also make things faster, including:
- Improved performance for integers of less than one machine word
- Faster calls and returns through better handling of frames
- Better object memory layout, and reduced memory management overhead
- Zero overhead exception handling
Combined with optimizations to the way core modules load, function calls now work, and the interpreter itself starts up, you’ll see a 10–60% increase in processing speeds, depending on what your code is trying to do. On average, though, you’ll benefit from a 25% speed increase without making a single change to your code!
For long-suffering Pythonistas that have had to trade off the slowness of Python for its ease of use, the dramatic increase in performance is a welcome addition. But it’s not the only feature introduced in Python 3.11.
New Features and Functionality in Python 3.11
The full list of improvements introduced in Python 3.11 includes:
General changes
- Easier Debugging with Fine-Grained Error Locations in Tracebacks
- Previously, errors would be identified by line. Now the specific error on the line will be highlighted, eliminating ambiguity and speeding up debugging.
- Simpler Error Handling with Exception Groups and except*
- Previously, if some task threw multiple errors, you’d have to deal with them one at a time. With 3.11, you can now address all those errors together, simplifying exception handling.
- Better Packaging with Support for Parsing TOML in the Standard Library
- TOML, or Tom’s Obvious Minimal Language is a configuration file format similar to YAML. It contains consistent and predictable build information about your project. With 3.11 adding tomllib to the standard library, there’s no longer any excuse to use setup.py to package your project anymore. Learn how to migrate your project from setup.py to pyproject.toml.
- Data Class Transforms
- Python provides no way for libraries with dataclass-like semantics to declare their behavior to type checkers. With 3.11, you can now add the “typing.dataclass_transform” decorator to denote how a given function, class, or metaclass behaves like a dataclass.
- Asyncio Supports Task Groups
- Much like how Async works in JavaScript, TaskGroup lets users run nested tasks, and continue running them even if one fails. Errors can still be raised after processing finishes using the Exception Groups described above.
- Regex Enhancements with Support for Atomic grouping ((?>…)) and possessive quantifiers (*+, ++, ?+, {m,n}+)
- The re (regex) module is getting a bit of an overhaul that not only makes it faster, but also means you can finally take advantage of common regex syntax found in most other languages’ implementations of regular expressions.
Typing and Language Changes
- Self Type
- If you’ve ever used a class method that returns self you probably found it awkward to annotate, and as a result, you’d get some weird results from your analysis tools. Now, “typing.Self” lets you annotate the return value of a class method as just “Self.”
- Variadic Generics
- Python provides support for type hints that now include TypeVarTuple. This allows you to specify a placeholder for tuples, which is helpful when you’re expecting a specific array structure.
- Arbitrary Literal String Type
- Previously, type annotations had no way to indicate a given variable needed to be a string defined in source code. The new typing.LiteralString annotation lets linters test whether a variable is either a string defined in source or a new string composed of only source-defined strings.
- TypedDict Required vs Missing Items
- TypedDict currently doesn’t support declaring some keys as required and others as potentially-missing. 3.11 introduces Required() and NotRequired() to provide a way for you to account for these.
Next Steps:
As usual, the ActiveState Platform will make Python 3.11 available within days of release so you can upgrade your Python environments to take advantage of the speed boost. To get prepared, you can sign up for a free account now and create your own environment with the latest version of Python 3.10 (currently 3.10.5).
