What’s new in Python 3.7

Python 3.7 adds new classes for data handling, optimizations for script compilation and garbage collection, and faster asynchronous I/O

Serdar Yegulalp Feb 07th 2018

Python 3.7, the latest version of the language aimed at making complex tasks simple, has officially entered its beta release phase. The final version of Python 3.7 is due in June 2018, but from this point onward no new features will be added to the Python 3.7 release.

The most significant additions and improvements to Python 3.7 include:

  • Data classes that reduce boilerplate when working with data in classes.
  • A potentially backward-incompatible change involving the handling of exceptions in generators.
  • A “development mode” for the interpreter.
  • Nanosecond-resolution time objects.
  • UTF-8 mode that uses UTF-8 encoding by default in the environment.
  • A new built-in for triggering the debugger.

Python data classes

Python is known far and wide as a fast and convenient way to manipulate structured data. Python provides classes to structure data and attach common behaviors to instances of that data, but classes with many initializers have long suffered from needing a lot of boilerplate code to instantiate them. For example:

class User():
    def __init__(self, name, user_id, just_joined=True):
        self.name = name
        self.id = user_id
        self.just_joined = just_joined

To automate this kind of class instantiation, Python 3.7 introduces a new module, dataclasses, as described in PEP 557. It provides a decorator that allows the above behavior to be reproduced in a much simpler way:

@dataclass
class User():
    name: str
    user_id: int
    just_joined: bool=True

The resulting class works like a regular Python class. You can also declare certain fields “frozen” or immutable, and automate (or manually override) the creation of special methods on properties, like __hash__ or __repr__.

Python generator exception handling

A change that has been in the works for some time now, as outlined in PEP 479, is designed to make it easier to debug a StopIteration exception raised by a Python generator. Previously, it was too easy for a generator to raise a StopIteration when it encountered another problem, instead of because it had run out of things to iterate through. This created a whole class of bugs that were hard to trace.

With Python 3.7, when a generator yields up a StopIteration exception, the StopIteration exception is translated into a RuntimeError exception so that it doesn’t silently bubble all the way up through the application’s stack frames. This means some programs that weren’t too discerning about how they handled generator behaviors will throw a RuntimeError in Python 3.7. In Python 3.6, this behavior produced a deprecation warning; in Python 3.7, it produces a full error.

One quick fix is to use a try/except block to catch the StopIteration before it propagates outside the generator. A better solution is to rethink how generators are constructed—for instance, to use a return statement to terminate a generator instead of manually raising StopIteration. See PEP 469 for more details on how to remedy this in existing code and how to guard against it in new code.

Python development mode

A new command-line switch for the Python interpreter, -X, lets the developer set a number of low-level options for the interpreter. With Python 3.7, the option -X dev enables “development mode,” a slew of runtime checks that normally have a big impact on performance, but are useful for a developer during the debugging process.

Options activated by -X dev include:

  • Debug mode for the asyncio module. This provides more detailed logging and exception handling for asynchronous operations, which can be difficult to debug or reason about.
  • Debug hooks for memory allocators. This is useful for those writing CPython extensions. It enables more explicit runtime checks on how CPython allocates and frees memory internally.
  • Enabling the faulthandler module, so that tracebacks are always dumped after a crash.

Python time functions with nanosecond resolution

A new class of time functions in Python 3.7 return time values with nanosecond precision. Even though Python is an interpreted language, Python core developer Victor Stinner made a case for reporting time with nanosecond accuracy. The biggest reason is to avoid losing precision when dealing with converting time values that are recorded by other programs, such as databases.

The new time functions use the suffix _ns. For instance, the nanosecond version of time.process_time() is time.process_time_ns(). Note that not all time functions have nanosecond equivalents, as some of them don’t benefit from it.

Python UTF-8 mode

Python has long supported UTF-8 for easy handling of strings and text. But the locale in the surrounding environment is sometimes still ASCII, not UTF-8, and the mechanisms for detecting locale aren’t always reliable.

Python 3.7 adds what’s called “UTF-8 mode,” enabled by way of the -X command line switch, that assumes UTF-8 is the locale provided by the environment. In POSIX locales, UTF-8 mode is enabled by default, but it’s disabled by default elsewhere to avoid breaking backwards compatibility. It’s worth experimenting with having UTF-8 mode on by default, but it shouldn’t be enabled in production until you’re confident that all interactions between Python and the surrounding environment use UTF-8.

Built-in breakpoint() function

Python comes with its own built-in debugger, although it can hook into third-party debugging tools as well, as long as they can speak to Python’s internal debugging API. What Python has lacked until now, though, is a standardized way to programmatically trigger the debugger from within a Python app.

Python 3.7 adds breakpoint(), a built-in function that causes execution to switch to the debugger when called. The debugger in question doesn’t have to be Python’s own pdb; it can be any debugger that has previously been set as the debugger of choice. Previously, the debugger would have had to be set manually, then invoked, which made for more verbose code. breakpoint() makes it possible to invoke the debugger in a single command, and to keep a clean separation from setting up the debugger and invoking it.

Other new Python 3.7 features

Dozens of other changes have landed in Python 3.7. Here are some others you’re likely to encounter when working with the latest version of Python:

C API for thread-local storage support

The Thread Specific Storage (TSS) API, described in PEP 539, replaces the legacy Thread Local Storage (TLS) API. People customizing CPython, or writing CPython extensions that make use of the interpreter’s internal APIs, will need to be aware of this.

Module attribute access customization

When you create a module in a Python program, you can now customize the behavior of attribute access on instances of that module. You can do this by just creating a __getattr__ method inside the module, same as you would for a class. This way, things like requests for nonexistent functions or methods inside the module can be intercepted, flagged, or proxied.

Python importlib resources

The importlib module can now be used to read “resources,” or binary artifacts shipped with a Python application such as a data file. This way, a developer can access those files through importlib’s abstractions, so it doesn’t matter if they’re stored in a .zip file or in a directory somewhere on the system.

Under-the-hood optimizations

Many individual operations are now faster:

For the complete rundown of what’s new in Python 3.7, the Python Software Foundation has published a document covering all the details. It will be updated up to the official release of Python 3.7 to include any last-minute changes.

Download Python 3.7

While Python 3.7 is still in beta, it can be obtained from the Python Software Foundation’s pre-release download page.