None

Python3 Type Checking

Scaling Python

Python is a great language for prototype development, but sometimes we want to develop production software with Python or we suffer the most recent cycle of prototypes becoming production. This is where the attributes that made Python a strong prototyping tool often interfere with the requirements of enterprise software (e.g. high reliability, complex deployment, multiple developers). Unittests go a long way towards ensuring robust execution of key mechanisms, but how can we catch typos in new code or nuanced return values?

Type Verification

PEP 3107 defined a grammar for arbitrary annotations on functions and variables across for Python 3.0+ and PEP 484 standardized the framing and interpretation for type annotations. Neither enforced any type-checking behavior, but instead made the type annotations available at run-time (via the __annotations__ attribute). mypy provides validation of type annotations with a reasonable type inference system that propagates types where appropriate (except for lists/dicts).

Example (from here)

import typing
def read(vStream : str) -> typing.Tuple[int, int]:
    rval = 0
    for idx in range(len(vStream)):
        rval = (rval * 128) + ord(vStream[idx])
        if not ord(vStream[idx]) & 128: return (idx+1,rval)
        rval = rval - 128 + 1
    return (0,rval)

Thoughts

  1. Type annotations are still fairly new to the Python ecosystem as reinforced by the lack of types in the standard library. Typeshed has compiled/developed type signatures for most standard library functions, but this does mean one can't actually type check the code against the most recent implementation of the standard library which can cause disconnects.

  2. Type annotations are still fairly new to the Python ecosystem; particularly non-standard library packages rarely use type annotations and often aren't available in Typeshed. This left the team either developing such annotations from scratch or extending other community efforts (e.g. numpy type extended here).

  3. mypy is still under active development and generally usable, though not exceptionally robust. While type checking sometimes "just worked", developers often found themselves tweaking code patterns or writing new code workarounds to satisfy type handling (e.g. initialize before use, typing.cast)

  4. Code becomes substantially more detailed/entrenched with potentially verbose type annotations

  5. Type checking as a mandatory pre-commit step caught and solved many errors that would have caused run-time failures.

Closing

Adding type checking to non-trivial program was significantly more work than initially expected, but the tools ultimately worked and met our needs for higher reliability. I look forward to future improvements of the Python type checking ecosystem that hopefully make it much easier to add, more robust, and more widespread.