The rapid development of the post-modern Python world

The rapid development of the post-modern Python world

If modern python has an iconic feature, it is simply that Python is increasingly vague about its own definition. Many projects over the past few years have greatly expanded Python and rebuilt the meaning of "python" itself.

At the same time, the emergence of new technologies has encroached on Python's share and brought new advantages:

1. Go-(Goroutines, Types, Interfaces)
2. Rust-(Traits, speed, Types)
3. Julia-(speed, Types, multiple Dispatch)
4. Scala-(Traits, speed, Types)
5. Clojure (Metaprogramming, DSLs, protocols)

This is a short guide to Python's response to these new technologies, libraries, and models:

Meta-programming

Macropy is a meta-programming framework that provides a variety of syntactic constructs that extend the Python AST by compiling modern language elements into standard Python code. For example, we can measure the type of algebraic data:

From macropy.case_classes import Case@caseclass Nil ():    pass@caseclass Cons (x, xs):    passcons (1, Cons (2, Cons (3, Nil ())))

Then the pattern matches the type of the declaration:

def reduce (OP, my_list):    switch (my_list):        if Cons (x, Nil ()):            return x        elif Cons (x, xs):            Return op (x, Reduce (OP, xs))

The vanishing part is still a meta-programming system along the CAMLP4 route, extensible stage . However, Mython provides a PGEN2 parsing framework that defines a new syntax for reference blocks to solve this problem.

MY[NAMEDTUPLEDEF] Point (x, y): Passmy[c]:    Add (int x, int y) {        return x + y;    } Print "Regular Python"

Type

Python is a dynamic type language and is proud of it. I certainly don't want to stir up the type of jihad, but there are certainly universities that believe that building a reliable application requires more than just using unit testing. The Benjamin Pierce defines the type system as follows:

... An easy-to-handle syntax that proves the absence of a specific program behavior by classifying the phrase according to the type of the computed value

The focus is on proving the properties of the running space, and the running space for all program behaviors instead of simply listing the space for a limited number of cases. Full-static typing is confusing for Python to be correct, but there is definitely a more appropriate scenario between overly dynamic types and static type guarantees. MyPy project has found a good balance that allows both typed and no-type code to be stored in the language's hyper-focus at the same time. For example:

def simple_typed (X:int, y:int), int:    return x + ysimple_typed (1, 2)     # type-checks succesfully# fails:arg Ument 2 to "simple_typed" have incompatible type # "float" simple_typed (1, 2.0) # fails:argument 2 to "Simple_typed" have Inc ompatible type "str" simple_typed (1, "foo")

Of course, there is not much use for C language. So we're not limited to simple types of functions, and parameter types have generics, pointer types, and a variety of built-in type-level functions.

From typing import Iterator, Typevar, Generic, Function, Listt = Typevar (' T ') def example_typed (X:iterator[int]), It ERATOR[STR]: For    i in x:        yield str (i) def example_generic (X:iterator[t]), Iterator[t]: for    i in x:        y Ield I

We can also define more advanced generic structures such as functor and cell

A = Typevar (' a ') b = Typevar (' B ') class Functor (Generic[a]):    def __init__ (self, xs:list[a), None:        self._sto Rage = XS    def iter (self)-iterator[a]:        return iter (self._storage) def fmap (F:function[[a], b], xs:functor[ A]) Functor[b]:    return Functor ([F (x) for x in Xs.iter ()]) class Monad (Generic[a]):    def __init__ (self, val:a )-None:        self.val = Valclass Idmonad (Monad):    # Monad M + A-m a    def unit (self, x:a), Mona D[B]:        return Idmonad (x)    # Monad m + m A--B    def bind (self, x:monad[a], f:f Unction[[a], Monad[b]]), Monad[b]:        return F (x.val)    # Monad m = + m (m a), m a    def join (self, X : Monad[monad[a]]), Monad[a]:        return X.val

Speed

The most important recent development of the "high-performance" Python is the higher-level dataframe container that the Pandas library provides. Pandas mixes a variety of Python operations, uses numpy for some operations, uses Cython for some, and even uses C for some internal hash tables. Panda's non-dogmatic approach to the underlying architecture has made it a standard repository in the field of data analysis. Pandas's development embodies many of the things that make the numerical Python ecosystem successful.

In [1]: From pandas import Dataframein [2]: Titanic = dataframe.from_csv (' titanic.csv ') in [3]: Titanic.groupby (' Pclass '). Survived.mean () pclass1st       0.6191952nd       0.4296033rd       0.255289name:survived

However, the most recent attempt to improve Python performance is to use the LLVM compiler to selectively compile some Python code snippets for native code. Although different technologies are implemented in different ways, most are similar to the following:

1. Add adorners such as @jit or @compile on the function.
2. The AST or bytecode of the function is extracted into the compiler pipeline, mapped to the internal AST in the pipeline, and given a specific set of input types that determines how the given function logic is reduced to machine code.
3. The compiled function is called with a set of types, the parameters are checked, and the code is generated under the given type. The generated code, along with the parameters, is cached so that subsequent calls are distributed directly to the local code.

These projects have increased interest in Python language technology and Llvmpy project development, and I suspect that llvmpy is more important in Python's history than a specific JIT compiler.

The simplest example (from an excellent kaleidescope tutorial) is to create a simple local multiply -add function, and then invoke it by unpacking three python integers:

Import Llvm.core as Lcimport llvm.ee as Lemod = LC. Module.new (' mymodule ') I32 = LC. Type.int (+) Funty = LC. Type.function (LC. Type.int (), [I32, I32, i32]) MADD = LC. Function.new (mod, funty, "multiply") x = Madd.args[0]y = Madd.args[1]z = Madd.args[2]block = Madd.append_basic_block ("L1" ) Builder = LC. Builder.new (block) x0 = Builder.mul (x, y) x1 = Builder. Add (x0, z) builder.ret (x1) Print Modtm = Le. Targetmachine.new (features= ", Cm=le. Cm_jitdefault) EB = Le. Enginebuilder.new (mod) engine = Eb.create (tm) ax = le. Genericvalue.int (i32, 1024x768) ay = le. Genericvalue.int (I32, 1024x768) az = le. Genericvalue.int (I32, 1024x768) ret = engine.run_function (MADD, [Ax, Ay, AZ]) print ret.as_int () print mod.to_native_assembly ()

The code compiled above generates the following LLVM IR.

Define I32 @multiply (I32, I32, i32) {L1:  %3 = Mul i32%0,%1  add i32%3,%2  ret i32%4}

Although this example is not intuitive, you can generate a fast JIT ' d function that integrates well with libraries such as NumPy, making the data a chunk of the unpacking memory.

Interface

Decomposing the behavior into composable units, rather than explicit inheritance hierarchies, is a python that does not solve the problem and often leads to nightmare-like complex use of mixin. However, this problem can be mitigated by emulating statically defined interfaces using the ABC module.

Import Heapqimport collectionsclass Heap (collections. sized):   def __init__ (self, initial=none, Key=lambda x:x):       Self.key = key       If initial:           self._data = [(Key (item), item) for item in initial]           heapq.heapify (self._data)       else:           self._data = []   def pops (self): C9/>return Heapq.heappop (Self._data) [1]   def push (self, item):       Heapq.heappush (Self._data, (Self.key (item) , item))   def len (self):       return len (self._data)

For example, create an equivalence class that enables an instance of all classes to implement the EQ () method. We can do this:

from ABC import Abcmeta, Abstractmethodclass Eq (object            ): __metaclass__ = Abcmeta @classmethod def __subclasshook__ (cls, C): If CLS is Eq:                        For B in c.__mro__: If "eq" in b.__dict__: if b.__dict__["EQ"]: Return True break return notimplementeddef eq (A, B): If Isinstance (A, eq) and Isinstance (b, Eq) and type (a) = = Type (b): return A.eq (b) else:raise Notimplementederrorclass Foo (object): Def E  Q (Self, other): Return Trueclass Fizz (Foo): Passclass Bar (object): Def __init__ (Self, val): Self.val = Val def eq (self, Other): return self.val = = Other.valprint eq (foo (), foo ()) Print EQ (bar (1), bar (1)) Print eq (foo (), Bar (1)) Print eq (Foo (), Fizz ())  

Then extend this type of interface concept to multi-parameter functions, making query __dict__ more and more likely to occur, in the case of a combination of fragile. The key to the problem is to decompose all things into a single type of interface, when what we really want is a declaration that covers a set of multi-type interfaces. This disadvantage in OOP is the key to expression problems.

Languages such as Scala, Haskell, and rust provide a solution to this problem in the form of trait and typeclass. Haskell, for example, can automatically derive differential equations for all types of cross products.

Instance (floating A, Eq a) = Floating (Dif a) where    pi               = c PI    exp (c x)        = C (exp x)    exp (D x x ') c6/>= r where r = d (exp x) (x ' * r)    log (c x)        = C (log x)    log [email protected] (d x x ')   = d (log x) (x ' /p)    sqrt (c x)       = C (sqrt x)    sqrt (D x x ')    = r where r = D (sqrt x) (x '/(2 * R))

Asynchronous programming

Under this theme, we still have a lot of mend solutions that solve some of the problems, but introduce a whole set of limitations and patterns that run counter to regular Python. Gevent maintains Python's own consistency by splicing the underlying C stack. The generated API is elegant, but makes inference control flows and exceptions very complex.

Import geventdef foo ():    print (' Running in foo ')    gevent.sleep (0)    switch to foo again ') def Bar ():    Print (' Explicit context to bar '),    gevent.sleep (0)    print (' implicit context switch back to bar ') Gevent.joinall ([ C7/>gevent.spawn (foo),    gevent.spawn (bar),])

The control flow is shown below:

By Mend (monkey-patching), which is quite graceful for the standard library, we can mimic the Erlang-style actor behavior with asynchronous entry points and internal states:

Import geventfrom gevent.queue import queuefrom simplexmlrpcserver import Simplexmlrpcserverclass Actor (object):    _ Export = [        ' push ',    ]    address:        self.queue = Queue ()        Self._serv = Simplexmlrpcserver (Address, Allow_none=true, logrequests=false)        self.address = address for        name in Self._export:            self._serv.register _function (GetAttr (self, name))    def push (self, Thing):        self.queue.put (thing)    def poll (self):        While true:            print (Self.queue.get ())    def periodic (self): while        true:            print (' PING ')            Gevent.sleep (5)    def serve_forever (self):        gevent.spawn (self.periodic)        gevent.spawn (Self.poll)        Self._serv.serve_forever () def main (): From    gevent.monkey import patch_all    patch_all ()    serve = Actor ((' ', 8000))    serve.serve_forever ()

DSLs

The Z3 project is an extension API embedded in the Python object layer. Solving the N Queen problem with an example of Z3 can be described as a Python expression and an extended SMT to solve the problem:

From Z3 import *q = [Int (' q_%i '% (i + 1)) for I in range (8)]# per queen is in a column {1, ... 8}val_c = [and (1 <= q[i], Q[i] <= 8) for I in range (8)]# at the most one queen per Columncol_c = [Distinct (Q)]# Dia  Gonal constraintdiag_c = [If (i = = J,               True, and               (Q[i]-q[j]! = i-j, Q[i]-q[j]! = j-i)) for            I in range (8) For j in range (i)]solve (Val_c + Col_c + diag_c)

Other projects in Theano,sympy,pyspark use the overloaded operators based on Python expressions in a large number of ways.

From sympy import symbolfrom sympy.logic.inference Import Satisfiablex = symbol (' x ') y = symbol (' y ') satisfiable ((x | y) &am P (x | ~y) & (~x | y))

The rapid development of the post-modern Python world