High Performance Python

Reference source: python financial Big Data Analysis chapter eighth

There are several ways to improve performance:

1, cython, for merging Python and c-language static compilation generics

2, ipython.parallel, for parallel execution of code on a local or cluster

3, numexpr, for Fast numerical operation

4. Multiprocessing,python built-in Parallel processing module

5, numba, used to dynamically compile Python code for the CPU

6, numbapro, used to dynamically compile Python code for multicore CPUs and GPUs

To verify the performance differences of the same algorithm on the above implementations, we first define a function for testing performance

defPerf_comp_data (func_list, data_list, rep=3, number=1):     " "Function to compare the performance of different functions. Parameters func_list:list list with function names as strings Data_list:list list with data set names a  s strings rep:int number of repetitions of the whole comparison number:int number ofexecutions for Every function" "     fromTimeitImportRepeat Res_list= {}      forNameinchEnumerate (func_list): stmt= name[1] +'('+ data_list[name[0]] +')'Setup="from __main__ Import"+ name[1] +','+data_list[name[0]] Results= Repeat (stmt=stmt, setup=setup, repeat=rep, number=number ) res_list[name[1]] = Sum (results)/Rep Res_sort= sorted (res_list.items (), key =LambdaItem:item[1])     forIteminchRes_sort:rel= Item[1]/res_sort[0][1]        Print('function:'+ item[0] +', Av. time sec:%9.5f,'% item[1] +'relative:%6.1f'% Rel)

The algorithm for defining execution is as follows

 from Import def F (x):      return ABS (cos (x)) * * 0.5 + sin (2 + 3 * x)

The corresponding mathematical formula is

Generate data as follows

i=500000= Range (i)

The first implementation of F1 is to execute the F function in an internal loop, and then add each calculation result to the list, as follows

def F1 (a):      = []     for in  A:         res.append (f (x))     return Res

of course, there is more than one way to implement this approach, you can use iterators or the Eval function, I myself added to the use of generators and map methods of testing, found that the results have a significant gap, do not know whether science:

Iterator implementation

def F2 (a):      return  for  in a]

Eval implementation

def f3 (a):      ' abs (cos (x)) **0.5+ sin (2 + 3 * x) '     return  for inch

Generator implementation

def F7 (a):      return  for  in A)

Map implementation

def F8 (a):      return map (f, a)

Next is a few implementations of the Narray structure using NumPy

ImportNumPy as NP a_np=Np.arange (i)deff4 (a):return(np.abs (np.cos (a)) * * 0.5 + np.sin (2 + 3 *a))Importnumexpr as NEdeff5 (a): ex='ABS (cos (a)) * * 0.5 + sin (2 + 3 * a)'Ne.set_num_threads (1)     returnne.evaluate (ex)deff6 (a): ex='ABS (cos (a)) * * 0.5 + sin (2 + 3 * a)'Ne.set_num_threads (2)     returnNe.evaluate (ex)

The above F5 and F6 just use different number of processors, can be based on the number of CPU of their own computer to modify, not the bigger the better

Test the following

Func_list = ['F1','F2','f3','f4','f5','f6','F7','F8'] Data_list= ['a_py','a_py','a_py','a_np','a_np','a_np','a_py','a_py']perf_comp_data (func_list, Data_list)

The test results are as follows

function:f8, Av. Time sec:   0.00000,   relative:    1.0function:f7, av. time sec:   0.00001,   relative:    1.7function:f6, av. Time sec:   0.03787,   relative:11982.7function:f5, av. time sec:   0.05838,   relative:18472.4function:f4, Av. Time sec:   0.09711,   relative:30726.8function:f2, av. Time Sec:   0.82343,   relative:260537.0function:f1, av. time sec:   0.92557,   relative:292855.2function:f3, av. Time Sec:  32.80889,   relative:10380938.6

Find the shortest time of f8, adjust the time accuracy and then test again

function:f8, Av. Time sec:0.000002483,   relative:    1.0function:f7, av. time sec:0.000004741,   relative:    1.9function:f5, av. Time sec:0.028068110,   relative:11303.0function:f6, av. time sec:0.031389788,   Relati ve:12640.6function:f4, Av. Time sec:0.053619114,   relative:21592.4function:f1, av. time sec:0.852619225,   RelA tive:343348.7function:f2, Av. Time sec:1.009691877,   relative:406601.7function:f3, av. Time sec:26.035869787,
   
    relative:10484613.6
   

It is found that using map has the highest performance, followed by generators, and the performance of other methods is far worse. But using Narray data at an order of magnitude, the list data using Python is at an order of Magnitude. The principle of the generator is not to generate a complete list, but rather to maintain a next function internally, by iterating through the loop to generate the next element of the implementation of the method, so he does not have to traverse the entire loop during execution, and do not allocate the entire space, it spends time and space is not related to the size of the list, Maps are similar, and other implementations are related to the size of the List.

Not yet finished, to be continued ........ ........

High Performance Python