Machine learning to migrate from Python 2 to Python 3, something you need to be aware of ... __python

compiling | AI Technology Base Camp (rgznai100)

Participation | Lin Yu 眄

Edit | Donna

Python has become the mainstream language in machine learning and other scientific fields. It is not only compatible with a variety of depth learning frameworks, but also includes excellent toolkits and dependency libraries, which enable us to preprocess and visualize data.

According to the latest news, by the end of 2019, NumPy and many other scientific computing kits will stop supporting the Python 2 version, and 2018 years later NumPy all new feature versions will support only Python 3.

To make it easy for beginners to migrate from Python 2 to Python 3, I've collected some Python 3 features that I hope will help.



▌ Use the Pathlib module to better handle the path

Pathlib is the Python 3 default module for processing data paths, which helps us avoid the use of a large number of os.path.joins statements:

From Pathlib import Path
DataSet = ' Wiki_images '

Train_path = datasets_root/dataset/' Train '
Test_path = datasets_root/dataset/' Test '
For Image_path in Train_path.iterdir ():
With Image_path.open () as F: # Note, the Open is a method of path object
# do something and an image

Slide to the left to see the full code

In Python2, we need to implement the concatenation of the paths through the formation of cascading strings. Now with the Pathlib module, data path processing will become more secure, accurate, and more readable.

In addition, Pathlib. Path contains a number of methods, so that Python beginners will no longer need to search for each method:

P.exists ()
P.is_dir ()
P.parts ()
P.with_name (' Sibling.png ') # only change the name, but keep the folder
P.with_suffix ('. jpg ') # Only change the extension, but keep the folder and the name
P.chmod (Mode)
P.rmdir ()

The use of Pathlib will also save you much time. More features please view:

Official Documents-https://docs.python.org/3/library/pathlib.html

Reference Information-https://pymotw.com/3/pathlib/

▌ type hint (hinting) becomes a new member of the Python3

The following is an example of a type hint feature in compiler pycharm:



Python is more than just a scripting language, and today's data flow includes a number of logical steps, each of which includes different frameworks (sometimes including different logic).

The Type Hint toolkit is introduced in Python3 to handle complex and large projects, enabling the machine to better validate the code. Before that, different modules need to use a custom way to specify the type of string in the document (note: Pycharm can convert the old document string to the new type hint).

The following is a simple code example that uses the type hint feature to handle different types of data:

def repeat_each_entry (data):

<blah blah Nobody reads the documentation till
"""
index = numpy.repeat (Numpy.arange (len (data)), 2)
return Data[index]

The above code for multidimensional Numpy.array, astropy. Table and Astropy. Column, Bcolz, Cupy, Mxnet.ndarray and other operations are also applicable.

This code can also be used for pandas. Series operation, but this form is wrong:

Repeat_each_entry (Pandas. Series (data=[0, 1, 2], index=[3, 4, 5]) # returns Series with Nones inside

This is just a two-line code. Therefore, the behavior of complex systems is very difficult to predict, sometimes a function can lead to the whole system error. Therefore, it is helpful for large systems to know exactly which types of methods and to issue errors when these types of methods do not get the corresponding parameters.

def repeat_each_entry (Data:union[numpy.ndarray, Bcolz.carray]):

If you have a great code base, a type hint tool such as mypy will probably become part of the integration process for a large project. Unfortunately, the type hint feature is not yet powerful enough to give a hint for this fine-grained type of ndarrays/tensors. Perhaps, in the near future, we can have such a comprehensive type hint tool, which will become a powerful feature in the field of data science.

▌ from type prompt (run before) to type check (run time)

By default, the comment for a function has no effect on the operation of the code, it just helps you point out what each piece of code is going to do.

In the code run phase, many times the type hint tool does not work. This situation allows you to use tools such as enforce to enforce the type checking of your code and also to help you debug your code.

 
 @enforce. Runtime_validation 
 def foo (text:str)-> None: 
    print (text) 
 foo (' Hi ') # OK 
 Foo (5)    # fails 
 @enforce. Runtime_validation 
 def any2