Introduction to the Python development ecosystem

Introduction to the Python development ecosystemAuthor:Mir NazimOriginal article:Python Ecosystem-An IntroductionTranslator:Dccrazyboy

When developers switch from PHP, Ruby, or other development environments to Python, the biggest problem is lack of full understanding of the Python development ecosystem. Developers really want a guide or resource on completing most tasks, regardless of whether the methods used are standard or not.

The content mentioned below basically comes from my website, which stores some basic information about network application development in the Python environment, these materials are prepared for interns, graduate students, and experienced developers who have switched from other platforms to Python development.

This is not a perfect document. My goal is to make it a permanent document. I hope this document can be developed into a detailed tutorial.

Target readers

This is not a grammar book about Python. This tutorial will not teach you how to use Python in a fancy way to make you a Python hacker. By default, you already know the basic usage of Python. If you don't know, don't read it down. Let's take a look at Zed Shaw's free book "Learn Python The Hard Way", which teaches Python beginners.

I don't know if you are using Linux (preferably Ubuntu/Debian) or a Linux-like system. Is it Mao? Because Linux is the most familiar system. Apart from cross-browser compatibility testing, I have never developed it on Windows or Max OS X. The following is a tutorial on how to install Python on different platforms.

• Python 101: Setting up Python on Windows
• Official documentation for Python on Windows
• Official documentation for Python on Mac OS X

Search for the Python installation method that best suits your platform. I strongly asked Stack Overflow.

Version puzzles

Python 2.x is a stable version, and Python 3 is a new version. If you don't care, skip this section and check the Python installation section below.

When you start learning Python, installing a version of 3.x seems to be the first step, but it may not necessarily be what you want.

Currently, Python has two versions under development-2.7.x and 3.x( also known as Python3, Py3k or Python 3000 ). Compared with Python2, Python3 is another language. There are two or more major or small syntax differences between them. Today, Python2.6/2.7 is the most widely installed and applied version. Many mainstream codes and some important packages/frameworks/tools/utilities/modules do not support Python3.

Therefore, the safest choice is to use version 2.x( best version 2.7. If you fully understand Python3, use it again.

Python 3 Wall of Shame lists the compatibility of many packages in Python3. Take a good look at this before deciding to use Python3.

VM Selection

The Python interpreter or Python virtual machine has many implementations. CPython is the most popular implementation. CPython is also a reference interpreter for other virtual machine implementations.

PyPy is a Python interpreter implemented using Python, Jython is an interpreter running on JVM using Java, and IronPython is an interpreter implemented using Microsoft. net clr.

Unless the interpreter is very important, we generally use CPython.

If the above nonsense about the version and virtual machine makes you a headache, you should install CPython 2.7.x. Believe me!

Install Python

In most Linux/Unix systems, Max OS X comes with Python. If no or the version is too low, run the following command to install it:

Ubuntu/Debian and its derivative systems:

$ sudo apt-get install python2.7

Sudo is a Unix-like command that allows users to run programs with the permissions of other users (generally superusers, or root users. For more information, see Wikipedia.

Fedora/Red Hat and similar systems:

sudo yum install python2.7

To use RHEL, you may need to open EPEL Repositories to install it.

From this point on, I will use sudo in the example, and you need to make changes based on your system.

Understanding package

First, you need to understand that Python does not have a default package management facility. In fact, the concept of packages is quite weak in Python.

As you may already know, Python code is organized as a module. A module may consist of a single file containing a function, or a directory containing multiple modules. The difference between a package and a module is very small, and each module can be understood as a package.

What is the difference between a package and a module (if any )? To understand this, you should first understand how Python looks for modules.

Like other programming environments, some functions and classes (such as str, len, and Exception) in Python are available globally (called built-in functions. Otherwise, you need to manually import the data. For example:

>>> import os>>> from os.path import basename, dirname

This package must exist on your machine so that it can be imported by the import Statement. But how does Python know the location of these modules? The location information is automatically set when you install the Python Virtual Machine and depends on your target platform.

You can query the package path in sys. path. The following is the result in my notebook. The running environment is Ubuntu 11.10.

>>> import sys>>> print sys.path['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client', '/usr/lib/python2.7/dist-packages/ubuntuone-client', '/usr/lib/python2.7/dist-packages/ubuntuone-control-panel', '/usr/lib/python2.7/dist-packages/ubuntuone-couch', '/usr/lib/python2.7/dist-packages/ubuntuone-installer', '/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol']

The path of the Python search package is provided here. It starts from the top until a name is found. This indicates that if two different paths contain two packages with the same name, the search will stop when the first name is found, and the search will never go down.

As you guessed, the package search path is easily hijacked. To ensure that Python loads your package first, you need to do the following:

>>> sys.path.insert(0, '/path/to/my/packages')

Although this method is useful in many cases, be careful not to abuse it. Use it only when necessary! Do not abuse it!

The search path of the site module control package. When the Python virtual machine is initialized, it will be sub-imported. For more information, see the official documentation.

PYTHONPATH variable

PYTHONPATH is an environment variable used to add the default package search directory. It can be considered as a special PATH variable for Python. It is just a list of directories that contain Python modules through: splitting (not a Python list similar to sys. path ). It may be like the following:

export PYTHONPATH=/path/to/some/directory:/path/to/another/directory:/path/to/yet/another/directory

Sometimes you may not want to overwrite the existing PYTHONPATH, but just want to add a new directory to the header or tail.

export PYTHONPATH=$PYTHONPATH:/path/to/some/directory    # Appendexport PYTHONPATH=/path/to/some/directory:$PYTHONPATH    # Prepend

PYTHONPATH, sys. path. insert are not perfect. We 'd better not use these methods. Using them, you may be able to solve local development environment problems, but it may not work in other environments. There are many ways to achieve this effect. I will explain it one by one below.

We now understand how to find the installation package path in Python. Now let's go back to the problem. What is the difference between modules and packages? A package is a collection of modules, modules, and submodules. It is usually compressed into a compressed package. This includes 1) Dependency Information 2) Instructions for copying files to the standard package search path. 3) Compile command (if the Code must be compiled before installation ). That's all!

Third-party packages

From the very beginning, if you want to do some practical Python development, you will certainly use some third-party packages.

There are at least three methods for installing third-party packages on Linux.

1. Use the package management system (deb, rpm, etc.) that comes with the System)
2. Various tools developed through the community, such as pip and easy_install
3. Install from source file

These three aspects almost accomplish the same thing. That is, install the dependency and compile the code (if needed). Search for a standard software package containing the module package copy.

Steps 2 and 3 are basically the same on all operating systems. I hope you can find the third-party package installation method on the Stack Overflow.

Where can I find a third-party package?

Before installing third-party packages, find them first. There are several methods:

1. Dedicated release package in your system Package Manager.
2. Python Package Index (or PyPI)
3. A large number of source code servers, such as Launchpad, GitHub, and BitBucket

Use a dedicated release package

Installing the installation package through the Package Manager is as simple as installing other software using the command line or GUI. For example, install simplejson in Ubuntu

$ sudo apt-get install python-simplejson

Install using pip

Easy_install is no longer popular. We will mainly introduce pip, which is a substitute for easy_install.

Pip is a tool used to install and manage Python packages, just like Python Packet Index. Pip is not installed with Python, so we need to install it first. In Linux, the installation is generally as follows:

$ sudo apt-get install python-pip

Before installing any other packages, I always upgrade pip to the latest version through PyPI. the versions in the Ubuntu software library are generally outdated. I upgraded pip myself through pip.

$ sudo pip install pip --upgrade

To install any package, run the pip install package-name command. Therefore, run the following command to install simplejson:

$ sudo pip install simplejson

Deleting a package is also easy.

$ sudo pip uninstall simplejson

In general, pip will automatically install the latest stable version from PyPI, but sometimes we need to install a specific version of the package, because your project may be based on a special version. Therefore, you may need to use the following pip install command:

$ sudo pip install simplejson==2.2.1

We may need to upgrade/downgrade/reinstall the installation package. In this case, you can run the following command:

$ sudo pip install simplejson --upgrade         # Upgrade a package to the latest version from PyPI$ sudo pip install simplejson==2.2.1 --upgrade  # Upgrade/downgrade a package to a given version

Now, if you want to install a package in the development version, which is available in the version control repository, but not in PyPI, what should you do? Pip can handle this situation well, but before that, you need to install the version control repository on your own. In Ubuntu, you can install it as follows:

$ sudo apt-get install git-core mercurial subversion

After the version control warehouse is installed, the installation package from the version control warehouse is as follows:

$ sudo pip install git+http://hostname_or_ip/path/to/git-repo#egg=packagename$ sudo pip install hg+http://hostname_or_ip/path/to/hg-repo#egg=packagename$ sudo pip install svn+http://hostname_or_ip/path/to/svn-repo#egg=packagename

You can also install it from a local repository. Note that the following three slashes are file directories.

$ sudo pip install git+file:///path/to/local/repository

Note that if you use git for installation, you need to use git + git Prefix:

$ sudo pip install git+git://hostname_or_ip/path/to/git-repo#egg=packagename

Now you may be curious about this.EggHow to use it. Now you need to understand that an egg is a compressed Python package that contains source code and metadata. Pip sets up egg information before the installation package. You can find the egg name in the setup. py file of the code repository. Find the setup block and find a field similar to name = "something. It may look like the following code (this code is obtained from simplejson's srtup. py.

setup(name="simplejson", # <--- This is your egg nameversion=VERSION,description=DESCRIPTION,long_description=LONG_DESCRIPTION,classifiers=CLASSIFIERS,author="Bob Ippolito",author_email="bob@redivi.com",url="http://github.com/simplejson/simplejson",license="MIT License",packages=['simplejson', 'simplejson.tests'],platforms=['any'],**kw)

What if there is no setup. py file? How can I find the egg name? We don't need it. Copy the package source code to your project directory, and import it to use it directly.

-- User option

All the above examples install the packages to the system scope. If you use the -- user option during pip install, the package will be installed ~ /. Local directory. On my machine, as shown below:

$ pip install --user markdown2Downloading/unpacking markdown2  Downloading markdown2-1.0.1.19.zip (130Kb): 130Kb downloaded  Running setup.py egg_info for package markdown2Installing collected packages: markdown2  Running setup.py install for markdown2    warning: build_py: byte-compiling is disabled, skipping.    changing mode of build/scripts-2.7/markdown2 from 664 to 775    warning: install_lib: byte-compiling is disabled, skipping.    changing mode of /home/mir/.local/bin/markdown2 to 775Successfully installed markdown2Cleaning up...

Note:Markdown2Installed in the/home/mir/. local/bin/markdown2 directory.

There are many reasons why you don't want to install the package in the system directory. Later, I will explain how to set up an independent Python environment for each project.

Install from source code

Only one command is required from the source code installation package. decompress the package to a directory and then execute the following command.

cd /path/to/package/directorypython setup.py install

Although these installation methods are no different, the pip method is the best. Pip gives you the ability to easily upgrade/downgrade packages, and you have to manually download and decompress the package for manual installation. The manual installation package should make you the last option if all the other options fail (generally not possible ).

Install the package to be compiled

We have learned how to install most packages, but some packages are not introduced yet: Python packages that contain C/C ++ code must be compiled before installation. The best examples of these packages are database adapters and image processing libraries.

Although pip can process the source code for compilation and installation, I personally prefer the package provided by the Package Manager of the release version. It will install the compiled binary file.

If you still want to use pip for installation, the following is what needs to be done on the Ubuntu system.

Compiler tools:

$ sudo apt-get install build-essential

Python Development Environment (header files, etc ):

$ sudo aptitude install python-dev-all

If your system does not have python-dev-all, look at these similar names, such as python-dev and python2.X-dev.

Make sure that you have installed psycopg2 (PostgreSQL RDBMS adapter for Python) and you will need the PostgreSQL development file.

$ sudo aptitude install  postgresql-server-dev-all

After installing these dependencies, you can run pip install.

$ sudo pip install psycopg2

Note that not all packages can be compiled and installed using pip !. However, if you are confident in compilation and installation, or have sufficient experience in installing on your target platform. Then install it manually!

Python Development Environment

Different people prefer to build their own development environments in different ways, but in almost all programming communities, there is always one (or more) development environment that is more acceptable. Although there are no errors in using different development environments, some environment settings are easier to perform convenient tests and some repetitive/templated tasks are performed, makes daily work simple and easy to maintain.

Virtualenv

The most common method in the Python development environment is to use the virtualenv package. Virtualenv is a package used to create an independent Python environment. Now, the question is: why do we need an independent Python environment? To answer this question, allow me to reference virtualenv's own document:

The basic problems we need to deal with are package dependencies, versions, and indirect permissions. Imagine that you have two applications. One application requires version 1 of libfoo, And the other application requires version 2. How can I use these applications at the same time? If you install everything in/usr/lib/python2.7/site-packages (or any standard location on the platform), in this case, you may accidentally upgrade applications that should not be upgraded.

To put it simply, you can create different/independent Python environments for each project. You will install all required software packages for each project into their own environments.

Run the pip command to install virtualenv:

$ sudo pip install virtualenv

After virtualenv is installed, run the following command to create an independent python environment for your project.

$ mkdir my_project_venv$ virtualenv --distribute my_project_venv# The output will something like:New python executable in my_project_venv/bin/pythonInstalling distribute.............................................done.Installing pip.....................done.

What happened above? You have created a folder named my_project_venv to store your new independent Python environment. The -- distribute option enables virtualenv to use the release-based package management system instead of the packages obtained by setuptools. What you need to know now is that the -- distribute option will automatically install pip in the new virtual environment, so you do not need to install it manually. When you become a more experienced Python developer, you will understand the details.

Now let's take a look at the my_project_venv directory and you will see this structure:

# Only the directories and files to be discussed are listed here. | -- bin | -- activate # <-- activation file of virtualenv | -- pip # <-- independent pip of virtualenv | '-- python # <-- a copy of the python Interpreter '-- lib' -- python2.7 # <-- all new packages will be

Run the following command to activate virtualenv:

$ cd my_project_venv$ source bin/activate

After the execution is completed, the prompt may look like this:

(my_project_venv)$ # the virtualenv name prepended to the prompt

Run the deactivate command to exit the virtualenv environment.

(my_project_venv)$ deactivate

Run the following command to better understand the differences between the two. If you have already entered virtualenv, please leave first.

First, let's take a look at the one that will be called if you call the python/pip command.

$ which python/usr/bin/python$ which pip/usr/local/bin/pip

Try again! Open virtualenv this time and see what is different. The following is displayed on my HOST:

$ cd my_project_venv$ source bin/activate(my_project_venv)$ which python/home/mir/my_project_venv/bin/python(my_project_venv)$ which pip/home/mir/my_project_venv/bin/pip

Virtualenv copies the Python executable file, creates some useful scripts, and installs the software packages required by the project, you can install, upgrade, or delete these packages throughout the project lifecycle. It also modifies some search paths, such as PYTHONPATH, to ensure that:

1. When installing the installation packages, they are installed in the active virtualenv instead of the Python path within the system.
2. When importing code, virtualenv takes precedence over the packages installed in the current environment, rather than the packages installed in the system Python directory.

Also, by default, all packages installed within the system are visible to virtualenv. This means that if you install simplejson In Your Python directory, it will be automatically provided to all virtualenvs. This behavior can be changed. virtualenv with the -- no-site-packages option added when creating virtualenv will not read the system package, as shown below:

$ virtualenv my_project_venv --no-site-packages

Virtualenvwrapper

Virtualenvwrapper is a tool built on virtualenv. It allows you to easily create, activate, manage, and destroy virtual environments. Without it, the above operations will be quite troublesome. Run the following command to install virtualenvwrapper.

$ sudo pip install virtualenvwrapper

After installation, You need to configure it. The following is my Configuration:

if [ `id -u` != '0' ]; then  export VIRTUALENV_USE_DISTRIBUTE=1        # <-- Always use pip/distribute  export WORKON_HOME=$HOME/.virtualenvs       # <-- Where all virtualenvs will be stored  source /usr/local/bin/virtualenvwrapper.sh  export PIP_VIRTUALENV_BASE=$WORKON_HOME  export PIP_RESPECT_VIRTUALENV=truefi

To set WORKON_HOME and source/usr/local/bin/virtualenvwrapper. sh, you only need a few lines of code. The other parts are added according to my personal preferences.

Add the above configuration ~ /. Run the following command at the end of bashrc:

$ source ~/.bashrc

If you close all shell windows and labels and then open a new shell window or tag ,~ /. Bashrc will also be executed, and your virtualenvwrapper configuration will be automatically updated. The result is the same as running the preceding command.

To create, activate, disable, and delete a virtual space, run the following command:

$ mkvirtualenv my_project_venv$ workon my_project_venv$ deactivate$ rmvirtualenv my_project_venv

Tab completion is available in virtualenvwrapper ~

Go to the virtualenvwrapper homepage to find