Introduction to the Python eco-environment

Introduction to the Python eco-environment

Author:	Mir Nazim
Original:	Python Ecosystem-an Introduction
Translator:	Dccrazyboy
Original translation:	Introduction to the Python eco-environment

When developers transition from Php,ruby or other development environments to Python, the biggest problem is the lack of a full understanding of the eco-environment developed by Python. Developers are very interested in getting a guide or resource for completing most tasks, regardless of whether the method used is prescriptive.

What follows is basically from my website, which stores some basic information about Web application development in a Python environment, prepared for interns, graduate students and experienced developers who have been transferred from other platforms to Python development.

This is not a perfect document, my goal is to make it a permanent document , I hope this document can be developed into a detailed tutorial.

Target Audience

This is not a grammar book about Python. This tutorial will not teach you fancy python usage to make you a python hacker. I default you already understand the basic usage of Python. If you don't know, don't look down. Take a look at Zed Shaw's free teaching of Python's introductory book, "Learn Python, the hard way."

I don't know if you're using Linux (preferably Ubuntu/debian) or a Linux-like system. For Mao? Because Linux is the system I know best. In addition to testing cross-browser compatibility, I have never developed on Windows or Max OS x. Here's a tutorial on how to install Python on different platforms.

• Python 101:setting up Python on Windows
• Official documentation for Python on Windows
• Official documentation for Python on MAC OS X

Search for the Python installation method that best suits your platform. I strongly set up to ask Stack Overflow.

Version of Confusion

Python 2.x is a stable version and Python 3 is a new version. If you don't care, skip this section directly and see the Python installation section below.

When you start learning Python, the 3.x version looks like the first step, but it may not necessarily be the one you want.

Python now has two versions in development -2.7.x and 3.x (also known as python3,py3k or Python 3000). Python3 is a different language than Python2. There are either large or small grammatical differences between them. Today python2.6/2.7 is the most widely installed and applied version. Many of the mainstream code and some important packages/frameworks/tools/utilities/modules do not support Python3.

Therefore, the safest option is to use the 2.x (2.7 best) version. If you fully understand Python3 's words, use it again.

Python 3 Wall of shame lists the compatibility of many packages in Python3. Take a good look at this before you decide to use Python3.

VM Pick

There are many implementations of Python interpreter or Python virtual machine, CPython is the most mainstream implementation. CPython is also a reference interpreter for other virtual machine implementations.

PyPy is a Python interpreter implemented with Python, and Jython is an interpreter that runs on the JVM with Jave, IronPython is an interpreter implemented with the Microsoft. NET CLR.

Unless the choice of interpreter is very important, we generally use CPython.

If the above-mentioned nonsense about versions and virtual machines gives you headaches, then install the 2.7.x version of CPython, trust me!

Install Python

Most of the Linux/unix system's Max OS X comes with Python. If not, or if the version is too low, you can install it by using the following command:

Ubuntu/debian and its derivative systems:

$ sudo apt-get install python2.7

sudo is a command that is a UNIX-like system that allows a user to run a program with the privileges of another user (typically superuser, or root). Please refer to Wikipedia for more information.

fedora/red hat and similar systems:

sudo yum install python2.7

With Rhel you may need to open Epel repositories to install.

From there, I'll use sudo in the example, and you'll need to make changes based on your system.

Understanding Packages

The first thing you need to understand is that Python does not have a default package management facility. In fact, the concept of a package is quite weak in python.

As you may already know, Python code is organized into modules. A module may consist of a single file that contains a function, or a directory that contains multiple modules. The difference between packages and modules is very small, and each module can be understood as a package.

So what is the difference between a package and a module (if any)? To understand this, you should first understand how Python looks for modules.

As with other programming environments, some functions and classes in Python (such as str,len,exception, etc.) are available globally (called Built-in functions). Others need to be entered via manual import . For example:

>>> Import os>>> from Os.path import basename, dirname

This package must exist on your machine so that it can be imported by import statement. But how does Python know where these modules are located? These location information is automatically set up when you install the Python virtual machine and depends on your target platform.

The path to the package can be queried in Sys.path. Here is the result in my notebook, the operating environment is Ubuntu 11.10.

>>> Import sys>>> Print sys.path[', '/usr/lib/python2.7 ', '/usr/lib/python2.7/plat-linux2 ', '/usr/ Lib/python2.7/lib-tk ', '/usr/lib/python2.7/lib-old ', '/usr/lib/python2.7/lib-dynload ', '/usr/local/lib/python2.7/ Dist-packages ', '/usr/lib/python2.7/dist-packages ', '/usr/lib/python2.7/dist-packages/pil ', '/usr/lib/python2.7/ dist-packages/gst-0.10 ', '/usr/lib/python2.7/dist-packages/gtk-2.0 ', '/usr/lib/pymodules/python2.7 ', '/usr/lib/ Python2.7/dist-packages/ubuntu-sso-client ', '/usr/lib/python2.7/dist-packages/ubuntuone-client ', '/usr/lib/ Python2.7/dist-packages/ubuntuone-control-panel ', '/usr/lib/python2.7/dist-packages/ubuntuone-couch ', '/usr/lib/ Python2.7/dist-packages/ubuntuone-installer ', '/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol ']

The path to the Python search package is given here. It will start at the top and find a name that matches. This indicates that if two different paths contain two packages with the same name, the search will stop when the first name is found, and will never look down.

As you can guess, the package search path is easily hijacked, in order to ensure that Python first loads your package, the following is required:

>>> sys.path.insert (0, '/path/to/my/packages ')

Although this method is useful in many cases, be careful not to misuse it. use only when necessary! Do not abuse!

The site module controls the search path for the package. When the Python virtual machine is initialized, it is CBD with the same import. If you would like more information, please see the official documentation.

Pythonpath variable

PYTHONPATH is an environment variable that is used to increase the default package search directory. You can think of it as a special PATH variable for python. It's just a pass : split, containing a list of Python module catalogs (not a Python list similar to Sys.path ). It may be similar to the following:

Export Pythonpath=/path/to/some/directory:/path/to/another/directory:/path/to/yet/another/directory

Sometimes you may not want to overwrite the existing PYTHONPATH , but just want to add a new directory to the head or tail.

Export pythonpath= $PYTHONPATH:/path/to/some/directory    # Appendexport pythonpath=/path/to/some/directory:$ PYTHONPATH    # prepend

PYTHONPATH , Sys.path.insert These methods are not perfect, and we'd better not use them. Using them, you may be able to solve the local development environment problem, but it may not work in another environment. There are many ways to achieve this effect, which I will elaborate on below.

We now understand how Python finds the installed package path, and now let's go back to the beginning of that problem. What is the difference between a module and a package? A package is a collection of modules or modules/submodules that are typically compressed into a compressed package. It contains 1) dependency information 2) the instructions to copy the file to the standard package search path. 3) Compile instructions (if the code must be compiled before installation). These are the things!

Third-party Packages

From the beginning, if you want to do some actual Python development, you will definitely use some third-party packages.

There are at least 3 ways to install third-party packages on Linux systems.

1. Use the system's own package management system (deb, RPM, etc.)
2. Various tools developed through the community, such as Pip , easy_install , etc.
3. Installing from a source file

These three aspects, almost accomplish the same thing. That is, install dependencies, compile the code (if necessary), and search the location for a standard package that contains a package copy of the module.

The second and third steps are basically the same on all operating systems. I hope again that you can find the third-party package installation method for your platform on the Stack Overflow.

Where do I find a third-party package?

Before you install a third-party package, you first need to find them. Here are a few methods:

1. The release-specific package in your System Package Manager.
2. Python Package Index (or PyPI)
3. Large number of source code servers, such as Launchpad, GitHub, BitBucket, etc.

Installing through a release-specific package

Installing packages through Package Manager installs packages as simple as using the command line or GUI to install other software. For example, installing Simplejson in Ubuntu

$ sudo apt-get install Python-simplejson

Install via PIP

Easy_install gradually ceased to be popular. We will mainly introduce Pip, which is a substitute for easy_install.

Pip is a tool for installing and managing Python packages, just like the Python Packet index. PIP is not installed with Python, so we need to install it first. Linux, typically installed like this:

$ sudo apt-get install Python-pip

Before installing any other packages, I always upgrade the PIP to the latest version of the Ubuntu Software Library by PyPI, which is generally behind the PyPI version. I upgraded PIP to it by PIP itself.

$ sudo pip install pip--upgrade

Now if you need to install any packages, you can run the pip install package-name command. So you can install Simplejsonwith the following command:

$ sudo pip install Simplejson

Deleting a package is also straightforward.

$ sudo pip uninstall Simplejson

In general, PIP will automatically install the latest stable version from PyPI, but sometimes we need to install a specific version of the package because your project may be based on a special version. Therefore you may need to use the pip install command similar to the following:

$ sudo pip install simplejson==2.2.1

We may need to upgrade/downgrade/reinstall the package. This can be done with the following command:

$ sudo pip install Simplejson--upgrade         # Upgrade a package to the latest version from pypi$ sudo pip install Simplejs on==2.2.1--upgrade  # Upgrade/downgrade A package to a given version

Now, if you want to install a developed version of the package, it's in the version control repository, but what about the PyPI? pip can handle this very well, but before you do, you need to install this version control repository yourself. Under Ubuntu, you can install the following:

$ sudo apt-get install Git-core mercurial Subversion

After you install the version control warehouse, the installation package from the version control warehouse is as follows:

$ sudo pip install git+http://hostname_or_ip/path/to/git-repo#egg=packagename$ sudo pip install hg+http://hostname_or_ ip/path/to/hg-repo#egg=packagename$ sudo pip install Svn+http://hostname_or_ip/path/to/svn-repo#egg=packagename

You can also install the same simple from the local warehouse, note that the following three slash is the file directory.

$ sudo pip install Git+file:///path/to/local/repository

One thing to note is that if you install using the git protocol, you need to use the git+git prefix:

$ sudo pip install Git+git://hostname_or_ip/path/to/git-repo#egg=packagename

Now you may wonder how these egg will be used. Now what you need to understand is that an egg is a compressed Python package that contains the source code and some meta data. pip establishes the egg information before the package is installed. You can find the egg name in the setup.py file in the code warehouse. Locate the Setup block and find a field similar to name= "something" . It may look like the following code (this code is obtained from the Simplejson srtup.py ).

Setup (name= "Simplejson", # <---this is your egg nameversion=version,description=description,long_description=long _description,classifiers=classifiers,author= "Bob Ippolito", author_email= "[email protected]", url= "/HTTP/ Github.com/simplejson/simplejson ", license=" MIT license ", packages=[' Simplejson ', ' simplejson.tests '],platforms=[' Any '],**kw)

What if there is no setup.py file? How do I find the egg name? Actually, we don't need to. Copy the package source to your project directory, and then import it to use it directly.

--user Options

All of the above example installs the package to the system scope. If you use the --user option at pip install , the package will be installed in the ~/.local directory. On my machine, as shown below:

$ pip Install--user markdown2downloading/unpacking markdown2  downloading Markdown2-1.0.1.19.zip (130Kb): 130Kb Downloaded  Running setup.py egg_info for package markdown2installing collected packages:markdown2  Running setup.py Install for MARKDOWN2    warning:build_py:byte-compiling is disabled, skipping.    Changing mode of BUILD/SCRIPTS-2.7/MARKDOWN2 from 664 to 775    warning:install_lib:byte-compiling are disabled, Skippin G.    Changing mode of/home/mir/.local/bin/markdown2 to 775Successfully installed markdown2cleaning up ...

Note: The markdown2 is installed in the /home/mir/.local/bin/markdown2 directory.

There are a number of reasons why you do not want to install the package into the system directory. I'll explain later how to set up a separate Python environment for each project.

Install from source

Installing the package from the source only requires a command, unzip the package into a directory, and then execute the following command.

Cd/path/to/package/directorypython setup.py Install

Although these installation methods are not different, the pip approach is the best. pip gives you the ability to easily upgrade/downgrade packages, while manual installation you will have to manually download, unzip the installation. Manually installing the package should make your last choice if all else fails (generally unlikely).

Install packages that need to be compiled

We now have a look at the installation methods for most packages, but some packages are not covered: Python packages that contain C + + code, which need to be compiled before installation. The best examples of these packages are database adapters, image processing libraries, and so on.

Although Pip can handle the source code of the compiled installation, I personally prefer to use the package provided by the release version of the package manager. It will install a compiled binary file.

If you still want to install with Pip , here's what you need to do on your Ubuntu system.

Compiler-related tools:

$ sudo apt-get install build-essential

Python development environment (header file, etc.):

$ sudo aptitude install Python-dev-all

If your system is not Python-dev-all , look at these similar names Python-dev , python2. X-dev and so on.

Make sure you have installed psycopg2 (PostgreSQL RDBMS Adapter for Python) and you will need the PostgreSQL development files.

$ sudo aptitude install  Postgresql-server-dev-all

Once you have completed these dependent installations, you will be able to run pip install .

$ sudo pip install PSYCOPG2

It is also important to note that not all packages can be installed through PIP compilation! . But if you are confident about compiling the installation, or have enough experience with how to install on your target platform. Then be bold to install it manually!

Python development environment

Different people like to build their own development environment in different ways, but in almost all programming communities, there is always one (or more) development environment that makes it easier for people to accept. There are no errors in using different development environments, but some environment settings make it easier to test easily and do repetitive/templated tasks that make daily routines simple and easy to maintain.

Virtualenv

The most common method in a python development environment is to use the VIRTUALENV package. Virtualenv is a package that is used to create a standalone Python environment. Now, there's the question: Why do we need a standalone Python environment? To answer this question, allow me to cite Virtualenv's own documentation:

The

basic problem that we need to deal with is the dependency, version, and indirect permissions issues of the package. Imagine that you have two apps, one app requires Libfoo version 1, and another app requires version 2. How can I use these applications at the same time? If you install everything to the/usr/lib/python2.7/site-packages (or any platform's standard location) in this case, you may accidentally upgrade an application that should not be upgraded.

Simply put, you can create a different/separate Python environment for each project, and you will install all the required packages into their own separate environments for each project.

To install virtualenv using the pip command:

$ sudo pip install virtualenv

Once the virtualenv is installed, you can create a separate Python environment for your project by running the following command.

$ mkdir my_project_venv$ virtualenv--distribute my_project_venv# The output would something like:new python executable in My_project_venv/bin/pythoninstalling Distribute.............................................done. Installing Pip.....................done.

What happened up there? You created a folder my_project_venv to store your new standalone Python environment. The --distribute option enables virtualenv to use the new release-based package management system rather than the setuptools -acquired package. All you need to know now is that the --distribute option automatically installs pipin the new virtual environment, so you don't need to install it manually. When you become a more experienced Python developer, you will understand the details.

Now look at the my_project_venv directory and you will see this structure:

# only the directories and files that will be discussed are listed here. --bin|   | --Activate  # <--This virtualenv's activation file |   | --Pip       # <--This virtualenv's independent pip|   '--    a copy of Python # <--Python interpreter '--lib    '--python2.7 # <--All the new packages will be present

Activate this virtualenv by using the following command:

$ cd my_project_venv$ Source Bin/activate

After execution, the hint may look like this:

(my_project_venv) $ # The virtualenv name prepended to the prompt

Leave the VIRTUALENV environment with the deactivate command

(my_project_venv) $ deactivate

Run the following command to better understand the difference between the two, if you have entered the virtualenv please leave first.

First let's see if it calls the PYTHON/PIP command and it calls that one.

$ which python/usr/bin/python$ which PIP/USR/LOCAL/BIN/PIP

One more time! Open the Virtualenv this time and see what's different. The following is shown on my machine:

$ cd my_project_venv$ source Bin/activate (my_project_venv) $ which python/home/mir/my_project_venv/bin/python (my_ PROJECT_VENV) $ which PIP/HOME/MIR/MY_PROJECT_VENV/BIN/PIP

Virtualenv copies a copy of the Python executable file, creates useful scripts and installs the required packages for the project, and you can install/upgrade/remove them throughout the project's lifecycle. It also modifies some search paths, such as Pythonpath, to ensure that:

1. When packages are installed, they are installed in the currently active virtualenv, not in the system-wide Python path
2. When the import code occurs, VIRTUALENV will take precedence over the packages installed in the environment, rather than the packages installed in the System Python directory.

It is also important that, by default, all system-wide packages are visible to virtualenv. This means that if you install Simplejson in your system python directory, it will automatically be available to all Virtualenvs for use. This behavior can be changed, and the virtualenv that adds the --no-site-packages option when creating virtualenv will not read the system package, as follows:

$ virtualenv my_project_venv--no-site-packages

Virtualenvwrapper

Virtualenvwrapper is a tool built on virtualenv that makes it easy to create/activate/manage/destroy virtual environments without which it can be quite cumbersome to do the above. You can install Virtualenvwrapper with the following command.

$ sudo pip install Virtualenvwrapper

After installation, you need to configure it. Here is my configuration:

If [' id-u '! = ' 0 ']; Then  export virtualenv_use_distribute=1        # <--always use pip/distribute  export workon_home= $HOME/. Virtualenvs       # <--Where all Virtualenvs'll be stored  source/usr/local/bin/virtualenvwrapper.sh  Export pip_virtualenv_base= $WORKON _home  export Pip_respect_virtualenv=truefi

Setting workon_home and source/usr/local/bin/virtualenvwrapper.sh only requires a few lines of code, and the rest is added to my personal liking.

Add the above configuration to the end of ~/.BASHRC , and then run the following command one time:

$ source ~/.BASHRC

If you close all shell windows and tabs and then open a new shell window or tag, ~/.BASHRC will also be executed, and your virtualenvwrapper configuration will be updated automatically. The effect is the same as executing the above command.

New/Active/close/delete virtual space requires the following command:

$ mkvirtualenv my_project_venv$ Workon my_project_venv$ deactivate$ rmvirtualenv my_project_venv

tab completion is available in Virtualenvwrapper Oh ~

Go to the Virtualenvwrapper home page to find out more about it.

Dependency management via Pip and virtualenv

pip combined with virtualenv can provide basic dependency management for your project.

You can view the currently installed package version with the pip Freeze command. Here is the version of the package I used to write a blog:

$ pip freeze-ljinja2==2.6pyyaml==3.10pygments==1.4distribute==0.6.19markdown2==1.0.1.19

Note the- l option, which tells Pip to export only the packages installed in the currently active virtual space, excluding those packages that are installed in the global space.

You can save the results to a file and add it to your version control system.

$ pip freeze-l  > Requirements.txt

pip can also install packages from a file that contains the pip Freeze command.

$ pip Install-r requirements.txt

Other important tools

Although we discussed the basic Python version, virtual machine and package management, we still need some special tools to accomplish the task in our daily work. Although I can't tell the details of every tool, I try to give you a basic overview.

Editor

There are a number of good compilers for Python programming. For me personally, I prefer vim, but here I don't want to provoke an editor war .

There are plenty of Python-supported editors, such as: Vim/gvim, Emacs, gedit on Gnome, Kate on KDE, scribes, ActiveState Komodo edit/ide, Wingware's wing IDE, JetBrains on Pycharm, Eclipse's Pydev plugin. In addition, there are others, but these seem to be the most popular. You can choose whichever is best for you.

Pyflakes: Source Detection Tool

Pyflakes is a tool for checking Python source code and finding errors through textual analysis. It can detect syntax errors, and (partially) logical errors, imported but unused modules, variables that are used only once, and so on.

You can install via pip :

$ pip Install Pyflakes

Invoking it at the command line, the parameter fills the source file name as follows:

$ pyflakes filename.py

Pyflakes can also be integrated into the editor. This is what it looks like in my vim. Note the red wavy line:

Go to stack overflow yourself to find out how to add Pyflakes to the editor.

Pyflakes official website

Requests: an HTTP Library

Requests is a library that makes it easy to handle HTTP requests.

It is also installed by Pip :

$ pip Install requests

Here is an example:

>>> Import requests>>> r = Requests.get (' https://api.github.com ', auth= (' user ', ' pass ') >> > r.status_code204>>> r.headers[' content-type '] ' Application/json ' >>> r.content ...

Requests documentation

Flask: A micro-framework for Web development

Flask is a micro-framework based on Werkzeug and JINJA2.

Install via pip :

$ pip Install Flask

This is a simple example:

From flask Import Flaskapp = Flask (__name__) @app. Route ("/") def hello ():    return "Hello world!" if __name__ = = "__main__":    App.run ()

You can run it this way:

$ python hello.py * Running on Http://localhost:5000/

Flask Official Website

Django: A full-stack web development framework

Django is a full-stack web development framework. It provides orm,http libraries, forms processing, XSS filtering, templates, and other features.

Install via pip :

$ pip Install Django

Find more information on the Django website.

Fabric: Simplifying the use of SSH and deploying system administrator tasks

Fabric is a command-line tool used to simplify the use of the program for SSH or to run system administrator tasks. It provides a basic suite that can perform local or remote shell commands (normal or via sudo), upload/download files, and other accessibility features such as prompting the user for input, or aborting the action performed.

Can be installed via PIP :

$ pip Install Fabric

This is a task written through fabric:

From Fabric.api import Rundef host_type ():    run (' uname-s ')

You can execute this command on one or more servers:

$ fab-h localhost host_type[localhost] run:uname-s[localhost] out:LinuxDone.Disconnecting from localhost ... done.

Fabric website

SciPy: A Scientific Computing library

If your work involves scientific and numerical calculations, SCIPY is an essential tool for you.

SCIPY website This introduction:

scipy (read "sigh Pie") is an open source library of mathematics, science, and engineering. This is also a great library for Python in terms of numerical computation. The SciPy library relies on NumPy, which provides a convenient and fast n-dimensional matrix operation. The SCIPY library is compatible with matrices in the NumPy and provides many user-friendly and efficient numerical procedures, such as the integration and optimization of numerical values. At the same time, they run on all popular operating systems, are fast to install, and are free of charge. NumPy and scipy are easy to use, but powerful enough to meet some of the world's leading scientists and engineers. If you need to process numbers, display, or publish results on your computer, try SciPy!

SciPy

PEP 8:python Style Guide

Although it is not a software tool in itself, it is an important resource for Python.

PEP 8 is a document that describes the coding conventions for basic libraries in a Python distribution. The document's hope is to contract the code layout, variables, classes, and functions of Python developers around the world. Make sure you understand it and follow it. He will bring a lot of benefits to future Python use.

PEP 0008

The powerful Python standard library

Python's standard library covers a wide variety of functions. The library contains built-in modules (written in c) that provide access to the system's functions, such as file I/O, and a large number of Python-written modules that provide standardized solutions to solve many of the problems that arise in everyday programming. These modules are explicitly pumped out of the platform's details, providing platform-agnostic APIs designed to enhance the cross-platform portability of Python programs.

Recommended Reading

David Goodger's Code like a pythonista:idiomatic python contains a number of practical python examples and techniques.

Doug Hellmann's Python module of the Week series focuses on building a repository of sample code that uses the modules in the Python standard library.

Parting

So far, this tutorial is only about donuts. The Python world has a lot of tools, libraries and software to help you do your job. But you need to take the time to find them!

Python has a great community and a group of very smart maintainers who are very patient with the novice. So focus on your favorite open source projects, join their mailing lists, and share experiences with experienced developers. Your experience will be plentiful in the near future, and you will be one of them.

Finally I would like to quote the next Zen of Python . We think, learn, be inspired! Happy pythoning

>>> import thisthe Zen of Python, by Tim petersbeautiful is better than ugly. Explicit is better than implicit. Simple are better than Complex.complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren ' t special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should is one--and preferably only one--obvious the-do it. Although that's obvious at first unless you ' re dutch.now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it could be a good idea. Namespaces is one honking great idea – let's do more than those!

(turn) Python Eco-Environment introduction