A detailed description of how to package and publish Python modules

Objective

Yesterday the Vasp file processing library was packaged and uploaded to PyPI, now can be directly through the PIP and Easy_install to install Vaspy (at the same time welcome to use vasp do computational chemistry of children's shoes to add stars and participate in),

Vaspy's Gothub Address: https://github.com/PytLab/VASPy
Vaspy's PyPI Address: https://pypi.python.org/pypi/vaspy/

Because of their memory is really not good, afraid of time long forgotten, so here to strike the vaspy of their own procedures for the Python package and upload to summarize.

Vaspy Package file Structure

First write the entire file structure of the Vaspy package, followed by this as an example to illustrate:

vaspy/├──license├──manifest├──manifest.in├──readme.rst├──requirements.txt├──scripts│  ├──change_incar_ parameters.py│  ├──create_inputs.py│  └── ... ├──setup.cfg├──setup.py├──tests│  ├──incar_test.py│  ├──__init__.py│  ├──oszicar_test.py│  ├── outcar_test.py│  ├──testdata│  │  ├──contcar│  │  ├──dos_sum│  │  ├──elfcar│  │  └── ... │  └── ... └──vaspy  ├──__init__.py  ├──iter.py  ├──matstudio.py  └── ...  4 directories, SI files

Tools for packaging and installing third-party packages

Here we need to use tools such as Setuptools and Pip to package and publish and install our own packages, and to install wheel modules if needed to build wheel. If Python version >=2.7.9 or >=3.4,setuptools and Pip are already installed, you may need to update to the latest version

Pip install-u Pip Setuptools

You can use package management tools, such as

Yum Install pipsudo apt-get install pip

Install via get-pip.py script if it detects that wheel and Setuptools are not installed automatically

Python get-pip.py

Specific tool installation and introduction is not much to say, please refer to requirements for installing packages

The role of different files in the package

setup.py

This file is the most important document for packaging the entire project, which provides two main features:

Setup () function, the parameter of this function specifies how to configure your own project.
Command-line tools, including packaging, testing, publishing, and more. Can be viewed by the following command;

Python setup.py--help-commands

Setup.cfg

This file contains some of the default parameters for the build time, such as the--universal parameter when building Bdist_wheel

[Bdist_wheel]universal=1

The--universal parameter is used by default each time it is packaged, and the effect is similar:

Python setup.py bdist_wheel--universal

Readme.rst

This was originally written by Markdown, packaged and released to PyPI after the discovery PyPI does not support markdown rendering, the page is really a mess, and then use the Restrutruedtext syntax to re-write again. After all, the markup language syntax can basically be used in seconds, really can't find a template Bishi on the line.
Restructuretext syntax rules can be consulted in the official documentation: Quick Restructuredtext

In fact, there is another way is to use Pandoc to convert markdown to RST format, a convenient way is to use the Pyandoc module in the release of the automatic conversion.
Specific methods can be consulted: use Markdown README's in Python modules

Manifest.in

This file tells Setuptools when it is packaged that additional files need to be packaged, such as test data files for unit tests in my vaspy I use this file to include them. Of course readme,license These can also be packed in by it.
Here is the content of my own manifest.in:

Include Readme.rstinclude requirements.txtinclude licenserecursive-include scripts *recursive-include tests *

Specific grammatical rules can be consulted: the manifest.in template

vaspy/

This folder is the package that contains the Vaspy source code.

tests/

This folder is also a child package that contains the unit test script, in order to be able to use the Python setup.py test for unit testing, specifically added __init__.pys make it a package.

Parameters for Setup ()

Here are only a few of the parameters I use, other parameters of the specific use can be consulted: https://docs.python.org/3/distutils/setupscript.html

Name

versions = "Vaspy"

is the name of the entire project, and the name and version number will be used after packaging.

Version

From vaspy import __version__version = __version__

Description

is a short description of the project, a general sentence is good, will be displayed on the PyPI on the lower end of the name.

Long_description

is a long description, equivalent to a concise project, if this string is in RST format, PyPI will automatically render as HTML display. The contents of the Readme.rst can be read directly here.

Url

A connection to a package, usually a link on GitHub or a link to a readthedocs.

Packages

The list of sub-packages that need to be included, Setuptools provides find_packages () to help us find the package under the root path, which Distutil is not.

Setup_requires

This parameter defines the other dependencies (most basic) required for Vaspy installation and smooth running, which are installed when you install with PIP.
The difference between this parameter and Requirements.txt can be consulted: Install_requires vs Requirements files

Classifier

This parameter provides a series of classifications that will be categorized into different directories on PyPI.
Specific categories names and rules refer to: https://pypi.python.org/pypi?%3Aaction=list_classifiers

Test_suite

This parameter can help us to use

Python setup.py test

To run unit tests, no longer need to write a script separately such as run_tests.py to run unit tests.
Official explanation of this parameter:

A string naming a unittest. TestCase subclass (or a package or module containing one or more of the them, or a method of such a subclass), or naming a fun Ction that can is called with no arguments and returns a unittest. TestSuite. If The named Suite is a module, and the module have an additional_tests () function, it's called and the results are added To the tests to is run. If The named Suite is a package, any submodules and subpackages be recursively added to the overall test suite.

This means that the parameter can accept multiple types of parameters:

Receive UnitTest. TestCase subclasses, we can say that all unit tests are written to a test case, then import in, and then send you to Test_suite
The receive function object, which has no parameters and returns a unittest. TestSuite. So we can write a function alone, combine multiple test cases into a suite and then return, then import the function into the test_suite.

Module and package name, I just use this way, before my own tests are separate multiple scripts, so I add a __init__.py can turn it into a package, the package name passed to Test_suite,setuptools will magically run all the tests under this package to the side, This way I add a new script directly to the test script later, and the others do not need to be changed.

Operating effect:

zjshao@shao-pc:/mnt/d/dropbox/code/centos_code/vaspy$ python setup.py testrunning testrunning egg_infocreating Vaspy.egg-infowriting vaspy.egg-info/pkg-infowriting top-level names to vaspy.egg-info/top_level.txtwriting Dependency_links to vaspy.egg-info/dependency_links.txtwriting manifest file ' vaspy.egg-info/sources.txt ' reading Manifest file ' vaspy.egg-info/sources.txt ' reading manifest template ' manifest.in ' writing manifest file ' Vaspy.egg-info /sources.txt ' Running Build_exttest_compare (tests.incar_test. Incartest) Make sure we can compare the Incar objects correctly. ... oktest_eq (tests.incar_test. Incartest) Test __eq__ () function. ... ok ... Some output is omitted here  ----------------------------------------------------------------------Ran tests in 3.574s  OK

Publish your own Python package

1. First go to PyPI to register your account

2. Configure the ~/.PYPIRC as follows:

[distutils]index-servers =  pypi  pypitest  [Pypi]username:shaozhengjiangpassword:mypassword  [ Pypitest]username:shaozhengjiangpassword:mypassword

3. Then register and upload your own package to the test server

PyPI provides a test server that we can test on on this test server.

Python setup.py register-r pypitest

And then

Python setup.py sdist upload-r pypitest

If there is no problem we should not get any mistakes.

4. Upload to PyPI

If the above test is successful, we can follow the same steps to register and upload the package.

Python setup.py register-r pypipython setup.py sdist upload-r PyPI

Ok, then we can see our own package on PyPI (https://pypi.python.org/pypi/vaspy/).