A tutorial for generating egg under the Python setuptools framework _python

This article describes the content of the Setuptools framework, a PEAK project that provides a simpler package management and publishing capability than distutils.
Start

Setuptools modules are very "evasive". For example, if we download a package that uses setuptools instead of a distutils build, then the installation should work as we would like to do: usually using Python setup.py install. To do this, a package bundled with Setuptools will contain a very small boot module ez_setup.py in the archive file. The only thing to note here is that ez_setup.py is trying to download and install the required setuptools--in the background. Of course, this requires a machine that connects to the network. If the Setuptools is already installed on the local machine, then the background step is no longer needed, but if it needs to be installed manually, much of the transparency is lost. However, most systems now have an Internet connection, and it is not particularly troublesome to perform several special steps for machines that do not have network connectivity.

The real advantage of Setuptools is not in implementing the functionality that Distutils can achieve-although it does enhance distutils functionality and simplifies the content in setup.py scripts. The biggest advantage of Setuptools is its enhanced package management capabilities. It can use a more transparent method to find, download and install the dependency pack, and you can switch freely in multiple versions of a package, all installed on the same system, or you can declare a specific version of a package, and you can update to the latest version of a package with just one simple command. Most impressive of all, even though some of the package developers may have never considered any setuptools compatibility issues, we can still use these packages.

Let's discuss it in more detail below.

Guide

Tool ez_setup.py is a simple script that can boot the rest of the setuptools. It is somewhat confusing that the Easy_install script provided in the full Setuptools package is the same as the functionality implemented by ez_setup.py. However, the former assumes that Setuptools has already been installed, so it will skip the installation process behind the scenes. All two versions can accept the same parameters and switches.

The first step in this process is to download a small script ez_setup.py:
Listing 1. Download Boot script

% Wget-q http://peak.telecommunity.com/dist/ez_setup.py

You can then run the script without any arguments to install the rest of the Setuptools (if you do not take this step as a separate step, it will be completed the first time you install another package). You will see something similar to the following (depending on the version you are using, of course):
Listing 2. Guide Setuptools

% python ez_setup.py
downloading http://cheeseshop.python.org/packages/2.4/s/
 setuptools/ SETUPTOOLS-0.6B1-PY2.4.EGG#MD5=B79A8A403E4502FBB85EE3F1941735CB
processing Setuptools-0.6b1-py2.4.egg
Creating/sw/lib/python2.4/site-packages/setuptools-0.6b1-py2.4.egg
extracting Setuptools-0.6b1-py2.4.egg to/ Sw/lib/python2.4/site-packages
removing setuptools 0.6a11 from easy-install.pth file adding Setuptools 0.6b1
To easy-install.pth file
installing Easy_install script to/sw/bin
installing easy_install-2.4 script To/sw/bin

Installed/sw/lib/python2.4/site-packages/setuptools-0.6b1-py2.4.egg
Processing dependencies for Setuptools

Complete. This is what we need to do to ensure that the Setuptools is installed on the system.

installation package

For many Python packages, all you have to do to install these packages is to pass the names of these packages to ez_setup.py or Easy_install as a parameter. Now that you've loaded setuptools with a bootstrap script, you can use an internally simpler easy_install (actually it's a little different from the version we chose).

For example, suppose you want to install the Sqlobject package. The procedure is very simple, as shown in Listing 3. Note that Sqlobject relies on a package named Formencode; Fortunately, this is resolved nicely:
Listing 3. Install a typical package

% Easy_install sqlobject searching for sqlobject Reading http://www.python.org/pypi/SQLObject/Reading . org best match:sqlobject 0.7.0 downloading http://cheeseshop.python.org/packages/2.4/S/sqlobject/ SQLOBJECT-0.7.0-PY2.4.EGG#MD5=71830B26083AFC6EA7C53B99478E1B6A processing Sqlobject-0.7.0-py2.4.egg creating/sw/ Lib/python2.4/site-packages/sqlobject-0.7.0-py2.4.egg extracting Sqlobject-0.7.0-py2.4.egg to/sw/lib/python2.4/ Site-packages adding Sqlobject 0.7.0 to easy-install.pth file installing Sqlobject-admin script To/sw/bin installed/sw/ Lib/python2.4/site-packages/sqlobject-0.7.0-py2.4.egg processing dependencies for SQLObject Searching for FormEncode
>=0.2.2 Reading http://www.python.org/pypi/FormEncode/Reading http://formencode.org best Match:formencode 0.5.1 Downloading http://cheeseshop.python.org/packages/2.4/F/formencode/formencode-0.5.1-py2.4.egg#md5= f8a19cbe95d0ed1b9d1759b033b7760d processing Formencode-0.5.1-py2.4.egg creating/sw/lib/python2.4/sIte-packages/formencode-0.5.1-py2.4.egg extracting Formencode-0.5.1-py2.4.egg To/sw/lib/python2.4/site-packages

 Adding Formencode 0.5.1 to easy-install.pth file Installed/sw/lib/python2.4/site-packages/formencode-0.5.1-py2.4.egg

As you can see from these messages, Easy_install to find information about the package on the www.python.org/pypi/, and then find out where it really can be downloaded (where the egg package is on the cheeseshop.python.org , and more about egg later.

It is now possible to install not only the latest version of a package (this is the default action). If you want, you can also provide a specific version requirement for Easy_install. Now let's try to install a post-beta version of Sqlobject.
Listing 4. Install the minimum version of a package

% Easy_install ' sqlobject>=1.0 '
searching for sqlobject>=1.0
Reading http://www.python.org/pypi/ sqlobject/
Reading http://sqlobject.org
No Local Packages or download links found for sqlobject>=1.0
Error:could not find suitable distribution for
 requirement.parse (' sqlobject>=1.0 ')

If this is the case (as it was written) Sqlobject's latest version is less than 1.0, then nothing will be installed.

Install the "naive" package

Sqlobject can be identified setuptools, but what if you want to install a package that is not yet compatible with Setuptools? For example, I never used setuptools for my "Gnosis Utilities" before this article. Now, however, let's try to install the package, known only in its HTTP (or FTP, SVN, CVS) locations (Setuptools can understand all of these protocols). My download Web site has versions of various Gnosis Utilities, and their naming takes on a common version style:
Listing 5. Installing a package that does not recognize setuptools

% easy_install-f Http://gnosis.cx/download/Gnosis_Utils.More/Gnosis_Utils searching For gnosis-utils Reading http://gnosis.cx/download/Gnosis_Utils.More/Best match:gnosis-utils 1.2.1 Downloading http:/ /gnosis.cx/download/gnosis_utils.more/gnosis_utils-1.2.1.zip processing Gnosis_utils-1.2.1.zip Running Gnosis_ Utils-1.2.1/setup.py-q Bdist_egg--DIST-DIR/TMP/EASY_INSTALL-CCRXES/GNOSIS_UTILS-1.2.1/EGG-DIST-TMP-SH4DW1 Zip_ Safe flag not set; Analyzing archive Contents ... gnosis.__init__: module references __file__ gnosis.magic.__init__: module references __ file__ gnosis.xml.objectify.doc.__init__: module references __file__ gnosis.xml.pickle.doc.__init__: module References __file__ gnosis.xml.pickle.test.test_zdump:module references __file__ adding gnosis-utils to Easy-install.pth file Installed/sw/lib/python2.4/site-packages/gnosis_utils-1.2.1-py2.4.egg Processing Dependencies for Gnosis-utils 

Fortunately, Easy_install can do all this well. It looks at the given download directory, identifies the highest available version, expands the package, and then repackage it into the "egg" format, which can be used for installation. The import Gnosis can now be run in a script. But what if you now need to test a script for a specific version of Gnosis Utilities? This is also very simple:
Listing 6. Install a specific version of a "naive" package

% easy_install-f http://gnosis.cx/download/Gnosis_Utils.More/
 "gnosis_utils==1.2.0"
searching for gnosis-utils==1.2.0
Reading http://gnosis.cx/download/Gnosis_Utils.More/best
match:gnosis-utils 1.2.0
Downloading http://gnosis.cx/download/Gnosis_Utils.More/
 Gnosis_utils-1.2.0.zip
[...]
Removing gnosis-utils 1.2.1 from easy-install.pth file adding Gnosis-utils 1.2.0 to
easy-install.pth file

Installed/sw/lib/python2.4/site-packages/gnosis_utils-1.2.0-py2.4.egg
processing dependencies for gnosis-utils==1.2.0

Two versions of Gnosis Utilities are now usually installed, and the current active version is 1.2.0. Switching the active version back to 1.2.1 is also very simple:
Listing 7. Modify the active version in system-wide

% easy_install "gnosis_utils==1.2.1"
searching for gnosis-utils==1.2.1 best
match:gnosis-utils 1.2.1
Processing Gnosis_utils-1.2.1-py2.4.egg
removing gnosis-utils 1.2.0 from easy-install.pth file
adding Gnosis-utils 1.2.1 to easy-install.pth file

Using/sw/lib/python2.4/site-packages/gnosis_utils-1.2.1-py2.4.egg
processing dependencies for gnosis-utils==1.2.1

This time, of course, only one version is active. But by putting two lines of similar content on each script, you can let the script choose the version that you want to use:
Listing 8. Using a version of a package in a script

From pkg_resources import require
require ("gnosis_utils==1.2.0")

By using these requirements, setuptools can add a specific version when the import statement is run (if you specify a greater than comparison, the latest available version).

Let the package identify Setuptools

I would prefer that users do not need to know the Gnosis Utilities download directory to install it. This can usually work because Gnosis Utilities has a list of information on the Python cheeseshop. Unfortunately, because I didn't consider setuptools, I set up a "python.org" entry for my Gnosis Utilities in the http://www.python.org/pypi/Gnosis% 20utilities/1.2.1. Specifically, the archive is named based on a pattern similar to gnosis_utils-n.n.n.tar.gz (these tools are also packaged as. zip and. tar.bz2 files, and the latest versions are packaged into Win32.exe installers, all of which Setuptools can be handled well). However, the spelling of the project name on the cheeseshop is slightly different from the "Gnosis Utilities". In fact, a small administrative version of Cheeseshop changes will create http://www.python.org/pypi/Gnosis_Utils/1.2.1-a as a post release version. The release archive itself doesn't change, but adds a bit of metadata to the cheeseshop. With a little effort, you can use a simpler installer (note that for testing purposes I ran a easy_install-m to remove the installed package).
Listing 9. Simple increase in recognition of setuptools

% Easy_install gnosis_utils
searching for gnosis-utils
Reading http://www.python.org/pypi/Gnosis_Utils/
Reading Http://www.gnosis.cx/download/Gnosis_Utils.ANNOUNCE
Reading http://gnosis.cx/download/Gnosis_Utils.More/Best
match:gnosis-utils 1.2.1
downloading [...]

I've ignored the rest of the process because it's the same as what you saw earlier. The only difference is that Easy_install is looking for metadata that matches the specified name on Cheeseshop (in other words www.python.org/pypi/) and uses that information to find the real download location. In this case, the listed. The announce file does not contain any helpful content, but Easy_install will continue to look at another listed URL, which will prove that it is a download directory.

About Egg

Egg is a package that contains all the package data. Ideally, egg is a file that uses zip compression, which includes all of the required package files. But in some cases, Setuptools will decide (or be told by a switch) that the package should not be zip compressed. In these cases, egg is just a simple, uncompressed subdirectory, but the contents are the same. Using a single version makes it easy to convert and saves a little disk space, but the egg directory is the same from functional and organizational structure. Java using JAR files all the time? Technology users will find egg very familiar.

Because of the change in the latest Python version (which requires 2.3.5+ or 2.4) to import hooks, you can simply use egg by setting up Pythonpath or Sys.path and importing the appropriate packages as usual. If you want to take this approach, you don't need to use Setuptools or ez_setup.py. For example, in the working directory used in this article, I put a egg in the Pyyaml package. Now I can use this package as follows:
Listing 10. The egg on the Pythonpath

% export Pythonpath=~/work/dw/pyyaml-3.01-py2.4.egg
% python-c ' Import yaml print yaml.dump ({"foo": "Bar", 1:[2,3] }) '
1: [2, 3]
Foo:bar

However, Pythonpath (or Sys.path in a script or Python shell session) is a bit of a fragile operation. The discovery of egg is best done in a new little. pth file. Any. pth files in site-packages/or Pythonpath are parsed to perform other import operations, similar to checking the directory locations that might contain packages. If you use Setuptools to handle the management of a package, you need to modify a file named Easy-install.pth when you install, update, and delete the package. And you can name this. PTH in the way you like it (as long as it has a. pth extension). For example, here is the contents of my easy-install.pth file:
Listing 11. A. pth file used as a egg location configuration

% cat/sw/lib/python2.4/site-packages/easy-install.pth
Import sys; sys.__plen = len (sys.path)
Setuptools-0.6b1-py2.4.egg
Sqlobject-0.7.0-py2.4.egg
formencode-0.5.1-py2.4.egg
Gnosis_ Utils-1.2.1-py2.4.egg
Import sys; New=sys.path[sys.__plen:]; del Sys.path[sys.__plen:;
 P=getattr (sys, ' __egginsert ', 0); sys.path[p:p]=new;
 Sys.__egginsert = P+len (new)

This format is a bit special: it's similar to a Python script, but it's not exactly. It should be explained that additional listed egg can be added there; better yet, Easy_install will implement this functionality at run time. You can also create any number of. pth files under Site-packages/, and each can list which egg are available.

Enhanced installation Scripts

The ability to install Setuptools naive packages described above (see Listing 6) is only partially valid. In other words, the package gnosis_utils did install, but not complete. All the common features can work, however, when egg is automatically generated, many supporting files are ignored-most of the documents with the extension. txt and the test files with the extension. XML (there are other README,. RNC,. rng,. Xsl and files around the child package). When installing, all of these support files are "best to have", without strict requirements must have. However, we still want to be able to include all the supporting files.

The setup.py scripts used by Gnosis_utils are actually very complex. In addition to listing the basic metadata, in line No. 467, it also tests the functionality and bugs of the Python version, solves some of the faults in the old version of Distutils, and skips the installation of unsupported parts (for example, if Pyexpat is not in the Python release Include: Process OS line terminator transformations, create multiple archive/installer types, and reconstruct MANIFEST files based on test results. The ability to handle these tasks is thanks to another maintainer of the package, Frank McIngvale, which allows Gnosis_utils to successfully install back to the Python 1.5.1 version, provided that this is necessary (not so rich in earlier versions). But the script I'm going to show you here isn't as complicated as distutils script: it simply assumes that a "normal" version of Python is already installed in the system. Even so, setuptools can make installation scripts so simple or appealing.

On the first attempt, let's create a setup.py script that is borrowed from the Setuptools manual and tries to use it to create a egg:
Listing 12. Setuptools setup.py Script

% cat setup.py
from setuptools Import setup, find_packages
setup (
  name = "Gnosis_utils",
  Version = " 1.2.2 ",
  packages = Find_packages (),
)
% python setup.py-q bdist_egg zip_safe not
set; flag arc Hive Contents
... gnosis.__init__: module References __file__
gnosis.doc.__init__: module references __file__
_init__: module References __file__
gnosis.xml.objectify.doc.__init__: module references __file__
gnosis.xml.pickle.doc.__init__: module references __file__
gnosis.xml.pickle.test.test_zdump:module References __file__

This effort is already at work, at least partially. Using these lines can indeed create a egg, but this egg has some similar drawbacks to the egg created with Easy_install: the lack of support for files that are not named with. Py. So let's try again, just a little bit harder:
Listing 13. Add a missing Package_data

From Setuptools Import setup, find_packages
setup (
  name = "Gnosis_utils",
  Version = "1.2.2",
  Package_ data = {': [' *.* ']},
  packages = Find_packages (),
)

This is all the action that needs to be done. Of course, depending on the actual situation, it is usually desirable to make some adjustments. For example, it might list the following:
Listing 14. To package a specific type of file type

Package_data = {' doc ': [' *.txt '], ' xml ': [' *.xml ', ' RELAX/*.RNC ']}

This translation is to include the. txt file in the doc/child package, include the. xml file in the xml/child package, and include all. RNC files in the xml/relax/child package.

Conclusion

In fact, this article only describes the superficial knowledge of custom operations that can be performed with a release that supports Setuptools. For example, if you now have a distribution (which can be the preferred egg format or another archive type), you can upload the archive and metadata to cheeseshop using a command. Obviously, the complete setup.py script should contain the same detailed metadata contained in the older version of the Distutils script, which is skipped for simplicity, but its parameter name is compatible with Distutils.

While it takes some time to fully adapt to the huge functionality provided by Setuptools, it really does make it easier to maintain your own packages and install foreign packages than distutils. If the content you care about is only the installation package, then what you need to know is covered in this article, but you may find some complexity in describing your own package, but it's still not as complex as using distutils.