tutorial on creating an egg in Python's setuptools framework

This article describes the content of the Setuptools framework, which is a sub-project of PEAK, which provides simpler package management and distribution capabilities than distutils.
Start

The Setuptools module is very "evasive". For example, if we download a package built using Setuptools instead of Distutils, then the installation should work just as we would expect: the Python setup.py install is usually available. To achieve this, packages bundled with Setuptools will contain a small boot module ez_setup.py in the archive file. The only thing to note here is that ez_setup.py is trying to download and install the required setuptools--in the background. Of course, this requires a network-connected machine. If the Setuptools is already installed on the local machine, then this background step is no longer necessary, but if it needs to be installed manually, then much of the transparency is lost. However, most systems now have an Internet connection, and it is not particularly troublesome to perform several special steps for a machine that is not connected to the network.

The real advantage of setuptools is not the ability to implement distutils-although it does enhance the functionality of distutils and simplifies the content in setup.py scripts. The biggest advantage of Setuptools is its enhanced package management capabilities. It can use a more transparent way to find, download, and install dependent packages, and you can switch freely across multiple versions of a package, which are installed on the same system, or you can declare a specific version of a package, or you can update to the latest version of a package with just one simple command. The most impressive is that even if some package developers may have never considered any setuptools compatibility issues, we can still use these packages.

Let's explore in detail below.

Guide

Tool ez_setup.py is a simple script that can boot the rest of the setuptools. It is somewhat confusing that the Easy_install script provided in the full Setuptools package is the same as the functionality implemented by ez_setup.py. But the former assumes that Setuptools has already been installed, so it will skip the installation process behind the scenes. All two versions can accept the same parameters and switches.

The first step in this process is to download a small script ez_setup.py:
Listing 1. Download the boot script

% Wget-q http://peak.telecommunity.com/dist/ez_setup.py

You can then run the script without any parameters to install the remainder of Setuptools (if you do not perform this step as a separate step, it will still be completed the first time you install the other package). You'll see something like this (depending on the version you're using, of course):
Listing 2. Boot Setuptools

% python ez_setup.pydownloading http://cheeseshop.python.org/packages/2.4/s/Setuptools/setuptools-0.6b1-py2.4.egg #md5 =b79a8a403e4502fbb85ee3f1941735cbprocessing setuptools-0.6b1-py2.4.eggcreating/sw/lib/python2.4/ Site-packages/setuptools-0.6b1-py2.4.eggextracting Setuptools-0.6b1-py2.4.egg to/sw/lib/python2.4/ Site-packagesremoving setuptools 0.6a11 from easy-install.pth fileadding setuptools 0.6b1 to Easy-install.pth fileinstalling easy_install Script to/sw/bininstalling easy_install-2.4 script to/sw/bininstalled/sw/lib/python2.4/ site-packages/setuptools-0.6b1-py2.4.eggprocessing Dependencies for Setuptools

Complete. This is what we need to do to ensure that Setuptools is installed on the system.

Install package

For many Python packages, the only thing you need to do to install these packages is to pass the names of these packages to ez_setup.py or Easy_install as a parameter. Now that the Setuptools has been loaded with a bootstrap script, you can use the more streamlined easy_install inside (which is actually a little different from the version we chose).

For example, suppose you want to install the Sqlobject package. The process is very simple, as shown in Listing 3. Note that Sqlobject relies on a package called formencode; Fortunately, this will be well resolved:
Listing 3. To install a typical package

% Easy_install sqlobjectsearching for sqlobjectreading http://www.python.org/pypi/SQLObject/Reading/HTTP/ Sqlobject.orgbest match:sqlobject 0.7.0Downloading http://cheeseshop.python.org/packages/2.4/S/sqlobject/ Sqlobject-0.7.0-py2.4.egg#md5=71830b26083afc6ea7c53b99478e1b6aprocessing sqlobject-0.7.0-py2.4.eggcreating/sw/ Lib/python2.4/site-packages/sqlobject-0.7.0-py2.4.eggextracting Sqlobject-0.7.0-py2.4.egg to/sw/lib/python2.4/ Site-packagesadding sqlobject 0.7.0 to easy-install.pth fileinstalling sqlobject-admin script to/sw/bininstalled/sw/ lib/python2.4/site-packages/sqlobject-0.7.0-py2.4.eggprocessing dependencies for sqlobjectsearching for FormEncode >=0.2.2reading http://www.python.org/pypi/FormEncode/Reading http://formencode.orgBest Match:formencode 0.5.1Downloading http://cheeseshop.python.org/packages/2.4/F/formencode/formencode-0.5.1-py2.4.egg#md5= F8a19cbe95d0ed1b9d1759b033b7760dprocessing formencode-0.5.1-py2.4.eggcreating/sw/lib/python2.4/site-packages/ Formencode-0.5.1-py2.4.eggextracting Formencode-0.5.1-py2.4.egg to/sw/lib/python2.4/site-packagesadding FormEncode 0.5.1 to Easy-install.pth Fileinstalled/sw/lib/python2.4/site-packages/formencode-0.5.1-py2.4.egg

As you can see from these messages, Easy_install to find information about the package on www.python.org/pypi/and then find where it really can be downloaded (here the Egg pack is on the cheeseshop.python.org Later, more on egg).

You can now not only install the latest version of a package (this is the default action). If you prefer, you can also provide a specific version requirement for Easy_install. Now let's try to install a post-beta version of Sqlobject.
Listing 4. Install the minimum version of a package

% Easy_install ' sqlobject>=1.0 ' Searching for sqlobject>=1.0reading http://www.python.org/pypi/SQLObject/ Reading http://sqlobject.orgNo Local Packages or download links found for sqlobject>=1.0error:could not find suitable Distribution for Requirement.parse (' sqlobject>=1.0 ')

If (this is the case when writing this article) the latest version of Sqlobject is less than 1.0, then nothing will be installed.

Install the "naive" package

Sqlobject can identify setuptools, but what if you want to install a package that is not yet compatible with Setuptools? For example, I never used setuptools for my "Gnosis Utilities" before this article. However, now let's try to install this package, known only for its HTTP (or FTP, SVN, CVS) location (Setuptools can understand all of these protocols). There are various versions of Gnosis Utilities on my download Web site, and their naming takes on a common version style:
Listing 5. Installing packages that do not recognize setuptools

% easy_install-f http://gnosis.cx/download/Gnosis_Utils.More/Gnosis_UtilsSearching for gnosis-utilsreading/HTTP// Gnosis.cx/download/gnosis_utils.more/best match:gnosis-utils 1.2.1Downloading Http://gnosis.cx/download/Gnosis_ Utils.more/gnosis_utils-1.2.1.zipprocessing gnosis_utils-1.2.1.ziprunning gnosis_utils-1.2.1/setup.py-q Bdist_egg --dist-dir/tmp/easy_install-ccrxes/gnosis_utils-1.2.1/egg-dist-tmp-sh4dw1zip_safe flag not set; Analyzing archive contents...gnosis.__init__: module references __file__gnosis.magic.__init__: module references __ file__gnosis.xml.objectify.doc.__init__: module references __file__gnosis.xml.pickle.doc.__init__: module References __file__gnosis.xml.pickle.test.test_zdump:module references __file__adding Gnosis-utils 1.2.1 to Easy-install.pth fileinstalled/sw/lib/python2.4/site-packages/gnosis_utils-1.2.1-py2.4.eggprocessing Dependencies for Gnosis-utils

Fortunately, Easy_install can do all this well. It looks at the given download directory, identifies the highest available version, expands the package, and then repackage it into the "egg" format, which can be used for installation. Import Gnosis can now run in a script. But what if you need to test a script for a specific version of Gnosis Utilities now? This is also very simple:
Listing 6. Install a specific version of the "naive" package

% easy_install-f http://gnosis.cx/download/Gnosis_Utils.More/"gnosis_utils==1.2.0" Searching for gnosis-utils== 1.2.0Reading http://gnosis.cx/download/Gnosis_Utils.More/Best match:gnosis-utils 1.2.0Downloading http://gnosis.cx /download/gnosis_utils.more/gnosis_utils-1.2.0.zip[...] Removing Gnosis-utils 1.2.1 from easy-install.pth fileadding gnosis-utils 1.2.0 to Easy-install.pth fileinstalled/sw/lib /python2.4/site-packages/gnosis_utils-1.2.0-py2.4.eggprocessing Dependencies for gnosis-utils==1.2.0

Two versions of Gnosis Utilities are now usually installed, and the current active version is 1.2.0. Switching the active version back to 1.2.1 is also straightforward:
Listing 7. Modifying the "active" version on a system-wide scale

% easy_install "gnosis_utils==1.2.1" Searching for gnosis-utils==1.2.1best match:gnosis-utils 1.2.1Processing Gnosis_ Utils-1.2.1-py2.4.eggremoving gnosis-utils 1.2.0 from easy-install.pth fileadding gnosis-utils 1.2.1 to Easy-install.pth fileusing/sw/lib/python2.4/site-packages/gnosis_utils-1.2.1-py2.4.eggprocessing Dependencies for gnosis-utils==1.2.1

Of course, this time only one version is active. But by placing two lines of similar content on each script, you can let the script choose the version you want to use:
Listing 8. Using a version of a package in a script

From pkg_resources import requirerequire ("gnosis_utils==1.2.0")

By using the above requirements, setuptools can add a specific version when the import statement is run (if a greater than comparison is specified, the latest available version).

Enable the package to recognize Setuptools

I would prefer that users don't need to know the Gnosis Utilities download directory to install it. This can usually work because Gnosis Utilities has a list of information on the Python cheeseshop. Unfortunately, because I didn't consider setuptools, I set up a "mismatch" entry for my gnosis Utilities on python.org http://www.python.org/pypi/Gnosis% 20utilities/1.2.1. Specifically, this archive is named according to a pattern similar to gnosis_utils-n.n.n.tar.gz (these tools are also packaged into. zip and. tar.bz2 files, and the latest versions are packaged as Win32.exe installers, all of which Setuptools can be handled very well). However, the spelling of the project name on Cheeseshop is slightly different from the "Gnosis Utilities". In fact, changes to a very small administrative version of Cheeseshop will create http://www.python.org/pypi/Gnosis_Utils/1.2.1-a as a post-release version. The release archive itself does not change much, but adds a bit of metadata to the cheeseshop. With minimal effort, you can use a simpler Setup program (note that for testing purposes, I ran a easy_install-m to remove the installed package).
Listing 9. Simple increase of the recognition of Setuptools

% Easy_install gnosis_utilssearching for gnosis-utilsreading http://www.python.org/pypi/Gnosis_Utils/Reading/HTTP/ Www.gnosis.cx/download/Gnosis_Utils.ANNOUNCEReading http://gnosis.cx/download/Gnosis_Utils.More/Best Match: Gnosis-utils 1.2.1Downloading [...]

I've ignored the rest of the process because it's no different than what you saw earlier. The only difference is that Easy_install is looking for metadata that matches the specified name on Cheeseshop (in other words www.python.org/pypi/) and uses that information to find the true download location. In this case, the listed. The announce file does not contain any helpful content, but Easy_install will continue to look at another listed URL, which proves to be a download directory.

About Egg

An egg is a package that contains all the package data. In an ideal situation, egg is a zip-compressed file that includes all the required package files. However, in some cases, Setuptools will decide (or be told by switch) that the package should not be zip compressed. In these cases, the egg is just a simple, uncompressed subdirectory, but the contents are the same. With a single version you can easily convert and save a bit of disk space, but the egg directory is functionally and structurally identical. Always use the Java of the JAR file? Users of the technology will find egg very familiar.

Because of changes in the latest Python version (which requires 2.3.5+ or 2.4) to import hooks, you can simply use the egg by setting up PYTHONPATH or Sys.path and importing the appropriate package as usual. If you want to use this approach, you do not need Setuptools or ez_setup.py. For example, in the working directory used in this article, I put an egg for the Pyyaml package. Now I can use this package, as follows:
Listing 10. The egg on the PYTHONPATH

% Export pythonpath=~/work/dw/pyyaml-3.01-py2.4.egg% python-c ' import yaml; Print Yaml.dump ({"foo": "Bar", 1:[2,3]}) ' 1: [2, 3]foo:bar

However, PYTHONPATH (or sys.path within a script or Python shell session) is a bit of a fragile operation. The discovery of egg is best done in a new. pth file. Any. pth files in site-packages/or PYTHONPATH are parsed to perform other import operations in a way similar to checking those directory locations that might contain packages. If you use Setuptools to handle the management of the package, you will need to modify a file named Easy-install.pth when you install, update, and delete the package. And you can name the. PTH in the way you like it (as long as its extension is. pth). For example, here is the contents of my easy-install.pth file:
Listing 11. . pth file used as the egg location configuration

% Cat/sw/lib/python2.4/site-packages/easy-install.pthimport sys; Sys.__plen = Len (sys.path) Setuptools-0.6b1-py2.4.eggsqlobject-0.7.0-py2.4.eggformencode-0.5.1-py2.4.egggnosis_ Utils-1.2.1-py2.4.eggimport SYS; New=sys.path[sys.__plen:]; del Sys.path[sys.__plen:]; P=getattr (sys, ' __egginsert ', 0); Sys.path[p:p]=new; Sys.__egginsert = P+len (new)

This format is a bit special: it's similar to a Python script, but it's not exactly. It should be explained that the additional listed egg can be added there; better yet, Easy_install will implement this functionality at run time. You can also create any number of. pth files under Site-packages/, each of which can be listed with which egg is available.

Enhanced installation Scripts

The ability to install Setuptools naive packages described above (see Listing 6) is only partially valid. In other words, the package gnosis_utils is indeed installed, but not complete. All the common features work, but many support files are ignored when the egg is automatically generated-most of them are documents with a. txt extension and a test file with an. xml extension (there are some other README,. RNC,. rng,. Xsl, and files around the child package). At the time of installation, all of these support files are "best to have", and there is no need to be strict. However, we still want to be able to include all the supporting documents.

The setup.py script used by Gnosis_utils is actually very complex. In addition to listing the basic metadata, in the No. 467 line of code, it also fully tests the functionality and bugs of the Python version, resolves some of the failures in the older version of Distutils, and bypasses the installation of unsupported portions (for example, if Pyexpat is not in the Python release) Include), process the conversion of the OS line terminator, create multiple archive/installer types, and rebuild the MANIFEST file based on the test results. The ability to handle these tasks is thanks to another maintainer of this package, Frank McIngvale, who will allow Gnosis_utils to successfully install back to the Python 1.5.1 version, if that is not the case in earlier versions. But the script I'm going to show you here isn't as complicated as the distutils script: it simply assumes that a "normal" version of Python is already installed in the system. Even so, setuptools can make installation scripts so simple or very appealing.

On the first attempt, let's create a setup.py script that was borrowed from the Setuptools manual and tried to use it to create an egg:
Listing 12. Setuptools setup.py Script

% Cat Setup.pyfrom setuptools Import Setup, Find_packagessetup (  name = "Gnosis_utils",  Version = "1.2.2",  Packages = Find_packages (),)% python setup.py-q bdist_eggzip_safe flag not set; Analyzing archive contents...gnosis.__init__: module references __file__gnosis.doc.__init__: module references __file_ _gnosis.magic.__init__: module references __file__gnosis.xml.objectify.doc.__init__: module references __file__ gnosis.xml.pickle.doc.__init__: module references __file__gnosis.xml.pickle.test.test_zdump:module references __ file__

This effort can already work, at least in part. Using these lines of content does create an egg, but this egg has some similar drawbacks to the egg created with Easy_install: lack of support for files that are not named with. Py. So let's try it again, just a little bit harder:
Listing 13. Add the missing Package_data

From Setuptools Import Setup, Find_packagessetup (  name = "Gnosis_utils",  Version = "1.2.2",  package_data = { ': [' *. * ']},  packages = Find_packages (),)

This is all you need to do. Of course, depending on the situation, it is usually desirable to make some adjustments to it. For example, it might list the following:
Listing 14. Package a specific type of file type

Package_data = {' doc ': [' *.txt '], ' xml ': [' *.xml ', ' RELAX/*.RNC ']}

The translation of this is: include the. txt file in the doc/child package, include the. xml file in the xml/child package, and include all the. RNC files in the xml/relax/child package.

Conclusion

This article actually only describes the knowledge of the surface of a custom operation that can be performed with a release version that supports Setuptools. For example, if you now have a release (which can be the preferred egg format or another type of archive), you can use a command to upload the archive and metadata to cheeseshop. Obviously, the complete setup.py script should contain the same detailed metadata that was contained in the old version of the Distutils script, and for the sake of simplicity, this article skips the content, but its parameter names are compatible with Distutils.

While it will take some time to fully adapt to the great features provided by Setuptools, it really does make it easier to maintain your own packages and install foreign packages than distutils. If all you care about is the installation package, what you need to know is covered in this article, but you may find some complexity when describing your own package, but it's still not as complex as using distutils.

