Bash instance, Part 1: Exploring the ebuild System

Daniel Robbins
President and CEO, Gentoo Technologies, Inc.

Daniel Robbins in his last articleBash instanceThe article details the Gentoo Linux ebuild system, which shows an excellent example of bash capabilities. Step by step, he shows you how to implement the ebuild system and involves many convenient bash technologies and design strategies. At the end of this article, you will have a good grasp of the technologies involved in making bash-based applications, and start coding your own Automatic Build system.

Go to the ebuild System
I have always been expecting this third and last article.Bash instanceSince we have already covered the basics of bash programming in articles 1st and 2nd, we can focus on more advanced topics such as Bash application development and programming. In this article, we will spend a lot of time coding and refining projects. the Gentoo Linux ebuild system will give you a lot of practical and real-world bash development experience.

I am the chief designer of Gentoo Linux (or the next generation Linux OS of beta version. One of my primary responsibilities is to ensure that all Binary packages (similar to rpm) are correctly created and used together. As you may know, the standard Linux system is not composed of a unified source tree (like BSD), but actually consists of more than 25 core packages for collaborative work. These include:

Package	Description
Linux	Actual Kernel
Util-linux	Collection of Linux-related miscellaneous programs
E2fsprogs	Set of utilities related to the ext2 File System
Glibc	Gnu c library

Each package is located in its own tar package and maintained by different independent developers or development teams. To create a release, each package must be downloaded, compiled, and packaged separately. The compilation and packaging steps must be repeated each time you want to fix, upgrade, or improve the package (and the package is indeed updated very quickly ). To help eliminate repeated steps involved in creating and updating packages, I created an ebuild system, which is almost all written in bash. To increase your Bash knowledge, I will show you step by step how to unpack and compile the ebuild system. When explaining each step, we will also discuss why some design decisions should be made. At the end of this article, you will not only have a good grasp of large-scale bash programming projects, but also implement a large part of the complete automatic building system.

Why Bash?
Bash is a basic component of the Gentoo Linux ebuild system. There are several reasons for choosing ebuild as the main language. First, its syntax is not complex and familiar to people, which is especially suitable for calling external programs. The automatic building system automatically calls the "glue code" of external programs, and bash is very suitable for such applications. Second, Bash's support for functions allows the ebuild system to use modular and easy-to-understand code. Third, the ebuild system uses Bash to support environment variables, allowing package maintenance personnel and developers to easily configure them online at runtime.

Building Process Review
Before discussing the ebuild system, let's review what is involved in the compilation and installation packages. For example, let's take a look at the "sed" package, a standard GNU text stream editing utility that is part of all Linux versions. First, download the source code tar package (sed-3.02.tar.gz) (see references ). We will store this file in/usr/src/distfiles and reference this directory using the environment variable "$ distdir. "$ Distdir" is the directory where all original source code tar packages are located. It is a large source code library.

The next step is to create a temporary directory named "work", which stores the decompressed source code. The "$ workdir" environment variable will be used to reference the directory later. To do this, enter the directory with write permission and enter:

Decompress sed to a temporary directory

$ mkdir work$ cd work$ tar xzf /usr/src/distfiles/sed-3.02.tar.gz

Decompress the tar package and create a directory containing all the source code named sed-3.02. The sed-3.02 directory will be referenced later using the environment variable "$ srcdir. To compile the program, enter:

Decompress sed to a temporary directory

$ Cd sed-3.02 $./configure -- prefix =/usr (autoconf generates the appropriate make file, which takes some time) $ make (it takes a little time to compile the package from source code)

Because this article only describes the steps for unpacking and compiling, We will skip the "make install" step. If you want to write a bash script to execute all these steps, the Code may be similar:

Sample bash script for unpacking/compiling

#!/usr/bin/env bashif [ -d work ]then # remove old work directory if it exists rm -rf workfimkdir workcd worktar xzf /usr/src/distfiles/sed-3.02.tar.gzcd sed-3.02./configure --prefix=/usrmake

Make code generic
Although this script can be automatically compiled, it is not very flexible. Basically, the bash script only contains the list of all commands entered in the command line. Although this solution can be used, it is best to make a suitable script that can quickly unpack and compile any package by modifying only a few lines. In this way, the workload required by the package maintenance personnel to add a new package to the release version is greatly reduced. Let's first try to use many different environment variables to make the build script more suitable:

New and more general scripts

#!/usr/bin/env bash# P is the package nameP=sed-3.02# A is the archive nameA=${P}.tar.gzexport ORIGDIR=`pwd`export WORKDIR=${ORIGDIR}/workexport SRCDIR=${WORKDIR}/${P}if [ -z "$DISTDIR" ]then # set DISTDIR to /usr/src/distfiles if not already set DISTDIR=/usr/src/distfilesfiexport DISTDIRif [ -d ${WORKDIR} ]then # remove old work directory if it exists rm -rf ${WORKDIR}fimkdir ${WORKDIR}cd ${WORKDIR}tar xzf ${DISTDIR}/${A}cd ${SRCDIR}./configure --prefix=/usrmake

Many environment variables have been added to the Code, but it basically executes the same function. However, if you want to compile any standard GNU autoconf-based source code tar package, simply copy the file to a new file (use a proper name to reflect the name of the new package it compiled ), then, change the values of "$ A" and "$ P" to A new value. All other environment variables are automatically adjusted to the correct settings, and the script works as expected. Although this is convenient, there is still room for improvement in the code. This code is much longer than the "transcript" script we created. Since one of the goals of any programming project is to reduce user complexity, it is best to significantly shorten the code or at least better organize the code. You can use a clever way to do this-split the code into two separate files. Save this file as "sed-3.02.ebuild ":

Sed-3.02.ebuild

#the sed ebuild file -- very simple!P=sed-3.02A=${P}.tar.gz

The first file is not important. It only contains environment variables that must be configured in each package. The second file contains the main part of the operation. Save it as "ebuild" and make it an executable file:

Ebuild script

#!/usr/bin/env bashif [ $# -ne 1 ]then echo "one argument expected." exit 1fiif [ -e "$1" ]then source $1else echo "ebuild file $1 not found." exit 1fiexport ORIGDIR=`pwd`export WORKDIR=${ORIGDIR}/workexport SRCDIR=${WORKDIR}/${P}if [ -z "$DISTDIR" ]then # set DISTDIR to /usr/src/distfiles if not already set DISTDIR=/usr/src/distfilesfiexport DISTDIRif [ -d ${WORKDIR} ]then # remove old work directory if it exists rm -rf ${WORKDIR}fimkdir ${WORKDIR}cd ${WORKDIR}tar xzf ${DISTDIR}/${A}cd ${SRCDIR}./configure --prefix=/usrmake

Now that you have split the build system into two files, I bet you are definitely thinking about how it works. Basically, to compile sed, enter:

$ ./ebuild sed-3.02.ebuild

When executing "ebuild", it first tries to "source" variable "$1 ". What does this mean? Remember what I said in the previous article: "$1" is the first command-line argument -- here, it is "sed-3.02.ebuild ". In bash, the "source" command reads bash statements from the file and then executes them, as if they appear directly in the file where the "source" command is located. Therefore, "source $ {1}" causes the "ebuild" script to execute commands that define "$ P" and "$ A" in the "sed-3.02.ebuild. This design change is really convenient, because if you want to compile another program, instead of sed, you can simply create a new. ebuild file and pass it as an independent variable to the "ebuild" script. In this way, the. ebuild file is very simple, and the complex operations of the ebuild system are stored in one place, that is, the "ebuild" script. In this way, you only need to edit the "ebuild" script to upgrade or enhance the ebuild system, and keep the implementation details outside the ebuild file. Here is a gzip sample ebuild file:

Gzip-1.2.4a.ebuild

#another really simple ebuild script!P=gzip-1.2.4aA=${P}.tar.gz

Add functionality
Well, we are making progress. However, I want to add some additional features. I want the ebuild script to accept another command line independent variable: "compile", "unpack", or "all ". This command line independent variable tells the ebuild script which step to execute the build process. In this way, you can tell the ebuild to unpackage the file, but not compile it (so that you can view the source code file before starting compilation ). To do this, a case statement is added, which tests "$2" and then performs different operations based on the value. The Code is as follows:

Ebuild, revision 2

#!/usr/bin/env bashif [ $# -ne 2 ]then echo "Please specify two args - .ebuild file and unpack, compile or all" exit 1fiif [ -z "$DISTDIR" ]then # set DISTDIR to /usr/src/distfiles if not already set DISTDIR=/usr/src/distfilesfiexport DISTDIRebuild_unpack() { #make sure we're in the right directory cd ${ORIGDIR} if [ -d ${WORKDIR} ] then rm -rf ${WORKDIR} fi mkdir ${WORKDIR} cd ${WORKDIR} if [ ! -e ${DISTDIR}/${A} ] then echo "${DISTDIR}/${A} does not exist. Please download first." exit 1 fi tar xzf ${DISTDIR}/${A} echo "Unpacked ${DISTDIR}/${A}." #source is now correctly unpacked}ebuild_compile() { #make sure we're in the right directory cd ${SRCDIR} if [ ! -d "${SRCDIR}" ] then echo "${SRCDIR} does not exist -- please unpack first." exit 1 fi ./configure --prefix=/usr make }export ORIGDIR=`pwd`export WORKDIR=${ORIGDIR}/workif [ -e "$1" ]then source $1else echo "Ebuild file $1 not found." exit 1fiexport SRCDIR=${WORKDIR}/${P}case "${2}" in unpack) ebuild_unpack ;; compile) ebuild_compile ;; all) ebuild_unpack ebuild_compile ;; *) echo "Please specify unpack, compile or all as the second arg" exit 1 ;;esac

A lot of changes have been made. Let's review the changes below. First, put the compilation and package steps into their respective functions. The function names are ebuild_compile () and ebuild_unpack () respectively (). This is a good step, because the code is becoming more and more complex, and the new function provides a certain degree of attention, making the code more organized. In the first line of each function, explicitly "cd" to the desired directory, because as the Code becomes more modular than linear, the possibility of executing a function in the current working directory is also increased due to negligence. The "cd" command explicitly puts us in the correct position and prevents future errors-this is an important step, especially when deleting files in functions.

In addition, a useful check is added at the beginning of the ebuild_compile () function. Now, it checks to ensure that "$ SRCDIR" exists. If it does not exist, it prints an error message that tells the user to unpackage the file first and then exit. If you want to, you can change this behavior so that when "$ SRCDIR" does not exist, the ebuild script automatically unpacks the source code file. You can use the following code to replace ebuild_compile () to do this:

New Code on ebuild_compile ()

ebuild_compile() { #make sure we're in the right directory if [ ! -d "${SRCDIR}" ] then ebuild_unpack fi cd ${SRCDIR} ./configure --prefix=/usr make }

One of the most obvious changes in the second version of the ebuild script is the new case statement at the end of the Code. This case statement only checks the second command line independent variable and then performs the correct operation based on its value. If you enter:

$ ebuild sed-3.02.ebuild

An error message is returned. Now you need to tell ebuild what to do, as shown below:

$ ebuild sed-3.02.ebuild unpack

Or

$ ebuild sed-3.02.ebuild compile

Or

$ ebuild sed-3.02.ebuild all

If the second command line independent variable is provided, an error message (* clause) is returned, and the program exits.

Code Modularization
Since the code is very advanced and practical, you may want to create several more advanced ebuild scripts to unpack and compile your favorite programs. If you do this, sooner or later you will encounter source code that does not use autoconf ("./configure"), or other scripts that use non-standard compilation processes. Some changes need to be made to the ebuild system to adapt to these programs. But before doing this, you 'd better first think about how to do it.

One of the highlights of hard coding "./configure -- prefix =/usr; make" into the compilation phase is that it works correctly most of the time. However, the ebuild system must adapt to the source code that does not use autoconf or make files normally. To solve this problem, we recommend that the ebuild script perform the following operations by default:

1. If there is a configuration script in "$ {SRCDIR}", execute it as follows:
   ./configure --prefix=/usr
   Otherwise, skip this step.

2. Run the following command:
   make

Since ebuild runs only when configure actually exists, it can now automatically adapt to programs that do not use autoconf but have standard make files. However, what should I do if the simple "make" method does not work for some source code? Some specific code is required to handle these situations to overwrite reasonable default values. To achieve this, the ebuild_compile () function is converted into two functions. The name of the first function (which can be treated as a "parent" function) is still ebuild_compile (). However, there will be a new function named user_compile (), which only contains reasonable default operations:

Split into two functions: ebuild_compile ()

user_compile() { #we're already in ${SRCDIR} if [ -e configure ] then #run configure script if it exists ./configure --prefix=/usr fi #run make make} ebuild_compile() { if [ ! -d "${SRCDIR}" ] then echo "${SRCDIR} does not exist -- please unpack first." exit 1 fi #make sure we're in the right directory cd ${SRCDIR} user_compile}

The reason for doing so may not be obvious, but be patient. Although this code works almost the same way as the previous version of ebuild, you can now do something that was not previously done-you can override user_compile () in the sed-3.02.ebuild (). Therefore, if the default user_compile () does not meet the requirements, you can define a new one in the. ebuild file to include the commands necessary for compiling the package. For example, there is an ebuild file for the e2fsprogs-1.18, which requires a slightly different "./configure" line:

E2fsprogs-1.18.ebuild

#this ebuild file overrides the default user_compile()P=e2fsprogs-1.18A=${P}.tar.gz user_compile() { ./configure --enable-elf-shlibs make}

Now, e2fsprogs will be compiled exactly as expected. However, for most packages, You can omit any custom user_compile () function in the. ebuild file and use the default user_compile () function.

How does the ebuild script know which user_compile () function to use? In fact, this is very simple. In the ebuild script, define the default user_compile () function before executing the e2fsprogs-1.18.ebuild file. If there is a user_compile () in the e2fsprogs-1.18.ebuild, it overwrites the default version defined earlier. If not, use the default user_compile () function.

This is a good tool and we have added a lot of flexibility without any complicated code (if not needed ). I will not talk about it here. However, we should also make similar changes to ebuild_unpack () so that users can overwrite the default unpacking process. This is very convenient if you want to fix or include a file in multiple files. Another good idea is to modify the decommission Code so that it can identify the tar compressed package compressed by bzip2 by default.

Configuration File
So far, we have already talked about a lot of inconvenient bash technologies. Now let's talk about another one. Generally, it is convenient to have a configuration file in/etc. Fortunately, it is easy to do this with bash. You only need to create the following file and save it as/etc/ebuild. conf:

/Ect/ebuild. conf

# /etc/ebuild.conf: set system-wide ebuild options in this file# MAKEOPTS are options passed to makeMAKEOPTS="-j2"

In this example, only one configuration option is included, but you can include more. One of the highlights of bash is that by executing this file, you can perform syntax analysis on it. This design tip can be used in most interpreted languages. After executing/etc/ebuild. conf, define "$ MAKEOPTS" in the ebuild script ". It will be used to allow users to pass options to make. This option is usually used to allow users to tell ebuild to execute parallel make.

What is parallel make? To increase the compilation speed of a multi-processor system, make supports parallel compilation programs. This means that make compiles a specified number of source files simultaneously (to use additional processors in a multi-processor system), rather than compiling only one source file at a time. Enable parallel make by passing the-j # option to make, as shown below: make -j4 MAKE="make -j4" This line of code instructs make to compile four programs at the same time.MAKE="make -j4"The independent variable tells make to pass the-j4 option to any sub-make process started by it.

Here is the final version of the ebuild program:

Ebuild, final version

#!/usr/bin/env bashif [ $# -ne 2 ]then echo "Please specify ebuild file and unpack, compile or all" exit 1fisource /etc/ebuild.confif [ -z "$DISTDIR" ]then # set DISTDIR to /usr/src/distfiles if not already set DISTDIR=/usr/src/distfilesfiexport DISTDIRebuild_unpack() { #make sure we're in the right directory cd ${ORIGDIR} if [ -d ${WORKDIR} ] then rm -rf ${WORKDIR} fi mkdir ${WORKDIR} cd ${WORKDIR} if [ ! -e ${DISTDIR}/${A} ] then echo "${DISTDIR}/${A} does not exist. Please download first." exit 1 fi tar xzf ${DISTDIR}/${A} echo "Unpacked ${DISTDIR}/${A}." #source is now correctly unpacked}user_compile() { #we're already in ${SRCDIR} if [ -e configure ] then #run configure script if it exists ./configure --prefix=/usr fi #run make make $MAKEOPTS MAKE="make $MAKEOPTS" } ebuild_compile() { if [ ! -d "${SRCDIR}" ] then echo "${SRCDIR} does not exist -- please unpack first." exit 1 fi #make sure we're in the right directory cd ${SRCDIR} user_compile}export ORIGDIR=`pwd`export WORKDIR=${ORIGDIR}/workif [ -e "$1" ]then source $1else echo "Ebuild file $1 not found." exit 1fiexport SRCDIR=${WORKDIR}/${P}case "${2}" in unpack) ebuild_unpack ;; compile) ebuild_compile ;; all) ebuild_unpack ebuild_compile ;; *) echo "Please specify unpack, compile or all as the second arg" exit 1 ;;esac

Note that/etc/ebuild. conf is executed at the beginning of the file. In addition, you must note that "$ makeopts" is used in the default user_compile () function ". You may be wondering, does this work?-after all, we reference "$ makeopts" before executing/etc/ebuild. conf that defines "$ makeopts" in advance ". Fortunately, this is fine, because the variable extension only happens when user_compile () is executed. When user_compile () is executed,/etc/ebuild. conf has been executed, and "$ makeopts" has been set to the correct value.

Conclusion
This article has already talked about many bash programming techniques, but it only involves some of the capabilities of bash. For example, Gentoo Linux ebuild not only automatically unpacks and compiles each package, but also:

• If no source code is found in "$ DISTDIR", it is automatically downloaded.
• Verify that the source code is not damaged by using the MD5 message digest.
• If requested, install the compiled application to the file system in use and record all installed files so that packages can be conveniently uninstalled in the future.
• If the request is sent, the compiled application is packaged into a tar compressed package (compressed as you want) so that it can be stored on another computer in the future, or install the CD-based installation (if you build the release CD.

In addition, the ebuild system has several global configuration options that allow you to specify options, such as the optimization flag used during compilation, whether Optional package support (such as gnome and slang) should be enabled by default in those packages that support it ).

It is clear that Bash has far more functions than this series of articles. We hope you have learned a lot about this incredible tool and will inspire you to use Bash to speed up and enhance development projects.

References

• Download the source code tar package (sed-3.02.tar.gz) from ftp://ftp.gnu.org/pub/gnu/sed ).
• InDeveloperWorksRead "Bash instance: Part 1 ".
• InDeveloperWorksRead "Bash instance: Part 1 ".
• Visit the Gentoo Project homepage.
• Visit the bash homepage of GNU.
• View bash online reference manual.

Transferred from:IBM developerWorks Chinese website