How to use Python to create your own Shell (on)

I 'd like to know how a shell (such as bash and csh) works internally. So to satisfy my curiosity, I used Python to implement a Shell named yosh (YourOwnShell. The concepts introduced in this article can also be applied to other programming languages. (Note: You can find the source code used in this blog post and release the code with the MIT license. On MacOSX10.11.5, I used Python2.7.10 and 3.4.3 for testing. It should be able to run in other... I 'd like to know how a shell (like bash, csh, etc.) works internally. So to satisfy my curiosity, I used Python to implement Yosh(Your Own Shell. The concepts introduced in this article can also be applied to other programming languages.

(Note: You can find the source code used in this blog post and release the code with the MIT license. On Mac OS x 10.11.5, I tested Python 2.7.10 and 3.4.3. It should be able to run in other Unix-like environments, such as Cygwin on Linux and Windows .)

Let's get started.

Step 0: Project structure

For this project, I use the following project structure.

yosh_project|-- yosh   |-- __init__.py   |-- shell.py

yosh_projectIs the project root directory (you can also name ityosh).

yoshIs the package directory, and__init__.pyYou can make it a package with the same name as the package directory (you can ignore it if you don't need to write it in Python .)

shell.pyIs our main script file.

Step 1: Shell Loop

When you start a shell, it will display a command prompt and wait for your command input. After receiving the input command and executing it (the article will explain it in detail later), your shell will return here again and wait for the next command in a loop.

Inshell.py, We will start with a simple main function, which calls the shell_loop () function, as shown below:

def shell_loop():    # Start the loop heredef main():    shell_loop()if __name__ == "__main__":    main()

Thenshell_loop()To indicate whether the loop continues or stops, we use a status flag. At the beginning of the loop, our shell will display a command prompt and wait to read the command input.

Import sysSHELL_STATUS_RUN = 1SHELL_STATUS_STOP = 0def shell_loop (): status = SHELL_STATUS_RUN while status = SHELL_STATUS_RUN: ### display command prompt sys. stdout. write ('>') sys. stdout. flush () ### read command input cmd = sys. stdin. readline ()

Then, we split the command (tokenize) and input and execute (execute) (we will implementtokenizeAndexecuteFunction ).

Therefore, our shell_loop () will be as follows:

Import sysSHELL_STATUS_RUN = 1SHELL_STATUS_STOP = 0def shell_loop (): status = SHELL_STATUS_RUN while status = SHELL_STATUS_RUN: ### display command prompt sys. stdout. write ('>') sys. stdout. flush () ### read command input cmd = sys. stdin. readline () ### split command input pai_tokens = tokenize (cmd) ### execute this command and obtain the new status = execute (pai_tokens)

This is our entire shell loop. If we usepython shell.pyStart our shell and it will display a command prompt. However, if we enter the command and press enter, it will throw an error because we have not definedtokenizeFunction.

To exit shell, enter ctrl-c. Later I will explain how to exit shell in an elegant form.

Step 2: split the tokenize command

When you enter a command in our shell and press the Enter key, the command will be a long string containing the command name and its parameters. Therefore, we must split the string (split a string into multiple tuples ).

At first glance, it seems very simple. We may be able to usecmd.split(), Separated by spaces. It is similarls -a my_folderCommand, because it can split the Command into a list['ls', '-a', 'my_folder']So that we can easily deal with them.

However, there are some similarecho "Hello World"Orecho 'Hello World'Parameters are referenced in single or double quotes. If we use cmd. spilt, we will get a list containing three tags.['echo', '"Hello', 'World"']Instead of a list with two tags['echo', 'Hello World'].

Fortunately, Python providesshlexIt can help us split commands like magic. (Note: We can also use regular expressions, but it is not the focus of this article .)

import sysimport shlex...def tokenize(string):    return shlex.split(string)...

Then we send these tuples to the execution process.

Step 3: Run

This is a core and interesting part of shell. When the shell executesmkdir test_dirWhat happened? (Note:mkdirIstest_dirParameter execution program, used to createtest_dirDirectory .)

execvpIs the first function required in this step. In our explanationexecvpLet's take a look at the actual effect of what we have done.

Import OS... def execute (cmd_tokens): ### execute OS .exe cvp (cmd_tokens [0], cmd_tokens) ### return status to inform shell_loop of waiting for the next command return SHELL_STATUS_RUN...

Run our shell again and entermkdir test_dirCommand, and then press Enter.

After we press the Enter key, the problem is that our shell will exit directly instead of waiting for the next command. However, the directory is created correctly.

Therefore,execvpWhat is actually done?

execvpYessystem callexec. The first parameter is the program name.vThe second parameter is a program parameter list (variable parameter quantity ).pIndicates that environment variables will be used.PATHSearch for the specified program name. In our last attempt, it will be based on ourPATHSearch environment variablesmkdirProgram.

(There are otherexecVariants, such as execv, execvpe, execl, execlp, and execlpe. you can google them for more information .)

execThe current memory of the calling process is replaced with the new process to be run. In our example, the memory of our shell process will be replacedmkdirProgram. Next,mkdirMaster process and createtest_dirDirectory. Finally, the process exits.

The focus here isOur shell process has beenmkdirReplaced by process. This is why our shell disappears and won't wait for the next command.

Therefore, we need other system calls to solve the problem:fork.

forkAllocates new memory and copies the current process to a new process. We call this new processSub-processThe caller process isParent process. Then, the sub-process memory will be replaced with the program to be executed. Therefore, our shell, that is, the parent process, can be free from the risk of memory replacement.

Let's take a look at the modified code.

... Def execute (cmd_tokens): ### forks a sub-shell process ### if the current process is a sub-process, the 'pid 'is set to '0' ### otherwise, if the current process is a parent process, the 'pid' value ### is the process ID of its child process. Pid = OS. fork () if pid = 0: ### sub-process ### replace OS .exe cvp with the program called by exec (pai_tokens [0], cmd_tokens) elif pid> 0: ### parent process while True: ### wait for the response status of its child processes (search by process ID) wpid, status = OS. waitpid (pid, 0) ### when its sub-process exits normally ### or when it is interrupted by the signal, the wait state if OS ends. WIFEXITED (status) or OS. WIFSIGNALED (status): break ### return status to inform shell_loop of waiting for the next command return SHELL_STATUS_RUN...

When our parent process callsos.fork()You can imagine that all source code is copied to a new sub-process. At this moment, the parent process and child process see the same code and run in parallel.

If the running code belongs to a sub-process,pidThe0. Otherwise, if the running code belongs to the parent process,pidThe id of the sub-process.

Whenos.execvpWhen a sub-process is called, you can imagine that all the source code of the sub-process is replaced with the code of the called program. However, the code of the parent process will not be changed.

When the parent process completes and waits for the child process to exit or terminate, it returns a status indicating to continue the shell loop.

Run

Now, you can try to run our shell and entermkdir test_dir2. It should be executed correctly. Our main shell process still exists and waits for the next command. Try to executelsYou can see the created directory.

However, there are still some problems.

First, try to executecd test_dir2And then executels. It should enter an emptytest_dir2Directory. However, you will see that the directory is not changedtest_dir2.

Second, we still cannot exit our shell elegantly.

We will solve such problems in the next article.

The above describes how to create your own Shell (on) in Python. For more information, see other related articles in the first PHP community!