How to create your own Shell with Python (top)

I'd like to know how a shell (like bash,csh, etc.) works inside. So to satisfy my curiosity, I used Python to implement a name called YoshShell (Your Own Shell). The concepts described in this article can also be applied to other programming languages.

(Hint: You can find the source code used by Ben Boven here, and the code is published as an MIT license.) On Mac OS X 10.11.5, I tested it with Python 2.7.10 and 3.4.3. It should be able to run in other Unix-like environments, such as Linux and Cygwin on Windows. ）

Let's get started.

Step 0: Project structure

For this project, I used the following project structure.

yosh_project|--yosh   |--__init__.py   |--shell.py

yosh_projectAs the project root directory (you can also name it simply yosh ).

yoshAs a package directory, and __init__.py can make it a package with the same name as the directory (if you do not write Python, you can ignore it.) ）

shell.pyIs our main script file.

Step 1:shell Cycle

When you start a shell, it displays a command prompt and waits for your command input. After you have received the input command and executed it (a later article will explain it in detail), your shell will be back here again and wait for the next instruction.

In shell.py , we start with a simple main function that calls the Shell_loop () function, as follows:

Def shell_loop ():    # Start the Loop heredef main ():    shell_loop () if __name__ = = "__main__":    Main ()

Then, in shell_loop() order to indicate whether the loop continues or stops, we use a status flag. At the beginning of the loop, our shell will display a command prompt and wait for the command input to be read.

Import Sysshell_status_run = 1shell_status_stop = 0def shell_loop ():    status = Shell_status_run while    status = = SH Ell_status_run:        # # shows command prompt        sys.stdout.write (' > ')        sys.stdout.flush ()        # # # Read command input        cmd = Sys.stdin.readline ()

After that, we slice the command (tokenize) input and Execute (execute) (we're about to implement tokenize and execute function).

Therefore, our shell_loop () will be as follows:

Import Sysshell_status_run = 1shell_status_stop = 0def shell_loop ():    status = Shell_status_run while    status = = SH Ell_status_run:        # # shows command prompt        sys.stdout.write (' > ')        sys.stdout.flush ()        # # # Read command input        cmd = Sys.stdin.readline ()        # # # # Shard Command input        cmd_tokens = tokenize (cmd)        # # # Execute the command and get the new state status        = Execute (cmd_ Tokens

This is our entire shell loop. If we use python shell.py launch our shell, it will display a command prompt. However, if we enter the command and press ENTER, it throws an error because we haven't defined the tokenize function yet.

In order to exit the shell, you can try entering ctrl-c. I'll explain later how to exit the shell in an elegant form.

Step 2: Command slicing Tokenize

When the user enters a command in our shell and presses the ENTER key, the command will be a long string containing the command name and its arguments. Therefore, we must slice the string (splitting a string into multiple tuples).

It seems very simple at first glance. We might be able to use cmd.split() a space to split the input. It works on similar ls -a my_folder commands because it splits the commands into a list ['ls', '-a', 'my_folder'] so that we can handle them easily.

However, there are echo "Hello World" echo 'Hello World' cases where arguments are referred to or quoted in single or double quotes. If we use cmd.spilt, we will get a list of 3 tokens instead of ['echo', '"Hello', 'World"'] 2 ['echo', 'Hello World'] .

Fortunately, Python provides a library of names that shlex can help us to split commands magically. (Tip: We can also use regular expressions, but it's not the focus of this article.) ）

Import Sysimport shlex...def tokenize (String):    return Shlex.split (String) ...

We then send these tuples to the execution process.

Step 3: Perform

This is a central and interesting part of the shell. What exactly happened when the shell was executed mkdir test_dir ? (Hint: mkdir is an test_dir executor with parameters that creates a test_dir directory named.) ）

execvpis the first function required for this step. Before we explain what we've execvp done, let's look at its actual effect.

Import Os...def Execute (cmd_tokens):    # # # Execute command    OS.EXECVP (cmd_tokens[0], Cmd_tokens)    # # # Return status to tell in Shell_ Loop waiting for the next command    return Shell_status_run ...

Try running our shell again, enter the mkdir test_dir command, and then press ENTER.

After we hit the ENTER key, the problem is that our shell will exit directly instead of waiting for the next command. However, the catalog was created correctly.

So execvp what has actually been done?

execvpIs exec a variant of the system call. The first parameter is the name of the program. vindicates that the second parameter is a list of program parameters (variable number of parameters). pindicates that an environment variable will be used to PATH search for the given program name. In our last attempt, it will find the program based on our PATH environment variables mkdir .

(There are other exec variants, such as EXECV, EXECVPE, Execl, EXECLP, execlpe; You can get more information from Google.) ）

execReplaces the current memory of the calling process with the new process that is about to run. In our example, our shell process memory is replaced with a mkdir program. Next, mkdir become the master process and create the test_dir directory. Finally, the process exits.

The point here is that our shell process has been mkdir replaced by the process. This is why our shell disappears and does not wait for the next command.

Therefore, we need other system calls to solve the problem: fork .

forkThe new memory is allocated and the current process is copied to a new process. We call this new process a child process , the caller process as the parent process . The child process memory is then replaced with the program being executed. As a result, our shell, the parent process, can be protected from memory substitution.

Let's look at the modified code.

... def execute (cmd_tokens):    # # # Fork A child Shell process    # # # # # # # # If the current process is a subprocess, its ' PID ' is set to ' 0    # # # is the process ID of its child process. PID = Os.fork () if pid = = 0: # # # # # # # # # # # # # # # # # # # # # # # # # #        Replace this subprocess with a program called exec        OS.EXECVP (cmd_tokens[0) C9/>elif pid > 0:    # # # Parent Process        while True:            # # # waits for its child process's response status (to be found in process ID)            wpid, status = Os.waitpid (PID, 0)            # # # # #            # # # # # # # when its child process exits gracefully or when it is signaled, end the wait state            if OS. wifexited (status) or OS. Wifsignaled (status):                break    # # # Returns the status to tell the Shell_loop to wait for the next command    return Shell_status_run ...

When our parent process is called os.fork() , you can imagine that all the source code is copied to the new subprocess. At this moment, the parent and child processes see the same code and run in parallel.

If the running code belongs to a child process, the pid 0 . Otherwise, if the running code belongs to the parent process, it pid will be the process ID of the child process.

When os.execvp called in a child process, you can imagine that all the source code of the child process is replaced with the code of the program being called. However, the code of the parent process is not changed.

When the parent process finishes waiting for the child process to exit or terminate, it returns a status indicating that the shell loop continues.

Run

Now, you can try running our shell and typing mkdir test_dir2 . It should be able to execute correctly. Our main shell process still exists and waits for the next command. Try to execute ls and you can see the directory you have created.

However, there are still some problems here.

First, try to execute cd test_dir2 , then execute ls . It should go into an empty test_dir2 directory. However, you will see that the directory has not changed test_dir2 .

Second, we still have no way to gracefully exit our shell.

We will solve such problems in the next article.