Compared with Java Virtual Machine? Directly run Python code without the need for the operating system

Compared with Java Virtual Machine? Directly run Python code without the need for the operating system

Josh Triplett started his speech on PyCon 2015 with a "Smile": porting Python to free him from operating the system: he and his Intel colleagues asked the interpreter to run on the GRUB boot program, BIOS, or EFI system. Even the rest of the speech has not been missed. He has many interesting things to talk about and many eye-opening demonstrations.

The initial idea of running Python on Boot Loader is to be able to test hardware, such as BIOS, extended firmware interface EFI) and advanced configuration and Power Interface ACPI ), instead of writing some "one-time test project" assembly. Traditionally, Intel has written a lot of test programs for DOSBIOS systems) or EFI systems. Both DOS and EFI do not provide environment protection, so that the program can stay in the memory and hardware to do what they need.

He just wanted to use his script to write the test code. "This is interesting ". He neither wants to write too much C-language code, nor wants to use the GRUB shell that can calculate C-class expressions as before. In fact, he said, "The less I write C code, the easier I will be ".

Over time, Python transplanted to GRUB has become a powerful tool to manipulate hardware. It also brings us back to the good times of using PEEK and POKE to manipulate hardware on Commodore 64 (or DOS. "The current hardware cannot be done," he said.

 

PYTHON in GRUB

BIOS Implementation Test Suite (BITS), as its name suggests, will run in GRUB on multiple firmware types: 32-bit BIOS or 32/64-bit EFI. It uses the original GRUB or GRUB 2. Standard PYTHON interpreter (such as CPython), but he apologized: it uses PYTHON2.7. The target audience of this tool is quite familiar with the language of this version. If not, he prefers to migrate to Python 3 later.

There is a "read-evaluate-output loop" Interactive Environment [read-eval-print loop (REPL)] that gives you full access to the Python language. It includes Tab completion, history, and row editing. A large number of fragments of a standard library have been running on BITS. Most importantly, this project has added some modules that support the platform: CPU, SMP (supported Ric multi-processing), ACPI, EFI, and others. INTEL has created a test set and written some test tools using the above modules using Python.

Triplett then switches from the slide to the prompt interface for running a Python interpreter in GRUB of the virtual machine. He entered two statements to the interpreter to show that it supports list parsing and any large integer such as bignums ).

To obtain a python interaction environment, GRUB needs to call a separate function:

 
 

  
  
PyRun_InteractiveLoop(stdin, "<stdin>"); 
 
 

It will process all REPL [Read-execute-output loop], including parsing and executing input, row editing, and so on.
These two parameters simply indicate where to obtain the input and what to output in traceback when an exception occurs as the source file. But there is still some work to do to call that function in GRUB.

Because tool chains and features from Linux Hosts cannot be used, this project cannot install and configure Python as usual. For GRUB, there is no GNU target Declaration, for example, the cpu-vendor-od-triple used for cross-compilation, and the target header file can be used. Therefore, BITS adds all Python source files to the GRUB component system. Essentially, it is only some C-language files necessary to add Python to GRUB. Generally, autoconf creates the apyconfig. h file in the Python component program to describe which functions exist on the platform. On the contrary, this project manually creates a large number of "No, I don't have this function" configuration parameters and a small number of "yes" entries.

Many functions listed in the pyconfig. h file are provided by or not by the operating system, but in this case, there is no operating system. Python requires a minimum of support functions and some additional configuration features. What this project needs to do is to provide any function that is desired and does not exist.

What does CPython need?

So, under what circumstances do you really need to run CPython? Triplett provides a large number of instances to prove when CPython needs to be run. Some common file operations are required. For example, use stat () to determine whether a path contains _ init _ or whether the file contains _ init __. Add simpleisatty () in bits. If the file descriptor is less than three, true is returned.) It is like executing a seek. To support those functions, you have to add a simple file descriptor table because GRUB's file functions use struct pointers instead of descriptors.

When the parser puts a character back in the input stream, Python also needs to use ungetc (). Instead of adding a character buffer, add "quick hack" to find the next character. The same applies to qsort () with open encoding. ungetc () is used. GRUB does not support sorting.

Another aspect that GRUB does not support is floating point operations. The project team found a licensed Floating Point Library FDLIBM. It does not use any floating point hardware acceleration, which is very useful in GRUB environments. This means that the floating point operation can be used even if the firmware does not fully initialize the floating point computing hardware.

When using Python, we use printf () and sprintf () in large quantities (). In most cases, the GRUB version works well, but the special format "%" output is not supported yet. It turns out that Python frequently uses formatted string output.

Before being detected and fixed, the odd BUG still exists.

This project also has some performance problems to solve. First, the startup time is unexpectedly long. This is very painful for the hardware, but it is also very bad on the CPU analog circuit "We don't want to spend three days to boot "). Part of the problem comes from the Python interpreter, which calls usesungetc () every time it reads a data (). GRUB does not have many high-speed cache disks, so all I/O Ports access the disks directly.

By adding support for the. pyc (Python byte code) file format, this project can reduce a lot of syntax analysis work in advance. The host version is compiled at the same time as the GRUB version. It is used for compiling Python files at startup.

This substantially improves, but the start time is still slow due to stat. He said that in Linux, stat () takes only a few microseconds, But BITS takes several milliseconds. The added support for zipimport allows the project to package all. py files into a single ZIP file to avoid calling stat.

This project requires REPL read-evaluate-output loop with history and tab auto-completion), but generally the supported method is to use the Readline library of GUN. This library is supported by POSIX with terminal devices that can port operating system interfaces. Developers do not want to write a "C code file" to support it, so they use Python to write a read line support. CPython's PyOS_ReadlineFunctionPointer is called a set of C functions for new Python functions using C language APIs.

To be able to use other operations and a variety of Test suites, it is still urgent to build a dynamic menu for GRUB. GRUB already provides disks and file systems for devices like disk partitions and CD drives such as "(hd0)", "(cd)"). Therefore, BITS adds a "(python) "device and a file system working like a Linux user space. Note: Please add a ladder if it cannot be opened ). Therefore, Python code can access any memory file, such as the menu configuration file under (python)/menu. cfg, "even if we didn't write more C code," said Triplett.

Access hardware

Since the goal is to provide a friendly Testing hardware environment, Python needs to be able to access it. A module called "bits" is added to provide access to various hardware functions, such as CPUID, special module registers (MSRs), I/O ports, and memory ing I/O. He demonstrated these capabilities with several lines of code.

 
 

  
  
>>> import bits  
  
  
>>> from ctypes import *  
  
  
>>> c = bits.cpuid(0, 0)  
  
  
>>> c  
  
  
cpuid_result(eax=0x..., ebx=..., ecx=..., edx=...) 
 
 

He introduced the ctypes module to "operate original memory slices" in the next section ". For those who want to dig deeper, almost all the demos can be seen in this YouTube video speech. When cpuid () is called, The CPUID of CPU0 is returned and printed out. He asked: "Is this interesting? We are getting the Processor register information from Python ." Next, he uses Python to explain the result:

 
 

  
  
>>> buf = (c_uint32*3)(c.ebx, c.edx, c.ecx)  
  
  
>>> (c_char*12).from_buffer(buf).value  
  
  
'GenuineIntel' 
 
 

The three registers contain identifiers describing the processor type. He uses the types in the ctypes module to re-interpret the information of the three registers in the previous order in the form of strings. The result is displayed as the processor type.

Intel wants to test a highly parallel system, but GRUB only knows the CPU information that has been started. Therefore, BITS wake up each CPU in the system, put them into a sleep loop, and use the MWAITx86 monitor to wait for instructions) and wait for the arrival of work. Specific CPUs have dedicated wake-up functions and execution functions.

This project is also prepared to use Python to obtain ACPI information and methods. This reference refers to the implementation of ACPI component architecture (ACPICA) and adds it to BITS. Python binding is added because all the code is C. This method allows Python to call any ACPI method-as long as the parameter is first converted to the ACPI type and the result is converted to the Python type. He used a simple Python program to demonstrate how to display the hardware IDs of all devices in the virtual machine:

 
 

  
  
>>> import acpi  
  
  
>>> print acpi.dump('_HID') 
 
 

Triplett claims that he will not continue to explain the details of BITS hardware exploration. He has explained it in more detail in other speeches.

Intel also wants the system to use this firmware instead of the BIOS to access EFI. In name, this extension means that everything in EFI is a "protocol", and every one includes native C language function calls. To achieve this, the external function interface provided by libffi is transplanted to GRUB and added the function that supports EFI call conversion. This method is used with the c language interfaces and functions provided by Python c modules) to allow the interpreter to access EFI. He only demonstrated how to access EFI using Python:

 
 

  
  
>>> import efi  
  
  
>>> out = efi.system_table.ConOut.contents  
  
  
>>> out.ClearScreen(out)  
  
  
[ which clears the screen ]  
  
  
>>> out.OutputString(out, 'Hello world!\r\n')  
  
  
Hello world! 
 
 

After accessing EFI, Python can use the EFI file protocol to create directories and write files to the EFI file system. This is very helpful because GRUB can only read files. In addition, there is an image output protocol (GOP) that can read and write screen content. As he explained, slides are simple images that are actually displayed through BITS and EFI in a notebook. In the BITS environment, this slide and demo were made. Therefore, in fact, the whole demonstration is a demo. When he said these words, he applauded. In this way, no new lines of C code are required.

Finally, he saved the demo that he thought was the best, and started it as Python in the frame buffer of the EFI extensible firmware interface) GOP image group. When he finishes the last few lines of code, it is obvious that the machine began to recognize, calculate and display a X-sized Mandelbrot set mandebrot set) grayscale image. "In the EFI graphics Protocol, only eight lines of Python code are used to show irregular graph Fractal," he applauded )". It takes about 15 seconds to plot the image. It's a little slow, he said. It's not a Python problem, but it's because it uses pure software for floating point operations.

At the end of the conversation, Triplett pointed out that there is no hook function for Interrupt Processing in the BITS Background Intelligent transmission service), but this is easy to add. He said that Python code can also be added to BITS on Mirage OS and other "similar operating systems", and there is no big difference. The "next interesting project on the to-do list" is to add the efi tcp network protocol bound to Python and hook it to the Python socket module to see if BITS can be in that environment) run a simple HTTP service SimpleHTTPServer ). In this way, you can add a "Network REPLweb REPL)" to the BITS environment.

Python without an operating system