Run Python code directly without the OS and virtual machines

Josh Triplett began his speech at Pycon 2015 with a "laugh point": porting Python without operating system: He and his Intel colleagues let the interpreter run on the Grub boot, BIOS, or EFI system. He has a lot of interesting things to say, and a lot of demos that are open to the eyes of the audience.

The original idea of Python running on boot loader was the ability to test hardware, like BIOS, Extensible Firmware Interface (EFI), and Advanced Configuration and Power Interface (ACPI) without having to write some "one-time Test project" assemblies. Traditionally, Intel has written a number of test programs for DOS (BIOS system) or EFI systems. Both DOS and EFI do not provide environmental protection, so that the program can be stationed in memory and hardware to do everything they need.

He just wanted to write the test code with his script, "because it's interesting." He doesn't want to write too many C-language code, and he doesn't want to use the GRUB shell that calculates C-class expressions as before. In fact, he said, "The less C code writes, the easier it is for me."

Over time, Python ported to GRUB has become a powerful tool for manipulating hardware. It also brings us back to the wonderful time of using PEEK and POKE to manipulate hardware on Commodore (or DOS). "Those things are now hardware devices that can't be done," he said.

Python in Grub

The BIOS implementation Test Suite (BITS), as its name, will run in grub on a variety of firmware: 32-bit BIOS or 32/64-bit EFI. He uses the original grub or Grub 2. A standard-based Python interpreter (such as CPython), but he apologized: it uses PYTHON2.7. The target audience for this tool is quite familiar with this version of the language. If not, he prefers to migrate to Python 3 later.

There is a read-evaluate-output loop interactive environment [Read-eval-print Loop (REPL)] that gives you full access to the Python language. It includes tab completion, history, and line editing. A standard library of "massive fragments" has been running on bits all the time. Most importantly, this project has added some platform-supported modules: CPU,SMP (symmetric multi-processing), Acpi,efi, and others. Intel has created a test set and used Python to write some tentative tools for using the above modules.

Triplett then switches from the slide to the virtual machine's grub to run a Python interpreter in the prompt interface. He enters two sentences to the interpreter to show that it supports list parsing and arbitrary large integers (such as: bignums).

To get a python interactive environment, Grub needs to invoke a separate function:

" <stdin> "

It handles all repl[read-execute-output loops], including parsing and execution of input, line editing, and so on.
These two parameters simply indicate where to get the input and what to output in Traceback when an exception occurs as a source file. But there is still work to be done in order to invoke that function in grub.

Because the toolchain and features from the Linux host cannot be used, the project cannot install and configure Python as usual. For GRUB, there is no GNU target declaration (for example, cpu-vendor-od-triple for cross-compiling) and the target header file can be used. Therefore, BITS adds all the Python source files to the GRUB artifact system. Essentially, it's just some of the C-language files that grub needs to add Python. Typically, autoconf will create a apyconfig.h file in the Python artifact program to illustrate what features exist on the platform. Instead, this project manually created a lot of apyconfig.h files "No, I don't have this feature" configuration parameters and a handful of "yes" entries.

Many of the features listed in the Pyconfig.h file are (or are not) provided by the operating system, but in this case there is no operating system. Python does require a minimum number of support features, as well as some additional features that are configured. What this project needs to do is to provide any functionality that is desired and does not exist.

What does CPython need?

So, under what circumstances do you really need to run CPython? Triplett provides a number of examples to prove when you need to run CPython. Some common file operations are required, such as using stat () to determine whether a path contains __init__ or whether the file contains __init__. Add Simpleisatty () (in bits, the file descriptor is less than three to return true) as if going through a seek () execution. To support those features, you have to add a simple file descriptor, because the GRUB file feature uses struct pointers instead of descriptors.

When the parser puts a character back in the input stream, Python also needs to use UNGETC (). Instead of adding a character buffer, add "quick hack" to look for the next character. Adding an open encoded qsort () is the same as using ungetc (); GRUB does not have any support for sorting.

Another aspect that GRUB has not yet supported is floating-point arithmetic. The project team found a licensed floating-point arithmetic library, FDLIBM. It does not use any floating-point hardware acceleration, which is very useful in the grub environment. That means you can use floating-point arithmetic even when the firmware does not fully initialize the floating-point arithmetic hardware.

When using Python, we use printf () and sprintf () a lot. In most cases, the Grub version works fine, but the special format for "percent" (output a "%") is not yet supported. It turns out that Python frequently uses formatted string output.

Strange bugs still exist until they are discovered and repaired.

There are some performance issues that need to be addressed in this project. First, the startup time is unexpectedly long. This is a very painful thing for hardware, but it's also bad on the analog circuitry of the CPU ("We don't want to spend three days booting"). Part of the problem comes from the Python interpreter, which calls USESUNGETC () every time it reads a data. GRUB does not have too many cache disks, so all I/O ports have direct access to the disk.

By adding support for the. PYC (Python byte code) file format, this project can reduce many of the parsing work in advance. The version of the host and the version of GRUB are compiled at the same time for the Python file to compile at startup.

This makes a substantial improvement, but the startup time is still somewhat slow due to stat (). He says stat () takes only a few microseconds on a Linux system, but the bits version takes a few milliseconds. Increased support for Zipimport enables the project to package all. py files into a single zip file to avoid the call to stat ().

This project wants to do a repl with history and Tab AutoComplete (read ﹣ evaluation ﹣ output loops), but generally the way to get support is to use the ReadLine Library of the gun. This library is supported by the POSIX (Portable Operating system interface) with end devices. Developers don't want to write a "C code file" to support it, so they use Python to write a read-line support instead. CPython's pyos_readlinefunctionpointer is called a C-function collection of a new Python function using the C language API.

In order to be able to use other operations and a variety of test suites, it is still urgent to build a dynamic menu of GRUB. GRUB has provided disk and file systems for devices like disk partitions and CD drives (for example: "(hd0)", "(CD)") so BITS adds a "(Python)" Device and a file system that works like a Linux user space (translator Note: Do not open please add a ladder). So Python code can access any memory file, such as a menu configuration file under (Python)/menu.cfg, "Even if we don't write more C code," Triplett says.

Accessing Hardware

Since the goal is to provide a friendly test hardware environment, Python needs to be able to access it. A module called "bits" is added to provide access to various hardware functions such as CPUID, Special Module register (MSRs), I/O port, and memory mapped I/O. He shows these capabilities in a few lines of code.

Import bits    from Import *  >>> c = bits.cpuid (0, 0)  >>> C  cpuid_result (eax

He introduced the cTYPES module to "manipulate raw memory slices" in the next section of the demo. For those who want to dig deeper, almost all demos can be seen in this YouTube video presentation. The CPUID () call returns the CPUID of CPU0, which he then prints out. He asked, "Isn't that fun?" We are getting the processor's register information from Python. He then uses Python to explain the result:

>>> buf = (c_uint32*3) (C.EBX, C.edx, c.ecx)  >>> (c_char*12). From_buffer (BUF). Value  'genuineintel'

Three registers contain identifiers that describe the type of processor. He uses the type in the cTYPES module to re-interpret the information in the form of a string of these three registers (in the previous order), and the results are displayed as processor types.

Intel wants to be able to test highly parallelized systems, but Grub knows only the information about the CPU that was started. So bits wakes up each CPU in the system and puts them into a sleep loop, using mwait (x86 monitor waiting for instructions) to wait for work to come. Specific CPUs have special wake-up functions and execute functions.

The project is also ready to use Python to get ACPI information and methods. This references the implementation of the ACPI component schema (Acpica) and adds it to bits. Because it's all C code, the Python bindings are added. This approach allows Python to invoke any ACPI method-simply converting the parameter to an ACPI type and turning the result into a Python type. He used a simple Python program that demonstrated how to display the hardware IDs of all the devices in a virtual machine:

Import ACPI   print acpi.dump ('_hid'

Triplett claims that he will not go into the details of bits hardware exploration. He has explained it more thoroughly in other speeches.

Intel also wants the system to use this firmware instead of the BIOS to access EFI. This extension nominally means that everything is "protocol" in EFI, each of which contains native C-language function calls. To do this, the external function interface provided by LIBFFI is ported to grub and adds the ability to support the EFI call transformation. Using this approach and Python C-type modules (Python-provided C-type interfaces and functions) allows the interpreter to access EFI. He only used Python to demonstrate how to access EFI:

Import EFI   >>> out = efi.system_table. Conout.contents  >>> out . Clearscreen (out)  [which clears,the screen]'Hello world!\r\n')  

After accessing EFI, allow Python to use the EFI file protocol to create directories and write files to the EFI file system. This is very helpful because GRUB is only able to read files. Not only that, there is an image output protocol (GOP) capable of reading and writing screen content. As he explains, the slides are simple images that are actually displayed by bits and EFI in the notebook. In the context of bits, this slideshow and demo are made, so the fact that the entire presentation is a demo is a round of applause when he says these words. Doing so does not require any new C-language code.

Finally he saved the best demo and started it as python from the frame buffer of the EFI (Extensible Firmware Interface) GOP (picture Group), and when he finished the last few lines of code, it became clear that the machine began to recognize, calculate and display a 400x400-sized Mandelbrot A grayscale picture of Set (Mandelbrot collection). "In the EFI Graphics protocol, the irregular Graphics (Fractal) are displayed in only eight lines of Python code," he said to the people who applauded around him. It takes about 15 seconds to draw the image, which is a bit slow, he says, not a python problem, but a floating-point operation using pure software.

At the end of the conversation, Triplett pointed out that there is no interrupt processing hook function in bits (Background Intelligent Transfer Service), but this is easily added. He said that in the same way that the Mirage OS (and other "similar operating systems") can add Python code to bits, and it doesn't make much of a difference. The next interesting item on the backlog list is to add the Python-bound EFI TCP network protocol and hook to the Python socket module to see if you can run a simple HTTP service (simplehttpserver) under that environment (BITS). This allows you to add a "network REPL (web REPL)" to the BITS environment.

Run Python code directly without the OS and virtual machines