Working with Java virtual machines? No need to run Python code directly from the operating system

Josh Triplett started his speech on Pycon 2015 with a "Smile": porting Python to run without operating system: He and his Intel colleagues let the interpreter run on the Grub bootstrapper, BIOS, or EFI system. Even the rest of the speech, he has a lot of interesting things to say, there are many people to open the eyes of the demonstration.

The initial idea of Python running on the boot loader is to be able to test hardware, such as BIOS, Extensible Firmware Interface (EFI), and Advanced Configuration and Power Interface (ACPI) without having to write a "one-time test project" assembly. Traditionally, Intel has written a number of test programs for DOS (BIOS system) or EFI systems. Both DOS and EFI do not provide environmental protection, so the program can be in memory and hardware to do whatever they need.

He just wanted to use his script to write the test code, "because it's more interesting." He did not want to write too many C-language code, nor did he want to use the GRUB shell, which computes C-class expressions, as before. In fact, he said, "The less C code is written, the easier it is for me."

Over time, the Python porting to GRUB has become a tool for manipulating hardware. It brings us back to the good times of using PEEK and POKE to manipulate hardware on Commodore (or DOS). "Those things are not going to be done now," he said.

Python in Grub

The BIOS implementation Test Suite (BITS), like its name, will run on a variety of firmware grub: 32-bit BIOS or 32/64-bit EFI. He used the original grub or Grub 2. Based on the standard Python interpreter (such as CPython), he apologized: it uses PYTHON2.7. The target audience for this tool is quite familiar with this version of the language. If not, he prefers to migrate to Python 3 later.

There is a read-evaluate-output loop interactive environment [Read-eval-print Loop (REPL)] that gives you full access to the Python language. It includes tab completion, history, and line editing. A "large fragment" of a standard library has been run on bits. Most importantly, the project has added some modules to support the platform: CPU,SMP (symmetric multi-processing), Acpi,efi, and others. Intel has created a test set and used Python to write some tentative tools for using the above modules.

Triplett then switches from the slide to the virtual machine's grub to run a Python interpreter's prompt interface. He entered a two-sentence statement to the interpreter to show that it supports list resolution and any large integer (e.g. bignums).

To get a python interaction environment, GRUB needs to invoke a separate function:

Pyrun_interactiveloop (stdin, "");

It handles all repl[read-execute-output loops, including parsing and executing input, row editing, and so on.

These two parameters simply show where to get input and what to output in Traceback when an exception occurs as a source file. But there is still work to be done in order to call that function in grub.

Because the tools chain and features from the Linux host cannot be used, the project cannot install and configure Python as usual. For GRUB, there is no GNU target statement (for example, Cpu-vendor-od-triple for cross compilation) and target header files can be used. As a result, BITS adds all the Python source files to the GRUB widget system. Essentially, that's just some of the C language files necessary for grub to add Python. Typically, Autoconf creates a apyconfig.h file in the Python widget program to illustrate which features exist on the platform. Instead, this project manually creates apyconfig.h files with lots of "No, I don't have this feature" configuration parameters and a handful of "yes" entries.

Many of the features listed in the Pyconfig.h file are provided by (or not) the operating system, but in this case there is no operating system. Python does need a minimum number of support features, as well as some additional features that are configured. What this project needs to do is to provide any desired and non-existent functionality.

What CPython need?

So, under what circumstances do you really need to run CPython? Triplett provides a large number of examples to prove when to run CPython. Some common file operations are needed, such as: Use Stat () to determine whether a path contains __init__ or whether the file contains __init__. Adding Simpleisatty () (in bits, the file descriptor is less than three returns true) is like going through a seek () execution. To support those features, you have to add a simple file descriptor, because the GRUB file feature uses a struct pointer instead of a descriptor.

When the parser puts a character back in the input stream, Python also needs to use UNGETC (). Instead of adding a character buffer, add "quick hack" to find the latter character. Adding an open encoded qsort () is also the same as using ungetc (); GRUB does not have any support for sorting.

Another aspect of GRUB that is not supported is floating-point operations. The project team found a licensed floating-point operation Library FDLIBM. It does not use any floating-point hardware acceleration, which is very useful in the grub environment. This means that floating-point operations can be used even when the firmware does not fully initialize floating-point arithmetic hardware.

When using Python, we use printf () and sprintf () heavily. In most cases, the Grub version works fine, but does not support the special format of "percent%" (output a "%"). It turns out that Python frequently uses formatted string output.

Strange bugs still exist before they are discovered and repaired.

This project also has some performance problems to solve. First, the startup time is unexpectedly long. This is a painful thing for hardware, but it's also bad on the CPU's analog circuitry ("We don't want to spend three days booting"). Part of the problem comes from the Python interpreter, which calls USESUNGETC () every time it reads a data. GRUB does not have too many cache disks, so all I/O ports have direct access to the disk.

By adding support for the. PYC (Python byte code) file format, this project can reduce many of the parsing work in advance. The version of the host and the GRUB version are compiled at the same time for the Python file to compile at startup.

This has made a substantial improvement, but due to stat (), startup time is still a bit slow. He says that on Linux systems, stat () only takes a few microseconds, but bits can take several milliseconds to build. Increased support for Zipimport allows the project to package all. py files into a single zip file to avoid calls to stat ().

This project would like to have a history and tab automatic complement of REPL (read ﹣ ﹣ output loops), but the general way to get support is to use the gun ReadLine library. This library is provided by the POSIX (Portable Operating system interface) with terminal equipment to provide environmental support. Developers don't want to write a "C code file" to support it, so they use Python to write a read line support instead. CPython's pyos_readlinefunctionpointer is called a C-function set of a new Python function using the C language API.

In order to be able to use other operations and a variety of test suites, it is still imperative to build a dynamic menu of GRUB. GRUB has provided the device with disk and file systems like disk partitions and CD drives (for example: "(hd0)", "(CD)) so BITS adds a" (Python) "Device and a file system that works like a Linux user space (Translator Note: please add ladder). So Python code can access any memory file, such as a menu configuration file under (Python)/menu.cfg, "Even if we don't write more C code," Triplett said.

Access hardware

Now that the goal is to provide a friendly test hardware environment, Python needs to be able to access it. A module called "bits" is added to provide access to a variety of hardware functions, such as CPUID, Special module registers (MSRS), I/O ports, and memory mapped I/O. He showed these abilities in a few lines of code.

>>> Import bits >>> from ctypes import * >>> c = bits.cpuid (0, 0) >>> C Cpuid_result (E ax=0x ..., ebx= ..., ecx= ..., edx= ...)

He introduced the cTYPES module to "manipulate raw memory slices" in the next section of the demo. For those who want to dig deeper, almost every demo can be seen in this YouTube video speech. The CPUID () call returns CPU0 's CPUID, which he then prints out. He asked, "Isn't that funny?" We're getting the processor's register information from Python. "Then he uses Python to explain the result:

>>> buf = (c_uint32*3) (C.EBX, C.edx, c.ecx) >>> (c_char*12). From_buffer (BUF). Value ' Genuineintel '

Three registers contain identifiers that describe the type of processor. He uses the type in the cTYPES module to interpret the three registers (in the previous order) as strings, and the results are shown as processor types.

Intel wants to be able to test highly parallel systems, but grub only knows the information about the CPU that was started. So bits wakes up each CPU in the system and puts them in a sleep loop, using mwait (x86 monitor wait for instructions) to wait for work to come. A particular CPU has a dedicated wake function and execution function.

The project is also going to use Python to get ACPI information and methods. This refers to the implementation of the ACPI Component Architecture (Acpica) and adds it to bits. The python bindings are added because they are all C code. This approach allows Python to invoke any ACPI method-just first convert the argument to an ACPI type and turn the result into a Python type. He used a simple Python program to demonstrate how to display the hardware IDs of all the devices in the virtual machine:

>>> Import ACPI >>> print acpi.dump (' _hid ')

Triplett said he would not continue to delve into the details of bits hardware exploration. He has explained it in more detail in other speeches.

Intel also wants the system to use this firmware instead of the BIOS to access EFI. This expansion nominally means that everything in the EFI is "protocol", each containing the native C language function call. To do so, the external function interface provided via LIBFFI is ported to grub and adds the ability to support the EFI call conversion. Using this approach and Python C-type modules (Python-supplied interfaces and functions of C language types) allow the interpreter to access the EFI. He only used Python to demonstrate how to access the EFI:

>>> Import efi >>> out = efi.system_table. Conout.contents >>> out. Clearscreen (out) [which clears the screens] >>> out. OutputString (out, ' Hello World!rn ') Hello world!

After the EFI is accessed, Python is allowed to use the EFI file protocol to create directories and write files to the EFI file system. This is very helpful because GRUB is only able to read files. Not only that, there is an image output protocol (GOP) capable of reading and writing screen content. As he explains, slides are simple images that are actually displayed by bits and EFI in a notebook. In the context of bits, this slide and demo were made, so the fact that the whole demo was a demo was a round of applause when he said those words. This does not require any new line of C language code.

At last he saved the best demo and started it as python from the frame buffer of the EFI (Extensible Firmware Interface) GOP (frame), and when he knocked through the last few lines of code, it became apparent that the machine began to recognize and computed and displayed a 400x400 size Mandelbrot set ( The grayscale picture of the Mandelbrot collection). "In the EFI graphics protocol only eight lines of Python code show Irregular Graphics (fractal)," he said to the people clapping around. It takes about 15 seconds to paint the image, which is a bit slow, he says, but it's not a python problem, but a floating-point operation using pure software.

At the end of the conversation, Triplett points out that there is no interrupt handling hook function (hook) in bits (Background Intelligent Transfer Service), but this is easily added. He says it's possible to add Python code to bits, like the Mirage OS (and other "similar operating systems"), and it doesn't make much difference. The next interesting item on the to-do list is to add the Python-bound EFI TCP network protocol and hook to Python socket module to see if you can run a simple HTTP service (simplehttpserver) under that environment (BITS). This allows you to add a "network REPL (web REPL)" to the BITS environment.