Detailed description of the implementation principles of Lua 5.0

Lua 5.0Implementation principle learning notes are described in this article,LuaThe development language of small software was born in a scientific laboratory. It is widely used in game development by many large projects in the world. What do we think?LuaIs it widely used? I believe the answer lies in our design and implementation goals: to provide a simple, efficient, portable, and lightweight embedded scripting language. From January 1, 1993LuaSince its birth, it has become our main goal. These features areLuaReasons for widespread adoption in the industry.

The wide use of language features will inevitably put forward many new requirements. Some new features of Lua are generated based on development needs and use feedback. For example, corotine and garbage collection. These features are important for game development.

In this article, we will focus on some of the main new features in the implementation of Lua 5.0 compared with Lua 4.0.

 
 

  
  
Register-based virtual machine (Register-based virtual machine ):
 
 

Generally, most Virtual Machine implementations tend to be "stack-based execution methods ". The earliest were Pascal's P-machine, to today's Java's JVM and Microsoft. Net environments. However, register-based execution methods are becoming more and more important. For example, the Virtual Machine of Perl6 (Parrot) plans to adopt the register-based method. According to our solutions, Lua 5.0 is the first widely used Virtual Machine Based on register execution. Virtual machines will be discussed in section 7.

New algorithm for optimizing tables used as arrays:

You do not want other scripting languages. Lua does not provide the array type. Lua developers usually use integer-indexed tables to implement arrays. Lua5.0 uses a new algorithm to check whether a table is used as an array. If yes, the values in the table are stored in a real array, instead of adding them to the hash table. This algorithm will be discussed in section 4.

Closures implementation:

Lua 5.0Supports first-class functions within the lexical range. An array-based stack is used to store the data of the current activity to achieve a well-known language difficulty. Lua stores local variables in an array-based stack. When an embedded function is called and beyond the execution range of the current function, the local variables stored in the array-based stack will be copied to the heap. The implementation of Closures will be discussed in section 5.

The addition of coroutines:

Lua 5.0Coroutines is introduced in the language. Although the implementation of conroutine is similar to that of previous versions. To maintain the integrity of this article. We will discuss it in section 6.

The remaining chapters mainly supplement these discussions and provide background knowledge. In section 2, we introduce Lua's design goals and these goals xxxxx. In section 3, we describe how Lua represents a value ". Although these introductions are not the latest features of Lua, we need them to supplement other chapters. At last, we will show a simple performance test example in section 8 and draw some conclusions.

1,LuaOverview of design and implementation

As mentioned in the introduction, Lua's design goals are as follows:

Simple: we use the simplest language and the simplest C code to implement this language. This means a simple syntax and a small number of language structures, xxxxx.

Efficient: we are pursuing the fastest compilation and execution. This means a fast, intelligent, one-scan compiler, and a fast virtual machine.

Portable: We want Lua to run on as many platforms as possible. We achieve that Lua can be compiled on different platforms without any modification. Lua programs can be executed in the Lua interpreter on different platforms without any modification. This means a clean anci c implementation and careful consideration of the porting problem. For example, avoid using non-standard syntax and non-standard library in C language. It also ensures that the compilation can be cleaned in the C ++ compiler. We are pursuing non-warning compilation.

Embedding: Lua is an extension language designed to provide scripting for large programs. This goal means that the existing c api is very simple and powerful, but it depends on most built-in C types.

Low embedding cost: We hope Lua can be easily added to other applications without causing system expansion. This means that compact C code and sophisticated Lua core and extensions are added to existing applications in the form of libraries.

These Goals sometimes conflict with each other. For example, Lua is often used as a data description language to read and write configuration files. It is often seen as a large database with up to a few megabytes of Lua programs ). This means that we need a quick compiler. On the other hand, we hope that the Lua program can be executed as quickly as possible. This means that an intelligent compiler is required. Optimize code for virtual machines. Therefore, the implementation of the compiler needs to find a balance between the two requirements. However, the compiler cannot be too large; otherwise, it will cause the whole application to expand in size. In the current implementation, the size of the compiler accounts for about 30% of the total core. For applications with limited memory, such as embedded systemsCompilerOfLua. The Lua program needs to be pre-compiled into a binary file and loaded by a very small loading module at runtime.

LuaContains a manually compiled lexical scanner and a manually compiled recursive descent syntax analyzer. Before Version 3.0, Lua uses the syntax analyzer generated by yacc. However, manually compiled analyzer is smaller, more efficient, and more portable, and can be completely reimported to ensure multi-thread concurrent execution ). It also provides more complete error information.

Lua CompilerDo not generate intermediate layer code that is common to other compilers. He directly generates VM commands while parsing the program. Although there is no intermediate layer code, the Lua compiler still optimizes the code. For example, the code that delays the generation of basic expression variables and constants. When he parses these expressions, he does not generate commands, but uses a simple structure for representation. Therefore, it is easy to determine whether the operand of an instruction is a constant or a local variable. If yes, you can directly generate these constants or variables into the command, so as to avoid unnecessary "value" copying refer to Section 3 ).

To ensure thatCompilerWhen porting to the platform, Lua gave up some of the frequently used interpreter skills. For example, directly use a thread in the operating system. Instead, it is a standard while-switch branch loop. Although the c code in these places is complex and ugly, it is all to ensure portability.

We think we have achieved our design goals. Lua is now a highly portable language: It can be compiled with an ansi c compiler, from an embedded system to a large host. Lua is truly lightweight: in Linux, an independent interpreter contains all the standard libraries, with less than 100 kb in size and less than kb in key cores. Lua is efficient: third-party testing results show that Lua is the fastest in scripting languages. We also think that Lua is a simple language. Its syntax is like Pascal, and its semantics is similar to Schema.

2. Value Representation

LuaIs a dynamic language: the type is attached to a value rather than a variable. Lua has eight basic types: nil, boolean, number, string, table, function, userdata, and thread. Nil is a tag type with only one value; Boolean has only true and false; Number is a double-precision floating point Number, which is similar to the double type in C language, however, float or long game terminals and small systems can be used to compile the Lua kernel without double support on hardware. String is a byte array with length instructions, therefore, it can be used to store arbitrary binary data. Table is associative arrays, which can be indexed by any value except nil) and can store any value.

Function can be eitherLua FunctionIt can also be based onLuaVirtual Machine call protocol CFunction. Userdata is essentially a pointer to the user's memory block. There are two types of features: heavy-duty, memory blocks are allocated by Lua and released through the garbage collection mechanism; lightweight, memory block allocation and release are all managed by C code. Finally, thread indicates coroutine. All types of "values" are first-clast values: they can be stored in global variables, local variables, and tables. They can be passed as parameters.Function, Which can be returned through the function return value.

LuaThe value in is tagged union. That is, pairs (t, v), t is an integer sign that represents the v type. V is a union in C language. Nil has a single value. Boolean and number are implemented using the "unboxed" value: v is the value in union. This means that the union must be large enough to put a double. String, table, function, thread, and userdata are represented by pointer references. v is a pointer pointing to these structures. These structures share a general head and retain the necessary information for garbage collection. The rest of the structure is different from each category.

 
 

  
  
typedef struct  
  
  
{  
  
  
int t;  
  
  
Value v;  
  
  
} TObject;  
  
  
 
  
  
type union  
  
  
{  
  
  
GCObject *gc;  
  
  
void *p;  
  
  
lua_Number n;  
  
  
int b;  
  
  
}Value; 
 
 

Figure 1 shows the specific implementation of Lua values. TObject implements tagged units (t, v ). Value is the union of values. The value of Nil does not need to be displayed because its tag is sufficient to identify it. Field n is used to indicate number (lua_Number is double by default ). Field B indicates boolean. Field p indicates lightweight userdata. Field gc indicates other types of values, such as string, table, function, and heavy-duty userdata and thread. The content indicated by gc contains the information required for garbage collection.

Because tagged union is used to represent "values", the cost of data replication increases: On a 32-bit hardware that supports 64-bit floating points. TObject occupies 12 bytes. If the hardware alignment the double type by 8 bytes, 16 bytes are required ). To copy a value, you need to copy 3 or 4 machine characters. This is indeed a problem. It is difficult to find a better way to use ansi c. Some Dynamic Language smalltalk80) uses the free bit of each pointer to store type information. This can be implemented on many hardware. Because it is usually 4-byte, 8 bytes aligned according to a certain value ). The last two digits of the pointer are always zero. It can be used for other purposes. However, this method is not portable and cannot be used in ansi c. The standard C language does not guarantee that the pointer type matches any numeric type, nor does it have a standard bitwise Operation Method for the pointer.

Another point to reduce the size of the "value": Keep the tag, but do not put the double into the union. For example, all members are allocated objects in the heap, just like strings ). Python uses this technique, xxxx) but this will lead to a sharp drop in performance. Some improvements can be made. An integer can be expressed as an unboxed value and directly placed in the union, while a floating point number is placed in the heap. In this way, both performance and space can be solved, but the implementation is very complicated.

Like early explanatory languages, such as Snobol and Icon, Lua uses a hash table to store strings: stores unique versions of different strings. The string cannot be changed: Once initialized, it cannot be modified. The Hash value of a string is obtained through bitwise operations and computation. the Hash value is calculated and saved before the string is used for quick comparison and table indexing. If the string is long, the calculation of the Hash value does not traverse the entire string. It is important to improve the performance of long string operations because long strings are quite common in Lua programs. For example, the entire file is often read into the memory and stored as a long string.

3. Tables

Table is the unique data structure in Lua. Table is a key role in both language and language implementation. On the other hand, because there is no other data structure mechanism, the implementation of Table must be efficient and powerful enough.

Figure 2

Table is the associative array in Lua. That is, they can be indexed by any type of value except nil) and can store any type of value. The array size is also dynamically changed according to the operation.

Unlike other scripting languages, Lua does not provide the array type. An array is implemented by using an integer as the index table. Using table to implement array brings some advantages to the language. First, simplicity: there is no need to provide two sets of different operations for table and array respectively. In addition, programmers do not need to select between table and array during development. Because there is only table, there is no other choice. Sparse arrays are easily implemented in Lua. For example, in Perl, If you execute the following statement $ a [1000000000] = 1;, the "out of memory" error will occur. This statement causes the allocation of billions of elements.
In Lua, a = {[1000000000] = 1} only creates a table with only one element.

Before Lua 4.0, tables were implemented using hash tables: all key-value pairs were completely saved.

For an array, the key-value pairs of a = {10, 10, 10} are (1, 10), (2, 10), (3, 10 ).

When using table as an array, Lua 5.0 provides a new optimization algorithm. When it is found that a table is used as an array, it does not store the index of an integer, that is, the above 1, 2, 3. Instead, these values are put into a real array. This greatly improves the access efficiency. More accurately, in Lua 5.0, the table is implemented into a hybrid structure. Contains a hash table and an array. Figure 2 details the implementation. From the perspective of the program, these are transparent. Programmers do not need to worry about whether their data is stored in a hash table or an array. Lua automatically and dynamically adjusts the data in the table. Those keys are placed in the array from 1 to n; those keys are not integers or integers at all but beyond the array size range and are placed in the hash table.

 
 

  
  
xxxxxxxx 
 
 

The hybrid mode has two advantages. First, when an integer index is used for access, the performance is improved because the hash value does not need to be calculated. Second, because no index value needs to be stored, it saves a lot of space. That is, when a table is used as an array, it uses a real array internally, so there is no need to allocate additional storage space for the index value. If a table is regarded as an array only, its hash part is not stored; if it is regarded as an associative array only, its array part will not exist. This ensures that the memory space is not wasted. This is important because a large number of small tables are usually used in Lua programs.

 
 

  
  
xxxxxxxxxxxx 
 
 

4. Functions and Closures

When Lua compiles a function, it generates a prototype that contains VM commands, constants, and debugging information. At runtime, it creates a closure. The closure stores the following content: a reference to its prototype, a reference to the execution environment, and an array containing upvalues references.

A type of function is applied to the lexical range, which causes a problem. Is how to access the local variables of external functions. See the example in figure 3. When add2 is called, it accesses the local variables of its external functions. Then at that time, add has exited. If x is on the stack, the function's stack disappears, and the local variable disappears.

Most Process Languages adopt various methods to avoid similar problems: strictly define the lexical scope e.g ., python), does not provide a class of function e.g ., pascal), or both use e.g ., C ). Function language does not have such a problem. Xxxxxxxxxxx

Lua uses a structure called upvalue to implement closure. Any external local variables can be indirectly accessed through upvalue. Upvalue originally stores a pointer to the single element of the stack. When the local variable needs to be out of the function range, the variable itself will be moved to the upvalue. Because the access to external local variables is indirectly completed through upvalue, this movement is transparent to the program. For a function that declares this local variable, it accesses its own local variable and can be obtained directly from the stack.

If a function generates multiple internal closures, these closures must be able to correctly access the same local variables. To achieve this, Lua maintains a linked list for all upvalue of each stack. When a new closed package is created, it traverses all external local variables. For each local variable, he tries to find a matched upvalue in the linked list. If it can be found, the upvalue will be referenced directly; otherwise, a new upvalue will be created and added to the linked list. Note: The chain table search process is quite fast, because only one upvalue node is created for each local variable. Once upvalue is no longer referenced, it will be automatically reclaimed by the garbage collector.

In the case of multi-layer function nesting, the inner layerFunctionYou can still access the local variables defined by the non-direct outer function. Lua uses flat closure to solve this problem. Xxxxxxxxxxxxxx

5. Threads and Coroutines

From 5.0,LuaAn asymmetric coroutine is also called a semi-symmetric coroutine or semi-coroutine ). Create, resume, and yield are implemented through three Lua library functions. The create FUNCTION accepts a main function value as a parameter, and then creates a coroutine to execute this function. Returns a thread value to indicate the coroutine. Like other types, coroutine is also a type of value ). The resume function starts or continues executing a coroutine. The yield function suspends a coroutine and returns the execution permission toFunctionThe coroutine in which it is located.

Technically, each coroutine has a stack of its own. In fact, each coroutine has two more stacks, but we regard it as an abstract stack ). The coroutine in Lua can be stacked, that is, no matter how deep the nested function is, we can call suspend to suspend the current coroutine. The interpreter simply disables the entire stack of the current coroutine and enables the stack of another coroutine. The program can start a pending coroutine at any time. The garbage collector recycles the stacks of coroutines that are no longer in use.

Summary: DetailsLua 5.0After completing the introduction of the principle learning notes, I hope this article will help you!