Memory ing function MMAP in Linux and sample code

SELF: http://hi.baidu.com/flying5/blog good blog, you can go to have a look
Linux MMAP File Memory ing mechanism

MMAP: Memory Map

When talking about the concept of file ing, it is inevitable that virtual memory (SVR 4 VM) is involved. In fact,File ing is the central concept of virtual storage.

,
On the one hand, file ing provides users with a set of measures, as if users map files to a part of their own address space and use simple memory access commands to read and write files. On the other hand,
It can also be used in the basic organization mode of the kernel. In this mode, the kernel regards the entire address space as a ing of different objects such as files,
First, open the file with an open system call, and then use the read, write, lseek and other calls for sequential or instant I/O. This method is very inefficient,
Each I/O operation requires a system call. in addition, if several processes access the same file, each process needs to maintain a copy in its own address space, wasting the memory space.
However, if some mechanisms can be usedMap the page to the address space of the process

That isFirst, you can create a ing by generating some memory management data structures. When a process accesses the page, a page disconnection occurs. The kernel reads the page into the memory and updates the page table to the page.

And this method is very convenient for sharing the same copy.

VM is designed in an object-oriented way. The object here refers to the memory object:A memory object is a software abstract concept.
,
It describes the ing between the memory zone and the backup storage. The system can use multiple types of backup storage, such as swap space, local or remote files, and frame cache.
The VM system processes them in a unified manner and uses the same set of operations, such as reading pages or writing back pages. Different backup storage can implement these operations in different ways,
The system defines a unified interface, and each backup storage provides its own implementation method. In this way, the address space of a process is considered as a set of mappings mapped to different data objects.
All valid addresses are those mapped to Data Objects. These objects provide persistent backup storage for the pages mapped to them.Ing allows users to directly address these objects

.

It is worth mentioning that,
The VM architecture is independent of Unix systems. All UNIX system semantics, such as the body, data, and stack can be built on the basic VM system. At the same time,
The VM architecture is also independent of storage management. Storage Management is implemented by the operating system. For example, what kind of swap and request paging algorithms are adopted,
Whether to use the segmentation or paging mechanism for storage management, how to convert virtual addresses into physical addresses, and so on (Linux is a three level page
Table Mechanism). These are irrelevant to the concept of memory objects.

The following describes the implementation of VM in Linux.

A process should includeMm_struct (memory manage struct ),

This structure is an abstract description of the virtual address space of a process. It contains some management information about the virtual space of a process: start_code, end_code,
Start_data, end_data, start_brk, end_brk, and so on. In addition,
There is also a pointer to the process virtual memory area table (vm_area_struct: virtual memory area), which is arranged in the order of virtual addresses.
In a Linux Process, the address space is divided into multiple zones (VMA). Each zone (VMA) corresponds to a continuous area in the virtual address space,
VMA is an independent entity that can be shared and protected,VMA is the memory object mentioned above.

.
The following is the structure of vm_area_struct, where the first half is public and some data members irrelevant to the type, such as the pointer to mm_struct,
Address range, etc. The second half is a type-related member. The most important part is a pointer to vm_operation_struct vector table vm_ops,
Vm_pos vector table is a set of virtual functions that define interfaces unrelated to the VMA type. Each specific subclass, that is, each VMA type must implement these operations in the vector table.
The following operations are included: open, close, unmap, protect, sync, nopage, wppage, and swapout.

Struct vm_area_struct {

/* Public, irrelevant to VMA type */

Struct mm_struct * vm_mm;
Unsigned long vm_start;
Unsigned long vm_end;
Struct vm_area_struct * vm_next;
Pgprot_t vm_page_prot;
Unsigned long vm_flags;
Short vm_avl_height;
Struct vm_area_struct * vm_avl_left;
Struct vm_area_struct * vm_avl_right;
Struct vm_area_struct * vm_next_share;
Struct vm_area_struct ** vm_pprev_share;

/* Type-related */

Struct vm_operations_struct * vm_ops;
Unsigned long vm_pgoff;
Struct file * vm_file;
Unsigned long vm_raend;
Void * vm_private_data;
};
Vm_ops: open, close, no_page, swapin, swapout ......

After introducing the basic concepts of VM, we can talk about MMAP and munmap system calls.MMAP call is actually the creation process of a memory object VMA.

The MMAP call format is:

Void * MMAP (void * Start, size_t length, int Prot, int flags, int FD, off_t offset );

Where start is a ing
Address. Length indicates the ing length. If map_fixed of flags is not set, this parameter is usually ignored,
In the process address space, find the first idle area with the correct length. FD is the file handle of the ing file, offset is the offset address in the ing file, and Prot is the ing protection permission,
It can be prot_exec, prot_read, prot_write, prot_none, and flags,
It can be map_fixed, map_private, map_shared,
This parameter must be specified as one of map_private and map_shared,Map_private is used to create a copy-on-write ing. That is to say, if multiple processes are mapped to one file at the same time, only the same storage page is shared when the ing is created, however, if a process attempts to modify the page content, it will copy a copy to the process for private use, and any modifications to the process will not be visible to other processes.

Map_shared uses the same copy regardless of whether it is modified or not. Any modification made to the page by any process is visible to other processes.

The MMAP System Call implementation process is

:
1. First, locate the file to be mapped through the file system;
2. Permission check. The ing permission does not exceed the file opening method. That is to say, if the file is opened in read-only mode, a writable ing cannot be created;
3. Create a VMA object and initialize it;
4. Call the mmap function of the ing file to assign values to the vm_ops vector table;
5. link the VMA to the VMA linked list of the process. If it can be combined with the VMA before and after, it will be merged;
6. If vm_locked (the ing area is not swapped out) is required for ing, a page missing request is sent to read the ing page into the memory.

Munmap (void * Start, size_t length ):

This call can be considered as an inverse process of MMAP. it will close the ing of a segment of length starting from start in the process. If the region does not exactly correspond to a VMA, it may split several or more VMA.

Msync (void * Start, size_t length, int flags ):

Write back the modification of the ing area
In the backup storage, page write-back is not guaranteed during munmap. If msync is not called, modifications to the ing zone may be lost after munmap.
Among them, flags can be ms_sync, ms_async, ms_invalidate, and ms_sync must be returned after the write-back is complete,
Ms_async returns immediately after sending a write-back request. ms_invalidate uses the write-back content to update other Mappings of the file.
This system call is done by calling the sync function of the ing file.

BRK (void * end_data_segement ):

Extend the data segment of a process
End_data_segement specifies the address. This system call is similar to the MMAP implementation method. It also generates a VMA and then specifies its attributes.
However, you need to check the validity of the address before this operation, for example, whether the address is greater than mm-> end_code,
Check whether other VMA exists between end_data_segement and mm-> BRK. The VMA ing file generated by BRK is empty,
This is similar to the VMA generated by anonymous ing. We will not introduce anonymous ing further. The library function malloc is implemented through BRK.

 

========================================================== ========

Linux provides the memory ing function MMAP, which maps the file content to a memory segment (specifically, virtual memory). By reading and modifying this memory segment, to read and modify files, Let's first look at the MMAP function declaration:

• Header file:
  • <Unistd. h>
  • <Sys/Mman. h>
• Prototype: void * MMAP (void * ADDR, size_t length, int Prot, int flags, int FD, off_t offsize );
• Returned value: if the operation succeeds, the start address of the ing area is returned. If the operation fails, map_failed (-1) is returned ).
• Parameters:
  • ADDR: Specifies the starting address of the ing, which is usually set to null and specified by the system.
  • Length: how long the file length is mapped to the memory.
  • Prot: Protection Method of the ing area, which can be:
    • Prot_exec: The ing area can be executed.
    • Prot_read: The ing area can be read.
    • Prot_write: The ing area can be written.
    • Prot_none: The ing area cannot be accessed.
  • Flags: the features of the ing area, which can be:
    • Map_shared: write data to the ing area will be copied back to the file, and other processes that map the file can share.
    • Map_private: write operations on the ing area will generate a copy-on-write operation. modifications made to this area will not be written back to the original file.
    • In addition, several other flags are not frequently used. For details, refer to the Linux C function description.
  • FD: file descriptor returned by open, representing the file to be mapped.
  • Offset: the offset starting from the file. It must be an integer multiple of the page size, usually 0, indicating the ing from the file header.

The following describes the steps for memory ing:

• Open the file with an open system call and return the descriptor FD.
• Use MMAP to create a memory ing and return the ing first address pointer start.
• Perform Various operations on the ing (file), display (printf), and modify (sprintf ).
• Use munmap (void * Start, size_t lenght) to disable memory ing.
• Use the close system call to close the file FD.

Note:

When modifying a ing file, you can only modify the original length, but cannot increase the file length, because the memory has been allocated.

 

Introduction to the Linux-MMAP function (for example)

 

MMAP function is a system call in Unix/Linux. Let's take a look at the introduction to MMAP in section 12.2 of UNIX netword programming:
The MMAP function maps either a file or a POSIX shared
Memory object into the address space of a process. We use this function
For three purposes:
1. With a regular file to provide memory-mapped I/O
2. With special files to provide anonymous memory Mappings
3. With shm_open to provide POSIX shared memory between unrelated Processes

MMAP system calls are not designed completely for Memory Sharing. It provides different access methods for common files. A process can perform operations on common files like read/write memory. While
POSIX or System V shared memory IPC is purely used for sharing purposes. Of course, MMAP () shared memory is also one of its main applications.
MMAP system calls enable shared memory between processes by ing the same common file. After a common file is mapped to the process address space, the process can access the file like accessing the common memory without calling read (), write (), and other operations.

MMAP is widely used in our programs, and MMAP is used to access files like accessing common memory. It has been proved that calling MMAP is much faster than using conventional methods to frequently access a file and move the pointer back and forth.
Let's take a look at MMAP's definition:
Void * MMAP (void * ADDR, size_t Len, int Prot, int flags, int FD, off_t offset );

The FD parameter is the description of the file to be mapped to the process space. It is generally returned by open (). At the same time, FD can be specified as-1. In this case, map_anon In the flags parameter must be specified to indicate
The row is anonymous ing (not involving specific file names, avoiding File Creation and opening. Obviously, it can only be used for Kinship-related inter-process communication ).

Len is the number of bytes mapped to the address space of the calling process. It starts from the offset byte at the beginning of the mapped file.

The prot parameter specifies the access permission for the shared memory. The following values can be obtained: prot_read (readable), prot_write (writable), prot_exec (executable), and prot_none (inaccessible ).

Flags are specified by the following common values: map_shared, map_private, and map_fixed. Among them, map_shared and map_private are mandatory, while map_fixed is not recommended.
If map_shared is specified, the modifications made to the mapped memory also affect the file. If map_private is used, the modifications made to the mapped memory are only visible to this process and have no effect on the files.

The offset parameter is generally set to 0, indicating that the ing starts from the file header.

The ADDR parameter specifies that the file should be mapped to the starting address of the process space. Generally, a null pointer is specified. At this time, the task of selecting the starting address is left to the kernel for completion. The Return Value of the function is the address mapped from the last file to the process null. The starting address of a process operation can be the valid address of this value.

Take a look at the figure below (from section 2, section 12.2, Unix netword programming) to further impress MMAP:

Here, we will not detail MMAP parameters. You can refer to the MMAP manual page or section 12.2 of UNIX netword programming for further information. Finally, let's end this section with an example. As mentioned in section 4.2, the fileinformation array is written in binary form into a file named inforindex. The code is similar to this when you want to access the fileinformation array:

Struct stat st;
Char buffer = "inforindex ";
Fileinformation * _ fileinfoindexptr = NULL;
If (STAT (buffer, & St) <0)
{
Fprintf (stderr, "error to stat % s/n", buffer );
Exit (-1 );
} // MMAP the inforindex to _ fileinfoindexptr

Int FD = open (buffer, o_rdonly );
If (FD <0)
{
Printf ("error to open % s/n", buffer );
Exit (-1 );
}
_ Fileinfoindexptr = (fileinformation *) MMAP (null, st. st_size, prot_read, map_shared, FD, 0 );

If (map_failed ==_ fileinfoindexptr)
{
Printf ("error to MMAP % s/n", buffer );
Close (FD );
Exit (-1 );
}
Close (FD );

Sample Code
:
Http://zhoulifa.bokee.com/6614538.html

The following example shows how to map a file to the memory.

Source code:

************************ ******************** * Filename: MMAP. c * Purpose: indicates how to call MMAP to map files to memory. * Wrote by: zhoulifa (zhoulifa@163.com) Zhou Lifa (http://zhoulifa.bokee.com) Linux enthusiasts Linux knowledge disseminators sohowhose developers are best at C Language * Date Time: heavy snow day in Shanghai, which is said to have been unavailable for many years * Note: Anyone can copy the code and use these documents, including your commercial use. * But follow the GPL * Thanks: * Ubuntu 7.10 is completely normal. * Google.com I usually use Google search to find many useful materials. * Hope: more and more people are expected to contribute to the development of science and technology. * Technology stands on the shoulders of giants and makes progress faster! Thank you for your contributions to the open source team! **************************************** *****************************/ # Include <sys/Mman. h>/* For MMAP and munmap */ # Include <sys/types. h>/* for open */ # Include <sys/STAT. h>/* for open */ # Include <fcntl. h>/* for open */ # Include <unistd. h>/* For lseek and write */ # Include <stdio. h> Int main (INT argc, char ** argv) { Int FD; Char * mapped_mem, * P; Int flength = 1024; Void * start_addr = 0; FD = open (argv [1], o_rdwr | o_creat, s_irusr | s_iwusr ); Flength = lseek (FD, 1, seek_end ); Write (FD, "/0", 1);/* add an empty character at the end of the file so that the following printf works normally */ Lseek (FD, 0, seek_set ); Mapped_mem = MMAP (start_addr, flength, prot_read, // read allowed Map_private, // other processes are not allowed to access this memory area. FD, 0 ); /* Use the ing area .*/ Printf ("% s/n", mapped_mem);/* to ensure normal operation, the file name passed by the parameter is preferably a text file */ Close (FD ); Munmap (mapped_mem, flength ); Return 0; }

Compile and run this program:
Gcc-wall MMAP. c
./A. Out text_filename

Because the above method uses prot_read, the content in the file can only be read and cannot be modified. If it is changed to prot_write, the content of the file can be modified. Because
Maap_private is used, so this process can only use this memory area. If it is changed to map_shared, it can be accessed by other processes, for example:

# Include <sys/Mman. h>/* For MMAP and munmap */ # Include <sys/types. h>/* for open */ # Include <sys/STAT. h>/* for open */ # Include <fcntl. h>/* for open */ # Include <unistd. h>/* For lseek and write */ # Include <stdio. h> # Include <string. h>/* For memcpy */ Int main (INT argc, char ** argv) { Int FD; Char * mapped_mem, * P; Int flength = 1024; Void * start_addr = 0; FD = open (argv [1], o_rdwr | o_creat, s_irusr | s_iwusr ); Flength = lseek (FD, 1, seek_end ); Write (FD, "/0", 1);/* add an empty character at the end of the file so that the following printf works normally */ Lseek (FD, 0, seek_set ); Start_addr = 0x80000; Mapped_mem = MMAP (start_addr, flength, prot_read | prot_write, // write allowed Map_shared, // allow other processes to access this memory area FD, 0 ); /* Use the ing area .*/ Printf ("% s/n", mapped_mem);/* to ensure normal operation, the file name passed by the parameter is preferably a text file */ While (P = strstr (mapped_mem, "hello") {/* modify the file content here */ Memcpy (P, "Linux", 5 ); P + = 5; } Close (FD ); Munmap (mapped_mem, flength ); Return 0; }

Man-a mmap For more details   Http://hi.baidu.com/donghaozheng/blog/item/81885f0fdafa7b276059f3b1.html