Allocate large memory in Linux

Allocate large memory in Linux-general Linux technology-Linux programming and kernel information. For more information, see the following. Today, Liu Bin asked me a few questions about a problem he encountered when handling large files. At the beginning, Liu Bin asked how to handle a large file. I said that GCC has an option to process large files. Later, he told me some detailed information about his design. It turns out that he wants to index a large file, but the file is relatively large.

Indexing large files may cause some problems. The first is the memory problem. Of course, physical memory cannot be fully used and swap partitions are required. We have done a few experiments to verify the relationship between the large space allocated by the program and the swap partition.

1. Experiment on a machine without swap partitions.

Char * p = NULL;
For (int I = 0; I <10; I ++ ){
P = malloc (100000000 );
Bzero (p, 100000000 );
Printf ("###%# 0x \ n", p );
}

The program allocates 10 * MB of space, which is much larger than the system memory. As a result, the program has a segment error. If it is a small number, there will be no problem (nonsense, it is not necessary to deal with large files ).

If you comment out a line of bzero, you will find that the memory is not used much during running. It can be determined that the system is not allocated if no value is assigned during memory application.

2. Experiment on a machine with swap partitions.

Char * p = NULL;
For (int I = 0; I <10; I ++ ){
P = malloc (100000000 );
Bzero (p, 100000000 );
Printf ("###%# 0x \ n", p );
}

Physical memory: 512 M
Swap partition: 720 M

There is no problem with allocation, that is, the system is slow. If you use free to view memory usage, you will find that the physical memory is only 6 MB, and the swap partition occupies 500 mb.

Ft. Try again and increase the number!

Char * p = NULL;
For (int I = 0; I <10; I ++ ){
P = malloc (150000000 );
Bzero (p, 150000000 );
Printf ("###%# 0x \ n", p );
}

10*150 M> 512 M + 720 M. check whether a segment error occurs. Compiling and running. Haha, a segment error occurs. It seems that the prediction is correct.

3. Add swap partitions and try again

The next step is to add swap partitions. experiment again. The result should be clear, but I 'd like to take a look. There are two ways to add a swap partition: Add a hard disk as the swap partition and use a file as the swap partition. My hard disk is no longer empty, so I have to use files as swap partitions.

Dd if =/dev/zero of =./swapfile bs = 1 M count = 500
Mkswap./swapfile
Swapon./swapfile

After running the program, the allocation is successful. Yes!

4. Large Memory Access experiment.

Although the problem of large memory allocation is solved, the index of large files still requires random access to large memory, which is a problem. because 32-bit machines can only access 4 GB address space, if the file has 5 GB, how can they access memory other than 4 GB? Like above, malloc, the malloc parameter is a number of size_t, the maximum is 4 GB. We had to split the distribution. The same is true for access. Can we split the access?

5G = 4G + 1G
Void * temp = p + 4G;
Temp + = 5G-4G

You can access the 5g region.