Generate 1 TB of ultra-large files in Linux

Source: Internet
Author: User

There are two methods:

I. dd

Dd If =/dev/Zero of = 1t. img bs = 1g seek = 1024 COUNT = 0
BS = 1G indicates that 1G data is read and written each time. Count = 0 indicates that 0 times are read and written. Seek = 1024 indicates that 1024 blocks are skipped and the previous block size is 1g, so I skipped 1 TB in total!
This is the easiest way to create a large sparse file.
Ii. ftruncate64/ftruncate

If you use a system function, it is a little troublesome because it involves macro issues. I will describe it in detail based on an actual example. The option indicates the test item.
File sparse. C:// Option 1: whether to define macros related to large files
# DEFINE _ largefile_source
# DEFINE _ largefile64_source
# DEFINE _ file_offset_bits 64 # include <stdio. h>
# Include <sys/types. h>
# Include <sys/STAT. h>
# Include <fcntl. h>
# Include <errno. h>
# Include <string. h> # define FILENAME "bigfile"
# Define file_mode (s_irusr | s_iwusr | s_irgrp | s_iroth)
Int main (INT argc, char ** argv)
{
Int FD, RET;
Off_t offset; int Total = 0;
If (argc> = 2)
{
Total = atol (argv [1]);
Printf ("Total = % d/N", total );
} // Option 2: whether the o_largefile option exists
// FD = open (filename, o_rdwr | o_creat | o_largefile, 0644 );
FD = open (filename, o_rdwr | o_creat, 0644 );
If (FD <0 ){
Perror (filename );
Return-1;
}
Offset = (off_t) Total * 1024ll * 1024ll * 1024ll;
Printf ("offset = % LD/N", offset); // Option 3: whether to call a 64-bit System Function
// If (ftruncate64 (FD, offset) <0)
If (ftruncate (FD, offset) <0)
{
Printf ("[% d]-ftruncate64 error: % s/n", errno, strerror (errno ));
Close (FD );
Return 0;
} Close (FD );
Printf ("OK/N"); Return 0;
} Test environment:
Linux:/Disk/test/big # GCC -- version
GCC (GCC) 3.3.5 20050117 (prerelease) (SuSE Linux)
Linux:/Disk/test/big # uname-
Linux 2.6.11.4-20a-default #1 wed Mar 23 21:52:37 UTC 2005 i686 i686 i386 GNU/Linux Test results (pseudo code representation ):
1. When the macro definition is complete:
If {o_largefile = true & ftruncate64 = true}
OK;
Elseif {o_largefile = false & ftruncate64 = true}
OK;
Elseif {o_largefile = false & ftruncate64 = false}
No error is reported during running, but greater than 4 GB is not supported;
Elseif {o_largefile = true & ftruncate64 = false}
No error is reported during running, but greater than 4 GB is not supported;
Conclusion]: When the macro definition is complete, whether to call ftruncate64 is the key to deciding to support 4G or above files. o_largefile is useless. 2. Incomplete macro definition: _ file_offset_bits missing
First, declare that _ largefile64_source must be defined for o_largefile.
If {o_largefile = true & ftruncate64 = true}
Generate abnormal large files;
Elseif {o_largefile = false & ftruncate64 = true}
Generate abnormal large files;
Elseif {o_largefile = false & ftruncate64 = false}
No error is reported, but greater than 2g is not supported;
Elseif {o_largefile = true & ftruncate64 = false}
No error is reported during running, but greater than 4 GB is not supported;
Conclusion]: In the case of undefined _ file_offset_bits, The ftruncate64 call is invalid and will have unpredictable consequences. The test here is to generate a large file (> 1 Tb ), I cannot explain the reason. The purpose of o_largefile is to support large file systems in 32-bit systems and allow large files whose length cannot be expressed by 31-bit (2 GB) characters to be opened; in addition, off_t is of the unsigned int type, that is, up to 4 GB is allowed. Therefore, ftruncate supports up to 4 GB files. To sum up: To support files larger than 2 GB, define at least _ largefile64_source macro and set the o_largefile option. To support files larger than 4 GB, define all the preceding macros and call ftruncate64; the remaining matches are all incorrect! 【 Appendix]:
Main options of DD:
If the following column ends with a specified number, multiply it by the corresponding number:
B = 512, c = 1, K = 1024, W = 2, M = 1024 K, G = 1024 m
Case-insensitive. If = File
Input File name. The default value is standard input. Of = File
Output file name. The default value is standard output. IBS = bytes
Read bytes at a time (that is, the size of a block is bytes ). Obs = bytes
Write bytes at a time (that is, the size of a block is bytes ). BS = bytes
Set the size of the read/write block to bytes, which can replace IBS and obs. CBS = bytes
Bytes are converted at a time, that is, the size of the conversion buffer. Skip = Blocks
The blocks are skipped from the beginning of the input file and then copied. Seek = Blocks
The blocks are skipped from the beginning of the output file and then copied. (Usually only valid when the output file is a disk or tape) Count = Blocks
Copy only blocks. The block size is equal to the number of bytes specified by IBS. Conv = Conversion [, conversion...]
Use the specified parameter to convert the file. Conversion parameters: Convert ebcdic to ASCII. Ebcdic converts ASCII to ebcdic. IBM converts each line to a CBS record by converting ASCII to alternate ebcdic. Block. The missing part is filled with spaces. Unblock
Make the length of each line be CBS, and fill the remaining part with spaces. Lcase converts uppercase to lowercase. Ucase converts lowercase to uppercase. Noerror
Error notrunc not displayed
The output file is not truncated.

Sync fills each input block into IBS bytes, and the missing part is filled with null (NUL) characters.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.