A method of splitting large files under Linux system

In Linux to split large files, such as a 5GB log file, you need to divide it into small files, after segmentation to facilitate the normal text editor read.

Sometimes, you need to transfer 20GB of large files to another server, and you need to split it into multiple files, which makes it easier to transfer data.

The following five different examples, to explain the Linux partition of the large file method for your reference.

Example 1, split by 1000 rows per file

The split command splits the file into 1000 lines per file, and the filename is prefixed by prefix]aa, [prefix]ab, [prefix]AC, etc., the default prefix is x, and the number of rows per file is 1000 lines.

Command:

The code is as follows:

$ split Mylog-l 1000

$ wc-l *

4450 MyLog

1000 Xaa

1000 Xab

1000 Xac

1000 Xad

450 xae

Example 2, 20MB segmentation for each file

Split file for multiple 20MB files with the-B option.

Command:

The code is as follows:

$ split-b 20M Logdata

$ LS-LH | Tail-n +2

-RW-------1 Sathiya sathiya 102M 18:47 logdata

-RW-------1 Sathiya sathiya 20M 19:20 XAA

-RW-------1 Sathiya sathiya 20M 19:20 xab

-RW-------1 Sathiya sathiya 20M 19:20 xac

-RW-------1 Sathiya sathiya 20M 19:20 xad

-RW-------1 Sathiya sathiya 20M 19:20 xae

-RW-------1 Sathiya sathiya 1.6M 19:20 xaf

Example 3, specifying prefix segmentation with each file 50MB

Use the –bytes option to split the file into multiple 50MB files, –bytes similar to-B option, and specify a prefix in the second argument.

Command:

The code is as follows:

$ split--bytes=50m Logdata Mydatafile

$ ls-lh

Total 204M

-RW-------1 Sathiya sathiya 102M 18:47 logdata

-RW-------1 Sathiya sathiya 50M 19:23 MYDATAFILEAA

-RW-------1 Sathiya sathiya 50M 19:23 Mydatafileab

-RW-------1 Sathiya sathiya 1.6M 19:23 MYDATAFILEAC

Example 4, split file based on row number

Use the-l option to specify the number of rows to split the file into files with the same number of rows.

Command:

The code is as follows:

$ wc-l Testfile

2591 testfile

$ split-l 1500 Testfile Importantlog

$ wc-l *

1500 Importantlogaa

1091 Importantlogab

2591 testfile

Example 5, naming a split file with a digital suffix

Use the-D option to specify that the suffix is a number, such as 00,01,02 ... rather than AA,AB,AC.

Command:

The code is as follows:

$ split-d Testfile

$ ls

Testfile x00 x01 x02

Available options

The code is as follows:

Short option long option option description

The-b–bytes=size size value is a byte for each output file.

-c–line-bytes=size The maximum number of bytes in each output file for a single line.

-d–numeric-suffixes uses numbers as a suffix.

The-l–lines=number number value is the size of the columns per output file.

Short option long option option description

The-b–bytes=size size value is a byte for each output file.

-c–line-bytes=size The maximum number of bytes in each output file for a single line.

-d–numeric-suffixes uses numbers as a suffix.

The-l–lines=number number value is the size of the columns per output file.