Using split to split large files under Linux

split large files with split

In our actual production environment, will produce a lot of large files, the size of these files are different, some even have a good dozens of T size, then we analyze these files, then how to do? In this case, we can use a command split from Linux to implement the file segmentation.

Split's partitioning of the file supports segmentation of the number of rows, as well as support for splitting two modes by size. To illustrate, there is no concept of ' line ' in a binary file, so the binary cannot use row splitting, but only according to the size of the file.

In this experiment, we split up a 500M size file, such as:

650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M00/82/A2/wKioL1ddkhHzzeFzAACJsd5wDCY548.png "title=" Picture 1.png "alt=" Wkiol1ddkhhzzefzaacjsd5wdcy548.png "/>

Let's split this simulation file and note that the file is empty!

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/82/A2/wKioL1ddkhygVKtEAACNjrYN_60906.png "title=" Picture 2.png "alt=" Wkiol1ddkhygvkteaacnjryn_60906.png "/>

In, we can see that the file is data -l Span style= "font-family: ' The song Body '; The parameter is divided by the number of rows, 500 represents 500 line is divided into a file, in the above experiment, We can see that only one file is split, but if it is a 500m 500 Yes, but only a single file is split, so this file can be used to simulate a binary file to be split by size:

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/82/A4/wKiom1ddkRnCSR-5AAFAya5qYuM125.png "title=" Picture 3.png "alt=" Wkiom1ddkrncsr-5aafaya5qyum125.png "/>

For example, we split the file according to size 64M , splitting out a lot of small files, and more than enough 64M files are placed in a file, and the separated file names will be sorted in the default order.

Next, let's divide by the number of rows:

650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M01/82/A2/wKioL1ddkjXyXtiYAAENucW8BHg129.png "title=" Picture 4.png "alt=" Wkiol1ddkjxyxtiyaaenucw8bhg129.png "/>

In, we will copy the /etc/passwd file, we can see that the file has a Line, we are divided according to the size of the line, Will be divided into three small text, the name of the file is also named in the default order, the default size of the number of rows in a file.

We split the large file by split, we can get a lot of small files, so that we in the daily work of the analysis of the file will be much simpler, and will improve a lot of efficiency.

Using split to split large files under Linux