The entire process of encoding and conversion for a 3G text in Linux

Linux implements the entire process of encoding and conversion for a 3G text-Linux general technology-Linux programming and kernel information. The following is a detailed description. Linux commands involved in this process include: split, iconv, cat

Problem: There is a temporary a.txt encoded in gbk format. Now we need to convert it into UTF-8.

Difficulty: iconv conversion is performed in the memory. Therefore, 3G text cannot be directly converted.

Idea: use split to split files, convert ivonv to each word file, and then merge cat.

The procedure is as follows:

1) ll-h a.txt: Check the file size, 2.9 GB

2) Check the number of lines in the wc-l a.txt file, 2 million rows

3) split-l 20000000 a.txt chunk is split into five files by 10 million lines per file.

4) convert
Iconv-f gbk-t UTF-8 chunka> chunka_utf8-c
Iconv-f gbk-t UTF-8 chunkb> chunkb_utf8-c
Iconv-f gbk-t UTF-8 chunkc> chunkc_utf8-c
Iconv-f gbk-t UTF-8 chunkd> chunkd_utf8-c
Iconv-f gbk-t UTF-8 chunke> chunke_utf8-c

5) rm chunka chunkb chunkc chunkd chunke Delete the original file

6) combine cat chunk *> a.txt _ utf8

So far, the work has been completed.