Cut large files by line (linux split command Lite version), linuxsplit
Cutting large files by line (linux split command Lite version)
#-*-Coding: UTF-8-*-_ author _ = 'knowlifedeath ''' the Split command on Linux allows you to easily Split large files, this PY script is used to cut the file Python Version: 2.7 ''' import timeimport sysprint time on Windows based on the specified number of lines. strftime ('% Y-% m-% d % H: % M: % s', time. localtime (time. time () ''' the parameter '''bigfile = "E: \ Data \ Big.txt "OneUnitEnd = 1000000 # Number of split blocks ZeroFillWidth = 3 # zfill parameter is the serial number encoding length of small files seed = 0 # Number of Current Blocks UnitNo = 0 # Number of Current Blocks fSmallFile = open ('{0} \ samll1_1}.txt '. format (sys. pa Th [0], '0 '. zfill (ZeroFillWidth), 'w') fBigFile = open (BigFile) for line in fBigFile: rawLineData = line. strip ('\ n '). strip () if len (rawLineData )! = 0: seed + = 1 fSmallFile. write ('{0} \ n '. format (rawLineData) if seed = OneUnitEnd: fSmallFile. close () seed = 0 UnitNo + = 1 fSmallFile = open ('{0} \ samll1_12.16.txt '. format (sys. path [0], str (UnitNo ). zfill (ZeroFillWidth), 'w') # The last small file may be an empty file (full time) # a/w mode has little speed difference, hundreds of millions of data records are handled between 1 minute and a half to 2 minutes. # close multiple times. There will be no problem with fSmallFile. close () print time. strftime ('% Y-% m-% d % H: % M: % s', time. localtime (time. time ()))