Python multi-process chunking method for reading large files

In this paper, we describe the method of Python multi-process block reading large files. Share to everyone for your reference, as follows:

Reads oversized text files, uses multi-process block reads, and outputs each piece separately as a file

#-*-CODING:GBK-*-import urlparseimport datetimeimport osfrom multiprocessing import Process,queue,array,rlock "" "  Multi-process chunked Read file "" "WORKERS = 4BLOCKSIZE = 100000000file_size = 0def getfilesize (file):" "Gets the size of the file to read" "" Global file_size fstream = open (file, ' R ') Fstream.seek (0,os. Seek_end) File_size = Fstream.tell () fstream.close () def process_found (pid,array,file,rlock): Global FILE_SIZE Global J OB Global PREFIX "" "Process processing Args:pid: Process number array: Inter-process Shared Queue to mark the end of file blocks read by each process file: the names of the files read by each process first from the array Gets the current maximum value for the start position startpossition End position Endpossition (startpossition+blocksize) if (startpossition+blocksize)
 
  

  
More readers interested in Python related content can view the topic: Python string manipulation tips Summary, Python Introductory and Advanced classic tutorials, and Python file and directory Operations Tips Summary

  
I hope this article is helpful for Python program design.

  
