a dozens of G file wants to use Python multithreading reads improve processing efficiency, and results are never as expected. It was a reminder of the hair. A process-initiated thread will share a file handle, and a-thread-to-file operation (even read) will also affect the B-thread. , pictures from Mao Yan:
The test code is as follows:
# -*- coding: utf-8 -*-def threadfunc1 (demo, threadnum, startlinenum, Deallinenum): # Line Count counter line = 0 # skip several lines while line < startlinenum: linestr = demo.readline () line += 1 # deal specified number of lines while line < deallinenum: linestr = demo.readline () line += 1 print "thread-%s:%s" % (THREADNUM,&NBSP;LINESTR) Def threadfunc2 (demo, threadnum , startlinenum, deallinenum): linenum = 1 for line in Demo: if linenum >= startlinenum and linenum < (Startlinenum+deallinenum): print "thread-%s:%s" % (threadnum, line) elif (linenum >= (startlinenum+deallinenum)): break; linenum = linenum+1import threadingif __name__ == "__main__": # Initialize the demo file fileLoc = ' Demo.txt ' demo = Open (FileLoc, ' W ') for i in range (100000): demo.write ("line:" +str (i) + ' \ n ') demo.close () # Total number of pre-read files demo = open (fileLoc, ' R ') filetotalnum = 0 for line in demo: filetotalnum = filetotalnum + 1 demo.close () # set the number of threads TOTAL_THREAD = 6 # always open the file in the main process, pass the file handle to the thread-initiated function demo = open (FILELOC, ' R ') for threadnum in range (Total_thread): # number of rows processed per thread deallinenum = filetotalnum / (total_thread-1) # last thread to process the remainder leftnum = filetotalnum % (total_thread-1) if (threadnum != (total_thread-1)): # instantiating thread t = threading. Thread (target=threadfunc2,args= (demo, threadnum, threadnum*deallinenum, threadnum*deallinenum+ Deallinenum)) # start t.start () # wait for thread to end the process exit t.join () else : t = threading. Thread (target=threadfunc2,args= (demo, threadnum, threadnum*deallinenum, threadnum*deallinenum+ Leftnum)) t.start () t.join ()
The THREADFUNC2 function uses the handle parameter to read the file, but the result is that some threads have normal output, while others have no output. The visual cause is that the randomness of thread switching affects threads that would otherwise have been read normally (for example, thread switching occurs when some threads are positioned exactly to their processing range, while other threads have taken the time slice to continue processing on the basis of this position).
THREADFUNC1 is read using ReadLine, but is essentially the same as THREADFUNC2.
If the file handle that is passed to Threadfuncx is not opened in the main thread but is a file path, and the file is opened inside THREADFUNCX, the result is that each thread has output, but the output is not the content of the file fragment allocated to it, there is interference.
The details are not studied in detail, but in the case of no lock, Python starts the multithreaded read file to get unintended results.
The final solution is to use awk to "module -Cut" The file, and the different remainder of the mod corresponds to a processing thread.
Targetfile=xxxthreadnum=5 # Set Five threads for ((i=0; i<threadnum; i++)) do awk ' nr%n==t {print $} ' n= $threadNum t= $i $targ Etfile | Python dotask.py &done
Python Multithreaded processing files