Python Multithreaded processing files

Source: Internet
Author: User

a dozens of G file wants to use Python multithreading reads improve processing efficiency, and results are never as expected. It was a reminder of the hair. A process-initiated thread will share a file handle, and a-thread-to-file operation (even read) will also affect the B-thread. , pictures from Mao Yan:

The test code is as follows:

# -*- coding: utf-8 -*-def threadfunc1 (demo, threadnum, startlinenum,  Deallinenum):    #  Line Count counter     line = 0     # skip several lines     while line < startlinenum:         linestr = demo.readline ()          line += 1    # deal specified number of lines     while line  < deallinenum:        linestr = demo.readline ()         line += 1         print  "thread-%s:%s"  %  (THREADNUM,&NBSP;LINESTR) Def threadfunc2 (demo, threadnum ,  startlinenum, deallinenum):    linenum = 1     for line in Demo:        if linenum >= startlinenum and  linenum <  (Startlinenum+deallinenum):             print  "thread-%s:%s"  %  (threadnum, line)          elif  (linenum >=  (startlinenum+deallinenum)):             break;        linenum  = linenum+1import threadingif __name__ ==  "__main__":     #   Initialize the demo file     fileLoc =  ' Demo.txt '     demo =  Open (FileLoc, ' W ')     for i in range (100000):             demo.write ("line:" +str (i) + ' \ n ')     demo.close ()     #  Total number of pre-read files     demo = open (fileLoc, ' R ')     filetotalnum  = 0    for line in demo:             filetotalnum = filetotalnum + 1     demo.close ()     #  set the number of threads     TOTAL_THREAD = 6     #  always open the file in the main process, pass the file handle to the thread-initiated function     demo = open (FILELOC, ' R ')     for threadnum in range (Total_thread):         #  number of rows processed per thread         deallinenum =  filetotalnum /  (total_thread-1)         #  last thread to process the remainder         leftnum = filetotalnum %  (total_thread-1)          if (threadnum !=  (total_thread-1)):             #  instantiating thread             t  = threading. Thread (target=threadfunc2,args= (demo, threadnum, threadnum*deallinenum, threadnum*deallinenum+ Deallinenum))             #  start              t.start ()              #  wait for thread to end the process exit              t.join ()         else :             t = threading. Thread (target=threadfunc2,args= (demo, threadnum, threadnum*deallinenum, threadnum*deallinenum+ Leftnum))              t.start ()              t.join ()

The THREADFUNC2 function uses the handle parameter to read the file, but the result is that some threads have normal output, while others have no output. The visual cause is that the randomness of thread switching affects threads that would otherwise have been read normally (for example, thread switching occurs when some threads are positioned exactly to their processing range, while other threads have taken the time slice to continue processing on the basis of this position).

THREADFUNC1 is read using ReadLine, but is essentially the same as THREADFUNC2.

If the file handle that is passed to Threadfuncx is not opened in the main thread but is a file path, and the file is opened inside THREADFUNCX, the result is that each thread has output, but the output is not the content of the file fragment allocated to it, there is interference.

The details are not studied in detail, but in the case of no lock, Python starts the multithreaded read file to get unintended results.

The final solution is to use awk to "module -Cut" The file, and the different remainder of the mod corresponds to a processing thread.

Targetfile=xxxthreadnum=5 # Set Five threads for ((i=0; i<threadnum; i++)) do awk ' nr%n==t {print $} ' n= $threadNum t= $i $targ Etfile | Python dotask.py &done


Python Multithreaded processing files

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.