Simple implementation of setting checkpoints in Python

A checkpoint is actually a record of past history and can be considered as a log. however, this is simplified. for example, I have another piece of text. there are a pile of links in the text. my current task is to download the content from those addresses. in addition, due to network problems or website problems, each download may not be very successful. A chain disconnection or socket Exception error may occur. However, no matter what kind of errors are generated, I hope my program can continue to run. Or, you can stop running from the download link. Instead of running from the start. This problem is very simple. Because these links are context-independent (context-related situations need to be analyzed separately ). Therefore, I only need to record the last entry before the program stops, and hope to continue the previous work. Here, we use the counter method to record the original link. The Code is as follows:
# This exception is caused by the absence of checkpoint content in the original text. <br/> class checkpointmisscontenterror: <br/> pass <br/> # Move the File Read pointer FD to the content corresponding to the checkpoint <br/> # the Check Point rule is to read one or more lines of the file, after the operation, send one or more rows to <br/> # Check file check_point. Run the program again. <Br/> def gocheckpoint (FD, check_point): <br/> if not OS. path. isfile (check_point): <br/> f_check = open (check_point, 'w') <br/> f_check.close () <br/> f_check = open (check_point, 'R ') <br/> lines = f_check.readlines () <br/> If Len (lines)> 0: <br/> check_content = lines [-1] # locate the last row of the checkpoint <br/> check_content = check_content.strip ('/n/R ') <br/> # go to check point <br/> while true: <br/> content = FD. readline () <br/> If content = '': # EOF <br/> raise checkpointmisscontenterror <br/> If content. strip ('/n/R') = check_content: <br/> Break </P> <p> f_check.close () # Close the checkpoint

The above section is not enough. The following code should be added:

# Pseudocode <br/> def download (downloadlist, sleep_time): </P> <p> If OS. path. isfile (downloadlist): <br/> F = open (downloadlist) <br/> # check_point file name, here an automatic checkpoint file is generated <br/> check_point = file [0: file.rfind('.'{}}'_check.txt '<br/> util. gocheckpoint (F, check_point) # This is the gocheckpoint function in the code above <br/> f_check = open (check_point, 'A ') # Write Data in append mode </P> <p> try: <br/> while true: <br/> content = f. readline () <br/> If C Ontent = '': # EOF <br/> Break <br/> content = content. Strip ('/n/R') <br/> If content! = '': <Br/> # has download URL <br/> time. sleep (sleep_time) <br/> downloadoper (path, URL) # Here is the pseudo code .. it can be considered as urllib. request. the retrieve () function or urllib. request. urlopen () <br/> # Write the content to the checkpoint file after the response operation <br/> f_check.write (content + '/N ') <br/> f_check.flush () # required, otherwise it will be cached and will not be written to the hard disk <br/> failed T: # You are not afraid of exceptions, press F5 again. <br/> raise exception () <br/> return util. failure # This is the constant I set. You can think it is 0 or 1 <br/> finally: <br/> F. close () <br/> f_check.close () # Close the file <br/> Print ('downloading is done ........................ ') <br/> return util. success

After the operation is completed, write it into the checkpoint file. After the program crashes, the previous work can be continued as long as the checkpoint file is still in progress. However, the checkpoint here is too simple compared with the transaction processing checkpoint in the database.