Python simple code for setting checkpoints

A checkpoint is actually a record of past history and can be considered as a log. however, this is simplified. for example, I have another piece of text. there are a pile of links in the text. my current task is to download the content from those addresses. in addition, due to network problems or website problems, each download may not be very successful. A chain disconnection or socket Exception error may occur. However, no matter what kind of errors are generated, I hope my program can continue to run. Or, you can stop running from the download link. Instead of running from the start. This problem is very simple. Because these links are context-independent (context-related situations need to be analyzed separately ). Therefore, I only need to record the last entry before the program stops, and hope to continue the previous work. Here, we use the counter method to record the original link. The Code is as follows:

# This exception is caused by the absence of checkpoint content in the original text. class CheckPointMissContentError: pass # Move the File Read pointer fd to the corresponding content of the checkpoint # the check point rule is, read one or more lines of the file. After the operation, send one or more lines to # Check the file check_point. Run the program again. Def GoCheckPoint (fd, check_point): if not OS. path. isfile (check_point): f_check = open (check_point, 'w') f_check.close () f_check = open (check_point, 'R') lines = f_check.readlines () if len (lines)> 0: check_content = lines [-1] # Find the last row of the checkpoint check_content = check_content.strip ('/n/R') # go to check point while True: content = fd. readline () if content = '': # eof raise CheckPointMissContentError if content. strip ('/n/R') = check_content: break f_check.close () # Close the checkpoint

The above section is not enough. The following code should be added:

# Pseudocode def Download (downloadlist, sleep_time): if OS. path. isfile (downloadlist): f = open (downloadlist) # check_point file name. Here, a checkpoint file check_point = file [0: file.rfind('.'{}}'_check.txt 'Util is automatically generated. goCheckPoint (f, check_point) # This is the GoCheckPoint function f_check = open (check_point, 'A') in the above Code # Write try: while True: content = f in append mode. readline () if content = '': # eof break content = content. strip ('/n/R') if conte Nt! = '': # Has download url time. sleep (sleep_time) DownloadOper (path, url) # Here is the pseudo code .. it can be considered as urllib. request. the retrieve () function or urllib. request. urlopen () What's # After the response operation writes the content to the checkpoint file f_check.write (content + '/N') f_check.flush () # required, otherwise it will be cached, it will not be written into the hard disk. The kernel T: # is not afraid of exceptions. If you press F5 again, you can execute raise Exception () return Util. FAILURE # This is a constant set by me. If it is regarded as 0 or 1, you can finally: f. close () f_check.close () # close the file print ('downloading is done ........................ ') return Util. SUCCESS

After the operation is completed, write it into the checkpoint file. After the program crashes, the previous work can be continued as long as the checkpoint file is still in progress. However, the checkpoint here is too simple compared with the transaction processing checkpoint in the database.