Problems encountered during the process of web crawler

I wanted to be a simple reptile in the course design that I started in junior school last semester. Then there is the problem is I want to crawl out of the link into a buffer, full and then put into the queue to crawl to reduce the hard disk IO time, but there is a problem, that is the last crawl when the buffer is probably not full, or do a breadth search, the first layer of the link is filled with the buffer, How can I tell if the buffer data should be fetched directly or wait for a new link to be filled in?

The problem arises because the project was initially unfamiliar with the piece. After this problem arose, I consulted the classmates and consulted the data, only finally solved the problem.

Problems encountered during the process of web crawler