[Help] PHP executes the collection task and the result crashes, but each execution is fast. PHP executes the collection task. the write procedure is 1. extract the url from the database, 2. php remotely obtains the URL title, and downloads the remote url through http. 3. download the data of another website B, http. 4. after the download is complete, download another [help] PHP to execute the collection task. The result is crashed, but each execution is fast.
PHP executes the collection task. the write procedure is
1. extract the url from the database first,
2. php remotely obtains the URL title and downloads the remote URL through http.
3. download the data of another website B, http.
4. after the download, download the data of another station C, http.
5. each record downloads data from four remote websites A, B, C, and D, updates the corresponding records of the data table, and loops through the next record.
The result is:
If you refresh the page once, all 10 items are being loaded, but none of them have been completed (after three hours, none have been completed ).
However, if you only perform one operation at a time, it will be completed quickly. It is hard to understand and the code has never been changed.
------ Solution --------------------
Is php timeout or memory full?
Enable error. log to view the error log?
The most important thing to write this kind of collection program is to determine how to handle program errors.
------ Solution --------------------
Most of them are code problems... no code, no truth.
------ Solution --------------------
When every log is written into the file, you can see which step has been stopped.
------ Solution --------------------
Reference:
When every log is written into the file, you can see which step has been stopped.
Positive solution
------ Solution --------------------
You
10 pieces of code that cannot be completed at a time
And
A piece of code that is quickly completed
What is the difference?
------ Solution --------------------
Where is the server? Do not parse the DNS first.
------ Solution --------------------
I mean, siteA is also local? Do not collect data first. How long does it take to directly run the corresponding script of siteA?
------ Solution --------------------
You have not provided specific code and cannot give specific suggestions.
However, the code is provided, and you may not be in the mood to view and test it.
My personal opinion on such applications is as follows:
1. Normally, the page returns to the home page, and to a connection such as a topic.
Should be excluded, otherwise it will easily lead to an endless loop
2. the page collection procedure should be single
That is, run once to collect a page
3. The collection control program is used to schedule the page collection program.
You can use fsockopen to open the page collection program and then return it. the target page url is passed only as a parameter to the page collection program.
This simulates a multi-process system.
4. Specifically, the database is used as the intermediary for implementation, and all the URLs precipitated from the page are stored in the database (of course, deduplication is required)
The collection control program reads data from the database to determine whether to continue collection and the size of each collection.
------ Solution --------------------
Reference:
The dedecms download class is used.
Try another class instead of dedecms integration.
Find a function or CLASS with similar functions online.
------ Solution --------------------
Reference:
Reference: reference: the dedecms download class is used.
Try another class instead of dedecms integration.
Find a function or CLASS with similar functions online.
In fact, discuz's curl and fsocket won't work either, and some servers will determine that the access is not normal. Only the class of dede is successful .. I tried other things, but it was also very slow!
......
Inniu's face... brother, you have become a fairy ~~
Write a collection curl to encapsulate how difficult it is. In fact, it is troublesome to analyze and proofread various regular expressions.
------ Solution --------------------
That is, secondary development is not that difficult. What features do you want? Paste the URL and requirements to help you write a function.
------ Solution --------------------
Network problems are the main cause affecting the collection speed.
Ping to check the response time.
Do not use the dedecms class. you can use curl to write the class, which is very simple.
Set timeout for each request. Do not run on your own PC. find a server with high configuration to run. open multiple processes and classify your tasks. this improves the efficiency by several times.
------ Solution --------------------
Reference:
Some domain names direct to a host that does not exist.
Given these domain names, whether there is a head jump, or remove www to see.
