A few days ago began to run a data list, list needs to provide user name, whether there is a mobile phone number, whether there is a mailbox, user list I easily get to, but, the user list has more than 2000w, and to detect whether the user has a mobile phone number, whether there is a mailbox must be through an open security interface a user to request, Then parse the return value to know.
Here are the scenarios I'm dealing with:
1. Save the 2000w list to a temporary data sheet
2, with PHP program every time from the table to get 500 users, after detection of SQL update to generate the original record
3, in order to prevent the PHP program suddenly broken, with the shell script every 1 minutes to detect, PHP hangs off the restart
The reason I use Shell scripting as a daemon is that the detection interface between the phone and the mailbox is slow and it is impossible to detect 2000w users in 1-2 days.
Program Details:
1, the Temporary Save user List table users, the table structure is as follows:
Copy CodeThe code is as follows:
CREATE TABLE ' users ' (
' Account ' varchar (COMMENT ' username '),
' Has_phone ' tinyint (3) unsigned not NULL default ' 0 ' COMMENT ' whether there is a mobile number ',
' Has_email ' tinyint (3) unsigned not NULL default ' 0 ' COMMENT ' is there a mailbox ',
' Flag ' tinyint (3) unsigned not NULL default ' 0 ' COMMENT ' flag bit ',
PRIMARY KEY (' account '),
KEY ' flag ' (' flag ')
) Engine=innodb DEFAULT Charset=utf8 comment= ' list of tables ';
I first import more than 2000 W user name into this temporary table, Has_phone and Has_email the two fields by default is 0 (NO), flag bit flag indicates whether the user has been detected.
here is a partial table data:
9873aaa,0,0,0
adddwwwd876222,0,0,0
testalexlee,0,0,0
codejia.net,0,0,0
haohdouywaa21,0,0,0
2. PHP Script check_users.php
After the user list is imported into the table, then write a simple php script, the idea is: every time the loop from the table fetch flag=0 500 users, and then request the interface to determine whether the user has a mobile phone number, mailbox, generate a SQL, save to a sqls array, and so on 500 users all finished testing, Loop sqls Array, update the table of these 500 lists, and set the flag flag bit to 1, indicating that it has been detected, the next time will not be acquired.
Because the PHP script code is longer, here's a simple code description:
Copy CodeThe code is as follows:
Class users{
Private $data;
Private $sqls;
Private $nums; Determine if there are 500 users
Private $total _nums; Number of users currently detected
Fetch 500 users at a time
Private Function Getusers () {...}
Detect these 500 users and generate SQL
Private Function Checkuserinfo () {...}
Update these 500 users
Private Function Updateuserinfo () {...}
Run
Public Function run () {
$flag = true;
while ($flag) {
if ($this->nums! =) {$flag = false;}
if ($this->total_nums = = 10000) {
Exit (0); Run out of 1w users to exit, initiated by daemon process
}
$this->getusers ();
$this->checkuserinfo ();
$this->updateuserinfo ();
Sleep (1); Run 500 users to rest for 1 seconds to protect the user detection interface
}
}
}
$user = new Users ();
$user->run ();
?>
The above is a concise version of the PHP script, probably the meaning of the first version is not $total_nums this variable, is because the beginning of the script to run, found that only ran out of 4w more than a script to hang the ball, and later, because the connection database is not connected, the script has been hanging there. Adding this variable does not solve the problem, but after each run of 1w users, the PHP script exits and is restarted by the following shell script.
3. Shell script as Daemon
I added this shell script to crontab, which executes every 1 minutes, and this shell script is simple and detects check_ Users.php whether there is a process ID, if present, the PHP script is still running, the shell script does not do anything, if it does not exist, the PHP script has exit (0) ran out of 1w users, then the shell script to start the script, Go to the next 1w user list for detection.
As I mentioned above, if the PHP script is connected to the database when it is unable to connect, PHP will always hang the ball there, unable to exit. I added a time detection in the shell script, when the PHP script process exists, the calculation has existed for how long, if more than I expected time, the PHP script kill, and then restart.
The first example data, the results are similar to the following:
testalexlee,1,0,1
codejia.net,0,0,1
haohdouywaa21,1,1,1
9873aaa,0,1,1
adddwwwd876222,1,0,1
said at the end: the above user list data just to raise a chestnut, not too serious, 2000w data, I estimate to run for a period of time, because the detection interface is slow, the interface after receiving the request to even the table, check the table, and then return. In fact, the best way is to directly from the interface request table pull a list out, and then with the shell command processing will soon have results, but in the company is so, some things are not open, you understand ~ ~ ~
http://www.bkjia.com/PHPjc/328044.html www.bkjia.com true http://www.bkjia.com/PHPjc/328044.html techarticle a few days ago began to run a list of data, the list needs to provide a user name, whether there is a mobile phone number, whether there is a mailbox, user list I easily get to, but, the user list has more than 2000w ...
