Phpmysql million-level data deduplication _ PHP Tutorial

Phpmysql removes duplicate data from millions of data records .? PHP Tutorial defines a number of groups, used for storing the final result resultarray(when reading the uid.pdf file, fpfopen(test.txt, r); while (! Feof ($ fp) {$ uidfgets ($ fp); $ ui // Define an array to store the results after deduplication
$ Result = array ();
// Read the uid list file
$ Fp = fopen('test.txt ', 'r ');

While (! Feof ($ fp ))
{
$ Uid = fgets ($ fp );
$ Uid = trim ($ uid );
$ Uid = trim ($ uid, "r ");
$ Uid = trim ($ uid, "n ");

If ($ uid = '')
{
Continue;
}
// Use uid as the key to check whether the value exists
If (empty ($ result [$ uid])
{
$ Result [$ uid] = 1;
}
}

Fclose ($ fp );

// Save the result to a file
$ Content = '';
Foreach ($ result as $ k => $ v)
{
$ Content. = $ k. "n ";
}
$ Fp = fopen('result.txt ', 'w ');
Fwrite ($ fp, $ content );
Fclose ($ fp );
?>

// Define an array to store the results after deduplication
$ Result = array ();
// Read the first uid list file and put it in $ result_1
$ Fp = fopen('test_1.txt ', 'r ');
While (! Feof ($ fp ))
{
$ Uid = fgets ($ fp );
$ Uid = trim ($ uid );
$ Uid = trim ($ uid, "r ");
$ Uid = trim ($ uid, "n ");
If ($ uid = '')
{
Continue;
}
// Write $ result with uid as the key. if there is a duplicate, it will overwrite
$ Result [$ uid] = 1;
}
Fclose ($ fp );
// Read the second uid list file and perform deduplication
$ Fp = fopen('test_2.txt ', 'r ');
While (! Feof ($ fp ))
{
$ Uid = fgets ($ fp );
$ Uid = trim ($ uid );
$ Uid = trim ($ uid, "r ");
$ Uid = trim ($ uid, "n ");
If ($ uid = '')
{
Continue;
}
// Use uid as the key to check whether the value exists
If (empty ($ result [$ uid])
{
$ Result [$ uid] = 1;
}
}
Fclose ($ fp );
// The Results saved in $ result are after the deduplication, which can be output to the file. the code is omitted.
?>

Http://www.bkjia.com/PHPjc/630861.htmlwww.bkjia.comtruehttp://www.bkjia.com/PHPjc/630861.htmlTechArticle? Php Tutorial // define an array to store the result after deduplication $ result = array (); // read the uid list file $ fp = fopen('test.txt ', 'r '); while (! Feof ($ fp) {$ uid = fgets ($ fp); $ ui...