Fast deduplication of tens of millions of mysql merged tables

Mysqlselect * froma2kwlimit10; + ----------- + ------ + ---------- + | c1 | c2 | c3 | c4 | + ----------- + ------ + ---------- + | 662164461 | 131545534 | TOM0 | 20120520 | 226662142 | 605685564 | | TOM0 | 20120516 | 5270

Mysql select * from a2kw limit 10; + ----------- + ------ + ---------- + | c1 | c2 | c3 | c4 | + ----------- + ------ + ---------- + | 662164461 | 131545534 | TOM0 | 20120520 | 226662142 | 605685564 | | TOM0 | 20120516 | 5270

Mysql> select * from a2kw limit 10;

+ ----------- + ------ + ---------- +

| C1 | c2 | c3 | c4 |

+ ----------- + ------ + ---------- +

| 662164461 | 131545534 | TOM0 | 20120520 |

| 226662142 | 605685564 | TOM0 | 20120516 |

| 527008225 | 172557633 | TOM0 | 20120514 |

| 574408183 | 350897450 | TOM0 | 20120510 |

| 781619324 | 583989494 | TOM0 | 20120510 |

| 158872754 | 775676430 | TOM0 | 20120512 |

| 815875622 | 631631832 | TOM0 | 20120514 |

| 905943640 | 477433083 | TOM0 | 20120514 |

| 660790641 | 616774715 | TOM0 | 20120512 |

| 999083595 | 953186525 | TOM0 | 20120513 |

+ ----------- + ------ + ---------- +

10 rows in set (0.01 sec)