Experience Sharing: building a social engineering database TIPS

Experience Sharing: building a social engineering database TIPS

Recently, we have been building a social engineering database. There are also many articles on the Internet, but few details are involved. I would like to share some of my experiences with you here.

Test Environment

Test environment: windows server 2012 (x64, 16 GB memory), MySQL-5.0.90, php-5.2.14-Win32

Preparation tools: coreseek-4.1-win32, Phantom cattle source code

Build Process

1. First, check the fields of the table to be indexed to facilitate configuration in the csft_mysql.conf file.

Modify the csft_myxql.conf file. (Coreseek 3.2.14 does not support SQL _query_string =)

Note that the fields in SQL _query must be consistent with those in our nan1 table,

To support cjk (Chinese, Japanese, and Korean abbreviation) queries, we must use its dedicated charset_table

Therefore, we should add charset_table to index mysql (because the data volume is too long, this will be omitted. Please refer to my

Configuration File)

2. Let sphindexing support real-time indexing, so that we can solve a problem later. Later, it was found that the solution was still unsuccessful.

What is real-time index :)

 

It should be added below index mysql. For details, refer to the configuration file. Use the UTF-8 without after modifying the configuration file

Save the BOM format so that the program can read the configuration file.

3. Create an index and start

 

 

Because the test data volume is small, the program starts successfully.

If the data volume exceeds 0.1 billion, insufficient memory is displayed.

Set mem_limit = 1 M to 1 M to re-create the index.

Split table data and dynamically insert Data Based on Real-Time Indexes (ps: if Daniel has a better solution, please contact me)

For the test, we use the nan4 table for demonstration.

There are three ways to split a table.

Code: create table nan3 select distinctfirstname, lastname, email, username, password, hash, addr1, addr2, jumin1, jumin2, sex, ssn from nan4; create table nan3 Selectfirstname, lastname, email, username, password, hash, addr1, addr2, jumin1, jumin2, sex, ssn from nan4 group byfirstname, lastname, email, username, password, hash, addr1, addr2, jumin1, jumin2, sex, ssn from nan4; // the effects of the two statements are the same, removing username, password .... insert the nan3 table with the same content in sex. If someone doesn't understand what I mean, I suggest that I have a problem with the expression ability.

The Group by statement is similar;

Last night, I posted a statement that directly removes duplicate content from the table and thanked him (ps: I studied it with him for a night, but I was stupid .)

Code:delete from temps where id in (select id from (selectid from temps as s where (select count(*) from temps as awhere a.username=s.username and a.password =s.password)>1and id not in(select id from(select id,count(distinct username,password) from temps as s where (select count(*) from test4as a where a.username=s.username and a.password =s.password)>1 group by username) as sss))as ttt)

Note: Here id: the auto-increment ID is required.

Do not use the limit parameter and distinct parameter to split the table, which may cause freezing and deduplication multiple times.

create table tempss select * from nan3 limit 0,3;create table temp select * from nan3 limit 3,5;

 

 

OK. Now we have divided the table into two tables and manually added the auto-increment ID (the maximum value of temp id is 4 and the minimum value of temps id is 5) to the two tables. We will create an index for a temp table, and start

 

I don't know whether it is my character or what it is, so we need to slightly change the source code of search. php.

 

Search results

 

Insert into temp select * from tempss; // insert data from tempss to temp

 

Search results after insertion

I'm sorry, after my tests, the real-time index is restarted and the memory is insufficient, and we can only modify our index.

The corresponding field parameter value of the id.

Supplement TIPS

If you happen to have a database in South Korea or Japan, you just need to import it to the database first (the database encoding should be utf8), and the encoding should be set to 949.

Or euckr

 

The following statements may help you:

Create table test4 select username, password from test1 unionselect username, password from test2; // clear the table test2 and table test1 and insert it to test4. ps: The field data type can be different) alter table test4 default character set utf8 COLLATE utf8_general_ci; // alter table nan1 modify column username varchar (50) is not interpreted ); // modify the username data type alter table $ table add $ username varchar (50) null; // add a new field

Program, extract code 4su4