Rainbow Table Analysis

Source: ncph

==============================
Rainbow table
==============================

The rainbow table is created by philpe oechsilin in making a faster cryptanalytic time-memory trade-off.
An Improved precomputering table is proposed to increase the success rate and reduce the storage space.

Rainbowcrack-1.2-src is a Zhu shuanglei implementation of this, for his implementation write these notes that illustrate the rainbow table.

1. Organization and generation of rainbow table (rtgen description)

Rainbow table is composed of many 16 bytes rainbowchains.

The structure of rainbowchain is as follows:
Struct rainbowchain
{
+ 000 uint64 nindexs;
+ 008 uint64 nindexe;
};

The number of rainbowtables is determined by the character space, which is calculated in advance and passed in by argv [7.
Int nrainbowchaincount = atoi (argv [7]);

Nindexs are randomly generated by the function CWC. generaterandomindex () (this generation is purposeful and will be explained later)
At the same time, this index is also put into cchaindomaincontext. m_nindex, m_nindex starts to the intermediate record.
Nindexe is calculated by nrainbowchainlen.

The process is as follows:

1) m_nindex (START: nindexs) will be randomly generated. Refer to [1] For details on CWC. indextoplain ().
The entire conversion process is similar to binary to hexadecimal, but it is changed to the hexadecimal notation of the plaintext character length (m_nplaincharsetlen)

2) pre-set the hash function-CWC. plaintohash () for the characters obtained in step 1 ()

3) Reduce the generated hash. For the reduce function CWC. hashtoindex (NPOs), see [2].
The final m_nindex must still be within the character space range (. '. % m_nplainspacetotal ).

Repeat nrainbowchainlen in the preceding three steps and put m_nindex into nindexe. nrainbowchainlen is the chain length.
All rainbowchains repeat the preceding steps.

It can also be expressed as the following process:

Plain hash reduce
K1 -------> C -------> H --------> k2 -----> ....

Index plain hash CED hash
(New Index) <----- key to reducing storage space

Because a new index is generated throughout the process, although not recorded, it can still be calculated because all functions are unidirectional.
Therefore, it is unnecessary to generate nindexs in strict order. These new indexes are probably included.
The new index has a certain reduce range, which is caused by m_nreduceoffset. The success rate is also due to this reason, it is likely that
The entire random process does not cover every arrangement. To increase the coverage of multiple performanceoffset tables
There will still be a miss rate, even 0.01% or even smaller.

According to the paper of Philip oechsilin, the process of K1-> k2 is defined as fn. The main reason for different FN is that the reduce function is different.
(The reason for this difference is that the NPOs increase can be found in [2])

2. Sorting of rainbow tables (rtsort description)

Rtsort uses quick sorting and external sorting
The sorting object is rainbowchain. nindexe, which requires great attention.

External sorting is used only when the memory capacity is low.

External sorting process:

1) read the content of the corresponding size from the rainbow table File Based on the memory size.

2) use quick sorting to sort the corresponding chain and store it in the temp file.

3) the corresponding information is stored in the linked list of the csortedsegment structure.

4) Repeat steps 1-3 until all content of the rainbow table is read.

5) Merge all the items in the linked list. (instead of merging two items, merge them together)
Merge the smallest one from all sorted items. The sorted object is rainbowchain. nindexe, And the last 8 bytes.

++ ---
Some optimizations are also made:

Getnextchain will read m_nfilechaincount in the file in advance and put it in the array of rainbowchain. m_chain.
However, if the size exceeds 1024, only 1024 is read.
If (m_nchaincount = m_nnextchainindex) is a condition for judging the possibility of the next read.
Returns the URL (rainbowchain *) of m_chain [m_nnextchainindex] pointed to by m_nnextchainindex *)

Removetopchain is mainly used to update m_nnextchainindex, but after reading all, return true to let mergesortedsegment Delete this chain item (list. Erase)
--- ++

6) Add the merged items to the original file.

Preparesortedsegment function process 1-4
Mergesortedsegment function process 5, 6

3. Use of the rainbow table (rcrack description)

It should be said that the rcrack process is the look-up process.

1) Search for each RT File
For (I = 0; I <vpathname. Size () & HS. anyhashleft (); I ++)
{
Searchrainbowtable (vpathname [I], HS );
Printf ("\ n ");
}

2) In searchrainbowtable, the data is divided into two blocks Based on the memory size for search.
Call the searchtablechunk function.

3) The following are key functions of rcrack.

Searchtablechunk (pchain, nrainbowchainlen, nrainbowchaincountread, HS );

Pchain-The chain table stored in the rainbow table in the memory. The items in the rainbow table may be much larger than those in the memory,
Therefore, cmemorypool is used to allocate space based on the memory size. pchain uses cmemorypool. alloc to allocate space.

Length of nrainbowchainlen-rainbowchain

Nrainbowchaincountread-Number of rainbowchains read into the pchain. The file size is/16 (of course, it must be a multiple of 16)

HS-stores the hash to be checked, and also stores the crack results, whether the original hash is found, whether it is found, plaintext, and so on.
// ++
Vector <string> m_vhash;
Vector <bool> m_vfound;
Vector <string> m_vplain;
Vector <string> m_vbinary;
//--

The entire look-up process is as follows:
A. First, make some preparations.
Hash settings, conversion, and other work
The purpose of requestwalk is to generate an array that matches pchain [I]. nindexe for a hash to be cracked.
The requestwalk is created at the first time. The items to be matched are calculated based on the chain length and the position of the POs (see [4-1]).
It only needs to be defined by the position of POs, so it only needs to be calculated once at the beginning. Later, this hash can be used for calculation.

B. calculation and comparison
The first time hash uses the reduce function at the NPOs position (Rn-1 N is chainlen), it is compared with all items in pchain [I]. nindexe.
If a matched value is found, the checkalarm function is used to check the value. Based on the location of the guess, the checkalarm
Start pushing and repeat the f function step. When the final NPOs position is reached, the reduce function in the f function will not be used, but will only be pushed to the hash value (see [5]).
The index of the hash that can be pushed is expressed as the plaintext of the number.
Note that there may also be multiple matching values, because of the convergence of reduce functions,
The space of the original hash and reduce functions is reduced only because the reduce function only takes 8 bytes starting with hash for calculation.
Therefore, you need to search in a matching domain. If the checkalarm function fails to pass the verification, it indicates that this NPOs is not the desired position.
This is called false alarm.

If the match for the first time fails, the hash is assumed to have been launched by the hash function of the previous index.
Rn-2, get a new index, and then step by step push to the end, that is, the end of the chain (of course only once F N-1)
If the comparison process is the same as above, the location of the guess can be determined. If it is different, continue the same step, just move the POs position forward.
Until POS is 0.

Finally, put the result in the hashset (HS), whether the plain text is found or not found.
Checkalarm stores plaintext information in HS (see [5]).

4. Legacy issues

There are still many questions about parameter setting and optimization. One article about this cannot be found.
Chainlen's confirmation, some do not understand.
It is generally known that it is limited by M = m x L x m0 and t = t x L x T0 (that is, the optimal computing time and success rate are obtained based on the memory size)
You need to study it carefully. It is not complete yet...

====================
Appendix
====================

Function notes:

[1]

Void cchaindomaincontext: indextoplain () {int I; for (I = m_nplainlenmax-1; I >=m_nplainlenmin-1; I --) {If (m_nindex >=m_nplainspaceuptox [I]) {m_nplainlen = I + 1; break ;}// determine the m_nplainlen Size Based on the m_nindex size. // m_nindex is generated randomly, and the following intermediate step // m_nplainspaceuptox is used to calculate the size of pxx // the size of P when I is used. // calculate the probability of P. uint64 nindexofx = m_nindex-m_nplainspaceuptox [m_nplainlen-1] // The expected offset of the password length. /* // slow versionfor (I = M_nplainlen-1; I> = 0; I --) {m_plain [I] = m_plaincharset [nindexofx % m_nplaincharsetlen]; nindexofx/= m_nplaincharsetlen ;} * /// in fact, you can use the slow version above to complete the process. // to avoid the 64-bit division operation, // when the data is still 32-bit, it is truncated to 32 bits for // fast versionfor (I = m_nplainlen-1; I> = 0; I --) {# ifdef _ win32if (nindexofx <0x00000000i64) break; # elseif (nindexofx <0x00000000llu) break; # endif // m_plain is similar to hexadecimal, but the plaintext character length is in hexadecimal format. // the last digit is first calculated. m_plain [I] = M_plaincharset [nindexofx % m_nplaincharsetlen]; // The nindexofx/= m_nplaincharsetlen size based on the content of the plaintext character;} // The 64-bit computing is complete, why do we need to calculate 32 bits? (See) unsigned int nindexofx32 = (unsigned INT) nindexofx; For (; I> = 0; I --) {// m_plain [I] = m_plaincharset [nindexofx32% m_nplaincharsetlen]; // nindexofx32/= response; unsigned int nplaincharsetlen = response; unsigned int ntemp; # ifdef _ Win32 _ ASM {mov eax, nindexofx32xor edX, edxdiv nplaincharsetlenmov nindexofx32, eaxmov ntemp, edX} # else _ ASM _ volatile _ ("mov % 2, % eax;" "XOR % edX, % edX; "" divl % 3; "" mov % eax, % 0; "" mov % edX, % 1; ":" = m "(nindexofx32 ), "= m" (ntemp): "M" (nindexofx32), "M" (nplaincharsetlen): "% eax", "% edX "); # endifm_plain [I] = m_plaincharset [ntemp];}

[2]

Void cchain1_context: hashtoindex (INT NPOs) {m_nindex = (* (uint64 *) m_hash + m_nreduceoffset + NPOs) % m_nplainspacetotal; // NPOs is to be changed, 1 is added each time. // This is why each reduce function is different. // m_nreduceoffset and raombowtableindex are related. For details, see [3] // m_nreduceoffset = 65536 * nrainbowtableindex; // m_hash is the first 8 bytes of the hash value, because the hash value may exceed 8 bytes}

[3]

Rtgen lm alpha 1 7 3 2100 8000000 allbool cchain1_context: setrainbowtableindex; // divide the table to be calculated into several tables, and the "all" (argv [8] is only the name of the table to be generated) // repeats to increase the success rate. m_neffecceoffset = 65536 * nrainbowtableindex; return true ;}

[4]

Void ccrackengine: searchtablechunk (rainbowchain * pchain, int nrainbowchainlen, int nrainbowchaincount, chashset & HS) {vector <string> vhash; HS. getlefthashwithlen (vhash, cchain1_context: gethashlen (); printf ("Searching for % d hash % s... \ n ", vhash. size (), vhash. size ()> 1? "Es": ""); int nchain1_step = 0; int nfalsealarm = 0; int nchain1_stepduetofalsealarm = 0; int nhashindex; For (nhashindex = 0; nhashindex <vhash. size (); nhashindex ++) // verify each hash {unsigned char targethash [max_hash_len]; int nhashlen; parsehash (vhash [nhashindex], targethash, nhashlen ); // string-> binaryif (nhashlen! = Cchainshortcontext: gethashlen () printf ("Debug: nhashlen mismatch \ n"); // rqeuest chain1_bool fnewlygenerated; uint64 * pstartposindexe = m_cws.requestwalk (targethash, // prepare some structure nhashlen, cchainstmcontext: Digest (), cchainstmcontext: Digest (), cchainstmcontext: getplainlenmin (), cchainstmcontext: getpllenmax (), cchainstmcontext :: getrainbowtableindex (), nrainbowchainlen, FN Ewlygenerated); // printf ("Debug: using % s walk for % s \ n", fnewlygenerated? "Newly generated": "existing", // vhash [nhashindex]. c_str (); // returns int NPOs; For (NPOs = nrainbowchainlen-2; NPOs> = 0; NPOs --){

[4-1]

If (fnewlygenerated) // whether it is a new one. the corresponding information {cchain0000context CWC; CWC is returned in requestwalk. sethash (targethash); CWC. hashtoindex (NPOs); // This is r n-1, the second R n-2int I; for (I = NPOs + 1; I <= nrainbowchainlen-2; I ++) {CWC. indextoplain (); // The three steps are CWC. plaintohash (); // F n-1.cwc.HashToIndex (I); //} pstartposindexe [NPOs] = CWC. getindex (); // The obtained value and pchain [I]. compare all items of nindexe nchain1_step + = nrainbowchainlen-2-NPOs ;/ /Step 1} uint64 nindexeofcurpos = pstartposindexe [NPOs]; // search matching nindexeint plaintext = binarysearch (pchain, nrainbowchaincount, nindexeofcurpos); // binary search if (second! =-1) // {int region, nmatchingindexeto; region (pchain, nrainbowchaincount, nmatchingindexe, nmatchingindexefrom, nmatchingindexeto) is found; // the same region is found because it is completely possible to be the same. // due to the convergence of functions, the solution space of the original hash and reduce functions is only the reduced part of int I; for (I = nmatchingindexefrom; I <= nmatchingindexeto; I ++) {// The original relevant plain text is stored in the checkalarm function. // find a specific one and put it in and exit. The hash encryptanalysisif (checkalarm (pchain + I, NPOs, targethash, HS) // judge again. A forward process. {// printf ("Debug: discarding walk for % s \ n", vhash [nhashindex]. c_str (); m_cws.discardwalk (pstartposindexe); goto next_hash;} else // if not, it indicates a false positive. False alarm. {nchaininclustepduetofalsealarm + = NPOs + 1; nfalsealarm ++ ;}}} next_hash :;}// printf ("Debug: chain walk step: % d \ n", nchain1_step ); // printf ("Debug: false alarm: % d \ n", nfalsealarm); // printf ("Debug: chain walk step due to false alarm: % d \ n ", nchainwalkstepduetofalsealarm); m_ntotalchain1_step + = nchain1_step; m_ntotalfalsealarm + = nfalsealarm; strong + = nchain1_stepduetofalsealarm ;}

[5]

Bool ccrackengine: checkalarm (rainbowchain * pchain, int nguessedpos, unsigned char * phash, chashset & HS) {cchain1_context CWC; CWC. setindex (pchain-> nindexs); int NPOs; For (NPOs = 0; NPOs <nguessedpos; NPOs ++) // push from nindexs to the corresponding position {CWC. indextoplain (); CWC. plaintohash (); CWC. hashtoindex (NPOs);} CWC. indextoplain (); +-CWC. plaintohash (); \ only makes one hash, and there is no reduce function (CWC. hashtoindex (NPOs) is used to verify if (CWC. checkhash (phash) // verify the function, compare phash with the generated function {printf ("plaintext of % s is % s \ n", CWC. gethash (). c_str (), CWC. getplain (). c_str (); HS. setplain (CWC. gethash (), CWC. getplain (), CWC. getbinary (); // return true;} return false;

[6]

The purpose of requestwalk is to generate a hash to match pchain [I]. nindexe array. uint64 * cchain0000set: requestwalk (unsigned char * phash, int nhashlen, string signature, string signature, int nplainlenmin, int nplainlenmax, int nrainbowtableindex, int signature, bool & fnewlygenerated) {If (m_shashroutinename! = Shashroutinename // if the corresponding parameter changes, all the items are reset. | m_splaincharsetname! = Splaincharsetname | m_nplainlenmin! = Nplainlenmin | m_nplainlenmax! = Nplainlenmax | m_nrainbowtableindex! = Nrainbowtableindex | m_nrainbowchainlen! = Inline) {discardall (); // <----------- inline = inline; inline = splaincharsetname; inline = nplainlenmin; inline = nplainlenmax; m_nrainbowtableindex = nrainbowtableindex; inline = inline; chainwalk CW; memcpy (CW. hash, phash, nhashlen); CW. pindexe = new uint64 [nrainbowchainlen-1]; m_lchainwalk.push_back (CW); fnewlygene Rated = true; return CW. pindexe;} List <chainwalk >:: iterator it; for (IT = m_lchain0000.begin (); it! = M_lchain0000.end (); It ++) {If (memcmp (IT-> hash, phash, nhashlen) = 0) // checks whether the hash is new, {fnewlygenerated = false; return it-> pindexe; // If so, return the pointer to the 8-byte array} chainwalk CW; // chainwalk has two structures, one with the hash value, the pointer memcpy (CW. hash, phash, nhashlen); CW. pindexe = new uint64 [nrainbowchainlen-1]; // a storage used to match pchain [I]. array of mindexe. m_lchainwalk.push_back (CW); // if there is no whole hash, create a new one and add it to the listfnewlygenerated = true; return CW. pindexe; // The returned result is an array pointer with 8 bytes}