Shell routines--resolving Redis Read stability

problem background : PHP reads the REDIS data on the line, which is often unstable, and the data response is sometimes not.
workaround : Read multiple times. Read all the last unread data until you get it all.

In this paper, the shell is used to read multiple Redis data, each time the valid value (for our example, is to key, can get its value on the Redis is a valid value.) Other invalid), and re-run the invalid value again to iterate until all redis data has been fetched. Ps:redis data can be read by PHP or C, given the interface is very easy, in detail to participate in Phpredis. The implementation of this piece is not given in this article because of the possible involvement of the secret.

1. Method Overview:
   
   

   1. divides a key in source into n files. Throw in Redis parallel get value
   2. real-time statistics n files get value output total number of rows, and the number of valid lines
   3. n files after the end of the statistics, the full results are merged into Result/step/stepx.re s
   4. Deletes the previously parallel source file, and the resulting file
   5. places the key not acquired in result into source/step2/contsigns. As input for the next round. Divide it into n files again, run (the contsign is the Redis key)
   6. and finally write the results of each step to the Final.res file. Cat result/steps/step*.res >> final.res

   
   

2. Project structure:
   
   

   
   

   • getredis.php: Implementing get Redis Data
   • all.sh: Main program, run getredis.php in parallel;
   • analyze_result.sh: Real-time analysis of Redis get Data operation (2nd step), after the addition of the 第3-5 step (see the next section in detail);
   • source/: Stores the input data, where all/is the input for all of the original Redis. STEP2/is the key that is not acquired in this round, and will be the next round to get the Redis input, the remainder (such as XAA) is the n files that are currently in this round;
   • result/: Stores the result. The source/includes the output of all n files under the current round of source; steps/includes the results of the merge after each wheel output
     
     
     
     

3. Detailed implementation:
   
   
   all.sh:

 #Author: Rachel Zhang  # Email: [email protected]  for  file in  source /*do  {echo   $file  " if test -f   $file  then  php getredis.php  $file  > Result/ $file  fi  echo   $file  done ... "}& done  

Analyze_result.sh:

#Author: Rachel Zhang#Email: [Email protected]filefolder=result/Source/*#Filefolder =source/*Echo "##################################"Echo "in Folder $Filefolder"Echo "##################################"Nl=0hv=0 forFileinch $Filefolder Do    ifTest- F $file     ThenFline= ' WC- L $file| Awk' {print '} '' Nl=$ ($NL+$fline)) Fvalue= ' Cat$file|awk' begin{x=0;} {if ($) x=x+1;} End{print x;} '' Hv=$ ($HV+$fvalue))fi DoneEcho "Totally $nl lines"Echo "$HV lines have tag value"###################################combine results into one fileif["$#" -GT 0] Then    ifTest- e "Result/all.result"     ThenMV Result/all.result Result/all.result.bakfi     forFileinch $Filefolder     Do        ifTest- F $file         ThenCat$file>> Result/all.resultfi     Done    Echo "All the output is write in Result/all.result"    # put Null-value keys into source/step2/contsigns    ifTest- e Source/step2 ThenMkdirSource/step2fiCat result/all.result| Awk' {if ($2==null) print} '>Source/step2/contsigns nnull_value= ' WC- L Source/step2/contsigns | Awk' {print '} '`Echo "remaining ${nnull_value} keys to being processed"    # put Has-value key-value into Result/steps/step${step_id}.resStep_id=1     whileTest- eResult/steps/step${step_id}. res Dostep_id=$ (($step _id+1)) DoneCat Result/all.result | Awk' {if ($) print} '> Result/steps/step${step_id}. resEcho "Current valid results is writen in Result/steps/step${step_id}. res"    # Remove the current source files, generate new source files and re-run all.sh    Echo "Remove current source and result files?"
 (y/n) "read  answer if  [ $answer  = = " y "]; then  if  [ $Nnull _value  -gt  10 ]; then  rm source /* RM Result/  source /* CD  sourc e  && split -l  5000  step2/ Contsigns && CD  ... / echo   "now re-run Sh./all.sh?  
 (y/n)" read answer if [ $answer == "y" ]; then sh all.sh fi fi fifi

PS: The above analyze_result.sh can also remove the interactive part of the analyze_result.sh (no user input required). Through the crontab to make the timing, further self-initiated operation.

Shell routines--resolving Redis Read stability