Advanced Shell Tutorials

Background: I have summed up some knowledge points about my common shell scripting style. Also as part of the handover work document. Part of the content is written separately

#!/bin/sh# Shell Script Advanced Tutorial # 1. Common knowledge Points: Variable setting/date setting/formatting output/defining function/function parameter/footstep parameter/variable nesting and iteration # 2. Common environment:/database monitoring/local log monitoring/batch processing/regular FETCH table data/Backup # 3. Common circulation: for/while# 4. Common commands: sed/cut/awk/# 5.crontab Scheduled Tasks # Part I: Common knowledge points # 1. "Variable settings and variable substitution" # 1.1 global variables and local variables-position in script to differentiate # Global variables: directly defined in script # Local variables: In the loop statement/Custom function # from the position order: # Global variables: Generally at the beginning of the script, set first, or before the function definition/loop statement is defined. # Local variables: For local calls, such as in loop statements/custom functions; # 1.2 constant variable and dynamic variable-the value of the variable changes # format: # constant variable: eg:xxx= "or xxx=" "or or xxx=120 (number or The definition of the port does not use quotation marks) or xxx= "" "Error format: The same quotation mark cannot be applied xxx=" "" #动态变量: eg:xxx= "' anti-quote: The key on the left of the number 1 on the keyboard #动态变量, two common settings: #xxx = ' shell command ': direct the shell The result of the command is assigned to XXX (xxx= ' cat/data/a.txt|grep ' 3306 "will contain 3306 of the line directly assigned to the XXX variable) #xxx = ' $cmd ': cmd=" Hello World "(the contents of the CMD variable are assigned to XXX, CMD can be a constant variable or it can be a dynamic variable) #单引号和双引号和多个引号的区别: #单引号: Is a full reference: Give two examples: #1. \ n: The meaning of the carriage return T: is the meaning of the tab space definition: str1= ' hello World!\nhello world! ' output: Hello World!\nhello world! Output as-is. Define the SQL statement: sql1= ' select * from db1.tb1 where name= "Joho" and time> "$start _time" ' because the output is the same as the start_time variable will not be successfully replaced # Double quotation marks: output literal value except for special characters: for example, escape character: \ t \ r \ n variable substitution: $ #1. Definition: str2= "Hello world!\Nhello world! " Output two lines #2.sql2= "select * from db1.tb1 where name= ' Joho ' and time> ' $start _time '" Correct! Time must be quoted. #多个引号 (single or double quotation marks): Use environment: The variables defined include: single quotes, double quotes, variable substitution, and anti-quote #我使用过的唯一的例子: When you build a table in bulk: # seg01: Bulk Print the Build table statement 00~99, not very common for I in SEQ ' 0 "do num= ' printf '%02d ' $i ' #格式化0 to: 00~09 (two digits) sql= "" "CREATE Table ' bldb '. ' Log_message_ ' ' $num '" (' id ' int, ' name ' varchar ( Primary key (ID)) Engine=innodb; "" # 3 double quotes are quoted because: the Library name, table name, and field names of the statements exported by the SQLyog tool are quoted in reverse quotation marks. # 3 quotes inside the variable substitution, must use 3 single quotation marks, in order to take effect. # Manually remove the inverted quotation mark without 3 double quotes. done# Variable Example: user= ' admin ' paswd= ' pssword ' host= ' 172.2.1.1 ' port=3306dir1= '/data/scripts ' file1= ' ${dir1}/log.txt ' # Common variable substitution datemark= ' date ' +%y-%m-%d%h:%m:%s ' # dynamic variable sets the current time variable: 2017-08-06 22:09:00 # 2. "Date/Time setting" # Table name with date/time flag in table in time Field/script # Classification of time and date: Current time/n minutes before/n hours before/n minutes formatting of/# time:%Y: Year%m: Month%d: Day%H: Hours%m: minutes%s: Seconds # 2.1 Current time: nowtime= ' date ' +%y-%m-%d%h:%m:%s ' # 2017-08-06 22:09:00nowtime= ' Date ' +%y%m%d%h%m%s ' #20170806 220900# 2.2 n minutes ago: n minute ago/n day ago Beforetime= ' date-d ' 2 minute ago ' +%y-%m-%d%h:%m:%s ' # 2.3 Time range: last 5 minutesClock # No matter what the environment, the current time is not counted, because the current minute is incomplete # limit for time fields in SQL query conditions: Format style to match the format of the table Starttime= ' date-d ' 1 minute ago ' +%y-%m-%d%h:%m:%s ' Endtime= ' date-d ' 5 minute ago ' +%y-%m-%d%h:%m:%s ' # 2.4 table name: bldb.log_message_20170806#tb_mark= ' Date ' +%y%m%d ' # 20170806tb_name= "Bldb. Log_message_ ' Date ' +%y%m%d ' "# 1. Direct write: Tb_name=" BLDB is not recommended. Log_message_${tb_mark} "# 2. Variable substitution: Most commonly used: easy to modify # 2.5 time markers in Footsteps #echo" [${nowtime}]: succeeded! "#执行命令后 print output # 3." Formatted output ": # Benefits:    #1. Specification requirements for the output, such as: 00~09 instead of 0~9; floating-point number requirements: generally used to build 100 table # # #. Left and right: output aesthetic requirements, row and column alignment for easy viewing results # 3 in common: value/character/floating-point # Value #seg02:for i in ' seq 0 9 ' Do printf "%02d" "$i" # output 2 bits, not enough with 0 to complement, formatted directly after printing output done # string printf "%10s%s" "hello!" "It ' s a dog!" # output Two fields, the first field is 10 characters, default right alignment, insufficient left space to complement printf "%-10s%s" "hello!" "It ' s a dog!" #-: Left-justified, right-hand space # floating-point number num=98.2245printf "%2.2f" "$num" # formatted: 2 digits to the left of the decimal point, 2 digits to the right. Used for success rate/percent # 4. "Define function" # 4.1 general function (no reference) #定义格式: Get_info () {name= ' Jhon ' echo ' My NAME is ${name} '}# function call method: Direct call to function name # Direct script writes the name GET_INFO#SEG03: Full script vim getinfo.sh#-------------------------------(dashed line is useless) #!/bin/sh#shell script start setting # def:function Get_info () {name= ' Jhon ' echo ' My NAME is ${name} '}# excute:functionget_info#------------------- ------------# executes scripts and outputs./getinfo.sh My name is jhon# 4.2 define the parameter function: # Definition Format: function with 2 arguments # Function:get_info () {Port=$1ip=$2ech  o "MySQL PORT is: ${port} and MySQL HOST was ${ip}" result= ' mysql-uadmin-p ' password '-h${ip}-p${port}-nbe "SELECT * FROM Urcs_bldb. log_message_01; " ' # Read the table data, assign the output to the result variable echo "$result" >>/data/log.txt #将结果写入指定文件}# Call method: function name parameter 1 parameter 2get_info 3306 ' 172.21.1.1 ' GE T_info 3307 ' 172.21.1.2 ' # above is a two-time call function. Suitable for similar result output, different incoming variables # usually the function of the parameter and the script is used in conjunction with the reference to do the monitoring item is very force. The current network monitoring is doing so # 5. Script Reference # Environment: Use a script to handle similar operation classes. For example: Different MySQL with the port to distinguish, each MySQL monitoring item is the same. The only difference is the port, so the port can be used as a variable, as the script's parameter # script invocation: a script parameter./getinfo.sh all output in the 3306# script is 3306 database related./getinfo.sh 3307# All the output in the script is 3307 database-related # script parameters in the script to reflect the # script at the beginning of the position, set the script parameter variable db_port=$1# to represent the first parameter after the script, the second argument after the script; The script parameter is assigned to: Db_port. The Db_port variable can be referenced directly in the script to represent the port. # If the string is a script parameter, there are special symbols or spaces in the string that need to be enclosed in single quotes. #seg04: The script passes through the incoming port toMonitor a monitoring entry for the DB instance #!/bin/shdb_port=$1db_item=$2# represents a monitoring entry: such as number of connections, master-slave status .... Can be seen as a trigger action, which will say, in short, this is the amount of change echo "$db _port: $db _item" # execution mode:./getinfo.sh 3306 conn# output: 3306:conn# 6. "Nesting of script variables" # 6.1 general nested # seg05:#!/bin/sh# SQL query where Condition s1= ' timeout ' s2= ' error code ' ... # define function: Generate SQL statements based on incoming criteria , assign a statement to a variable and pass the variable to another SQL query function, returning or writing the query result to the file Create_sql () {s=$1sql= "SELECT * from Urcs_bldb. log_message_01 WHERE MESSAGE like "%" $S "%"; # regarding the use of quotation marks, the purpose is to generate the SQL that is normal on the line. echo $SQL}# definition SQL query function get_info () {sql=$1mysql-uadmin-p ' pasd '-h172.21.1.1-p3306-nbe "$sql";} Sql1= ' Create_sql "$S 1" # generates SQL statements from functions, assigns variables sql1sql1_data= ' get_info ' $SQL 1 "' #传入SQL1, functions produce query results, and assign values to sql1_data variables. echo $SQL 1_data >sql1.txt # writes the contents of a variable to a file # The second Part # Common environment # 1. "Database Performance parameter Monitoring" # monitoring script Ideas: # (1). There are a lot of monitoring items to avoid accessing the database every time, so by: show Global Status\g writes the results to a file # (2). Each monitoring item, defining a function to read information from a file, processed returns # (3) . triggers a function and returns a value by means of a script pass./xxx.sh action# (4). Multi-instance case, by port to differentiate, multiplexing script. This is accomplished by scripting:./xxx.sh Port Action # (5). There are n monitoring items, and one minute will trigger the script n times. The file is generated only when the first trigger is triggered. So you need to define the function, write the status information to the file, check if the file exists, jumpNot exist, read the information and write to it. # (6). Generate files with a time flag, so a cleanup mechanism is required. Define a cleanup function as a monitoring item that is called every 30 minutes in Zabbix. # Shorthand script: master-slave status monitoring # seg06:#!/bin/sh#. /etc/profile#. /etc/bashrc# name:slave.sh ############## definition MYSQL command variable: mysql_bin= "/usr/local/mysql5715/bin/mysql-umonitor-pmonitor "# define time flag for makefile info_time= ' date +%m%d%h%m ' # define output file directory Info_dir="/usr/local/zabbix/scripts/tmp/"# define MySQL Port mysql_port= ' $ ' # definition of generated filename info_file= '/usr/local/zabbix/scripts/tmp/slave_info_${mysql_port}_${info_time} ' ############## define function , determines whether the file exists, does not exist, gets slave information, and writes to the file; skips Get_info () {if [[-F ${info_file}]]; then Breakelse ${mysql_bin}-s/data/socket /mysql${mysql_port}.sock-e ' SHOW SLAVE STATUS \g ' >${info_file}fi}# definition from library delay function Seconds_behind_master () {sbm= '/bin/egr Ep-w ' Seconds_behind_master ' ${info_file} | awk ' {print $} ' echo ${sbm}}# defines SQL thread state slave_sql_running () {thd_sql= '/bin/egrep-w ' slave_sql_running ' ${info _file} | awk ' {print $} ' if [[${thd_sql} = ' No ']; Then echo 0 Else echo 1 fi}# define IO thread state SLAVe_io_running () {thd_io= '/bin/egrep-w ' slave_io_running ' ${info_file} | awk ' {print $} ' if [[${thd_io} = ' No '] ] ; Then echo 0 Else Echo 1 fi}# defines whether the function is master or from R_EMLP () {rmlp= '/bin/egrep-w ' Read_master_log_pos ' ${inf O_file} | awk ' {print $} ' emlp= '/bin/egrep-w ' Exec_master_log_pos ' ${info_file} | awk ' {print $} ' R_EMLP=$[${RMLP}-${EMLP}] echo ${r_emlp}}###################### #GET_INFO # Triggers every time the script executes: the function that gets the information # The following two variables if = 4, the description is the main library, the return value: 3,zabbix judged to be the master, will not trigger the alarm rmlp= '/bin/egrep-w ' Read_master_log_pos ' ${info_file} | awk ' {print $} ' relp= '/bin/egrep-w ' Relay_log_pos ' ${info_file} | awk ' {print $} ' # below: Case $ In, which, by judging the second argument passed by the script, triggers the corresponding function and returns the corresponding value if [[${RMLP} = 4]] && [[${relp} = 4]] ;        Then case $ in SLA_SBM) echo 3;;        Thd_sql) echo 3;;        Thd_io) echo 3;;        Sla_rem) echo 3;; *) Esac else case $ in SLA_SBM) Seconds_behind_master;;        Thd_sql) slave_sql_running;;        Thd_io) slave_io_running;;        Sla_rem) R_EMLP;; *) esacfi# define cleanup function # define cleanup function: Clean every 30 minutes, clean up the previous minute every time Glo_clear () {/bin/find ${info_dir}/*_info_*-ctime-1 | Xargs rm -ffile_num= '/bin/find ${info_dir}-name "*_info_*"-ctime-1 | Wc-l ' If [[${file_num} = = 0]]; Then echo 1else echo 0 fi} #测试:./slave.sh 3306 Thd_sql will get a return value # 2. "Local log log, keyword query" # SQL query local database table, total number of rows with error keywords, # script idea # ( 1). Error keyword query Many, in the standard format, put in a separate file. The script reads the file through a loop and makes a SQL build # (2). The results of the SQL query are stored in the corresponding file # (3). The output information has many field conditions, so the information is processed by other scripts # (4). # Shorthand script #!/bin/sh# define variable host= "localhost" port=9306user= "root" pswd = "root" db_name= "Logdb" tb_name= "Log_" ' Date + '%y%m%d ' mysql_cmd= '/home/mysql/percona-server/bin/mysql-u${user}-p$ {pswd}-h${host}-p${port}-s/tmp/mysql.sock "# time variable set, value range one minute before data time0= ' date + '%y-%m-%d%h:%m ' # Current time time1= ' date-d ' 1 Minutes ago "+ '%y-%m-%d%h:%m" # Last minute: 2017-08-05 10:34 seconds Add when SQL is defined The directory that defines the output file for the result: # tmp_dirfile_dir= "/data/scripts/shelldir" # defines the error keyword format file keyfile= '/data/scripts/keyfile.txt ' content: # # # # Too many connections### #connect. txt####### #timeout # # # #timeout. txt#### #定义SQL生成函数: Parameters: Keyword Get_sql () {key=$1 sql= ' Select A.servicename,ifnull (b.count_error,0) as Count_total from (select DISTINCT (ServiceName) from "${DB_NAME}.${TB _name} ' WHERE (Time >= "" ' ${time1} ': "" "" and Time<= "'" ' ${TIME0} ': "") as A left join (select ServiceName, COUNT (*) as Count_error from ' ${db_name}.${tb_mark} ' WHERE (Time >= "" ' ${time1} ': "" and Time <= "" ' ${ TIME1} ': ' and ' (Message like '% ' ' ' ${key} ' '% ' OR ' Error like '% ' ' ' ${key} ' '% ') group by ServiceName) as B on A.servic    Ename=b.servicename GROUP by A.servicename; " echo $SQL}# Define SQL query function: parameters: SQL statement, file name Get_info () {sql=$1file=$2 ${mysql_cmd}-nbe "${sql}" >>${file_dir}/${file}}# Iterate through a keyword file, sql generation, SQL query # Main program Cat ${keyfile}| While read linedo# intercepts keywords and output file names key= ' echo ${line}|awk-f ' # # # ' {print '} '' Outfile= ' echo ${line}|awk-f ' # # # ' {print $} ' # Generate SQL statement sql= ' get_sql ' $Key ' # Query results write to file Get_info ' $SQL ' $Outfile ' Don e# 3. "Batch processing" # commonly used for I in ' seq 0 ' do echo ' truncate table xxxx_$i ' done# command line: For i in ' seq 1 ';d o echo ' truncate tab Le xxxx_$i; ";    Done # 4 Batch Build Table program # #create tables#!/bin/sh#db_name= ' use renmaiinfluencedb ' for I in {0 *};d o len= ' expr length $i '  If [$len-eq 2];then num= $ielse num= "0${i}" fi echo ' CREATE TABLE ' logregister_ ' ' $num ' ' (' Id ' int ') not Null auto_increment COMMENT ' self-increment id ', ' awardid ' int (ten) NOT NULL COMMENT ' prize ID ', ' UserId ' int (ten) not null COMMENT ' winning user ' , ' couriername ' varchar () default NULL COMMENT ' Express name ', ' couriernumber ' varchar (+) default null COMMENT ' waybill number ', ' Create Date ' timestamp not NULL DEFAULT current_timestamp COMMENT ' winning time ', PRIMARY key (' Id '), key ' Ix_signlotterywinner_userid_ CreateDate ' (' UserId ', ' createdate ')) Engine=innodb auto_increment=1602 DEFAULT Charset=utf8 row_format=dynamic    Comment= ' punch card winning table '; ‘‘' Echo done===================================================================================================== ===#!/bin/sh#creat tablesfor i in ' seq 1 ' do var= ' printf '%02d\n ' $i ' echo ' ' CREATE TABLE ' logregister_ ' $var  ' ' ' (' id ' int () NOT NULL auto_increment COMMENT ' self-increment id ', ' awardid ' int (ten) ' NOT null COMMENT ' prize Id ', ' UserId ' int (10)  Not NULL COMMENT ' winning user ', ' couriername ' varchar default null COMMENT ' Express name ', ' couriernumber ' varchar (+) default null COMMENT ' waybill number ', ' createdate ' timestamp not NULL DEFAULT current_timestamp COMMENT ' winning time ', PRIMARY key (' Id '), key ' ix_s Ignlotterywinner_userid_createdate ' (' UserId ', ' createdate ')) Engine=innodb auto_increment=1602 DEFAULT CHARSET=utf8       Row_format=dynamic comment= ' punch-card winning table '; Echodone============================================================================#!/bin/sh # needs to be "'" in SQL Symbol Delete for i in ' SEQ 0 1 "Do step= ' printf%02d ${i} ' create_tab_sql=" Create TABLE Logregister_${step} (Id int (one) NOT null auto_increment COMMENT ' table ID ', UserId int (one) not NULL COMMENT ' user ID ', UserName varchar (+) COLLATE utf8_unic Ode_ci NOT null COMMENT ' username ', Lotterynumber tinyint (1) NOT null COMMENT ' remaining draw today ', Statetime date not null Commen  T ' current date ', lottery tinyint (1) NOT null default ' 0 ', Award_times tinyint (1) NOT null default ' 0 ', Feitionid Int (11) Default NULL COMMENT ' user fetion number ', PRIMARY key (Id), key idx (UserId) USING BTREE) Engine=innodb default Charset=ut    F8 collate=utf8_unicode_ci comment= ' daily user-available game Count table '; Echo-e ${create_tab_sql} echodone=============================================================================== ====#!/usr/bin/pythonimport stringfor i in Range (7,13): sql = "CREATE TABLE ' ugcfeedcontent_2015%02d ' like ' Ugcfeedcont ent_201503 '; '% i print (SQL) for I in Range (1,13): sql = "CREATE TABLE ' ugcfeedcontent_2016%02d ' like ' ugcfeedcontent _201503 '; "% i print (SQL) ======================================== the rest of the sections are written separately

　　

Advanced Shell Tutorials