Log Quick Filter Linux command Grep|uniq|wc|awk

In the past, my personal idea is that things run on the line try not to remember any log of water.

But then I changed, found online diary is an absolutely necessary things, especially in the current very popular micro-services, without the help of the log, like a blind, troubleshooting the basic guess, this can not.

Then print the log of each visit, especially the parameters of the interface and the return data and time consuming, which is the basis for the problem to be thrown to the upper layer and performance optimization. But the log volume should be very large, must pay attention to clean up in time.

So the problem is, when it comes to finding a problem, it's important to quickly locate the wrong place.

The log sample is as follows (the resulting log for a visit):

[ .- on- -  -: -: -] New [email protected]:112.65.141.22[ .- on- -  -: -: -] New [email protected]:112.65.141.22Param: <xml><tousername><! [cdata[gh_27923assff]]></tousername> <fromusername><! [cdata[osj3ut7w0vqgu-6k1wssdmaqpvu]]></fromusername> <CreateTime>1484386723</CreateTime> <msgtype><! [cdata[text]]></msgtype> <content><! [Cdata[http://Satwx.ddd.com/wap]] ></Content> <MsgId>6375392430314143735</MsgId> </xml>[ .- on- -  -: -: -]Cat@http://192.168.1.125:30008/eagleeye/monitor/logevent @visit, msg=> New [email protected]: 112.65.141.22 ...[ .- on- -  -: -: -] API begin: [Get] http://192.168.1.11:40003/micsite/api/ss/microsite/1.1/wchat/getacken?wchatname=wx477026f155386[ .- on- -  -: -: -] API End:cost:0.0081648826599121Sec,return data: {"Error":"4000063","msg":"No accesstoken information for this public account!","Status":"0"}[ .- on- -  -: -: -] Get token error [Code:status0]: {"Error":"4000063","msg":"No accesstoken information for this public account!","Status":"0"}[ .- on- -  -: -: -] API begin: [Get] https://API.WEIXIN.QQ.COM/CGI-BIN/USER/INFO?ACCESS_TOKEN=&OPENID=OSJ3UT7W0VQGU-6K1WHVI5MAQPVU&LANG=ZH_CN [ .- on- -  -: -: -] API End:cost:0.08442211151123Sec,return data: {"Errcode":41001,"errmsg":"access_token missing hint: [ohhxoa0723vr30!]"}[ .- on- -  -: -: -] Get Unionid Error: {"Errcode":41001,"errmsg":"access_token missing hint: [ohhxoa0723vr30!]"}[ .- on- -  -: -: $] POST https://A1.easemob.com/ddd/xyytest/token {"Grant_type": "Client_credentials", "client_id": "Yxa6895cuk3_eew3yseu_isqrq "," Client_secret ":" Yxa61qdxmd9yvncxd9zahrcmfftrz3m "}[ .- on- -  -: -: $] Return Data: {"Access_token":"ywmtf7ebhtfweeabvquozfw40qaaaa3a67aaaaaahz3lxqrf8r5bdiwr4fzyghzdwbpggddng322dddssvrzggb2_ Tqflf-cf4feo07wzyzjueq","expires_in":4205288,"appLication":"F3de5c50-adff-11e5-b762-c114fe2b2a45"}, cost:0.19826197624207 sec[ .- on- -  -: -: $] POST https://a1.easemob.com/dd/xyytest/messages {"Target_type": "Users", "target": ["Custom"], "msg": {"type": "TXT", "MSG": " Http:\/\/satwx.shoujidai.com\/wap "}," from ":" Hx_callback "," ext ":{"Weichat":{"Visitor":{"Source":"Weixin","msgId":"6375392430314143735","OpenID":"Osj3ut7w0vqgu-6k1whvi5maqpvu","MP":"wx812302f2923a0e2f","Usernickname":NULL}}}}[ .- on- -  -: -: $] Return Data: {"Action":"Post","Application":"F3de5c50-adff-11e5-b762-c114fe2b2a45", "Path":"/messages"}, Cost:0.15990281105042sec[ .- on- -  -: -: $] Output: (empty) [ .- on- -  -: -: $]----This TimeCost0.52057886123657SEC----

Question 1, I would like to see how many independent access to this machine? (There may be multiple days of access logs)

grep " \[2017-01-14 [0-9]{2}:[0-9]{2}:[0-9]{2}\] [0-9]+\. [0-9]+\. [0-9]+\. [0-9]+"awk'{if (!ip[$3]++) print $" ===> "$" "$" "  $4;} ' Sort Uniq  Less

To count the total number of IPs, simply followed by a wc-l pipe command:

Cat WC -L

Question 2, found that an external network interface is slow, how to see the interface is slow where there are?

grep " api.weixin.qq.com/cgi-bin/ " grep " api.weixin.qq.com " awk ' $6 > 5 '  Less     #grep -WC -L  #统计总共有多少地方

Question 3, false with error generation, but not sure access to the machine, whether you can see the access log and error log?

tail -F access_log error_log   dynamically prints both the access log and the error log, which proves access to the machine and can see the error if it occurs at the same time

4. How do I clean up the log? Run the timed cron script with crontab. Crontab-e

filepath=/data/logfind filepath-mtime +3'*.log'   RM

Question 5, temporarily check whether an interface is unobstructed? Using the Curl command

Curl-i http://coe.test.com:8080/micro_site/api/user/info  #加-I participate in return header information, more commands to view the Help document Ping  coe.test.com  #直接看该域名是否可用 #如果需要, or you can use the telnet command 

6. Using a Docker container to manage the test environment, some simple Docker commands should know

PS grep MyServer     -it myserver/bin/bash      #进入自己的docker Container

Question 7, other tricks?

Netstat-TUNLP          #查看网络端口监听情况 to determine if the server has problems PSgrep  httpd    #查看进程相关 pstree more vivid                 grep -F-F a.txt  b.txt #求文件交集 sortuniq -d  #求两文件交集 Sort uniq -u   #求两文件不重复的项

Speaking of log screening, in fact, just want to locate the problem where, here is a few more, for later use it!

For PHP interpreted language, debugging is relatively convenient, directly on the server to get rid of the attempt, but there is also a disadvantage is that there is no compiler to check your code, will hide some syntax errors. In general, printing the access logs together with the error log will allow you to quickly resolve the issue. Syntax errors are relatively well resolved, logic errors need to locate the code location, careful analysis to solve.

For compiled languages like Java, debugging is a bit trickier. Generally, when you change a small or one or two files, you can restart the server for quick debugging by replacing the. class file, but changes to multiple files are basically repackaged and uploaded. The class file of the war package can be easily replaced by FTP work. and. jar package is troublesome, you need to download the jar package on the server to local, with the compression tool opened, replace the class file directly, then upload the entire jar package. Java logs are relatively large, so cleaning up files can be more or less frequent.

Using SVN, git and other tools for code management is very convenient, but sometimes file modification, it is easy to forget to add the files, most will cause classnotfoundexception, quick fix.

For the nullpointerexception exception, locate the row that raised the error, observe which variable might be causing the error, possibly the interface response, and possibly the database problem.

For OutOfMemoryError exceptions, add memory or optimize code resolution.

Log Quick Filter Linux command Grep|uniq|wc|awk