Java Programmer's Path to Logging (3/3)-How to parse log

1. Description

As a programmer, we often have to do one thing: Get a log file and find out the information you want. This article summarizes what tools I used to analyze the log file to get the information I want, my work environment in recent years is the server on Linux, the work machine is windows, so I use the tool is mainly Linux on some command-line tools, Of course they can also be used on Windows, specifically see the following tool.
First of all, I'm just a very normal Linux user, so if some Linux commands or tools are not efficient, please educate me.

Digression: My view is that as a Java programmer, just need to know the operating system in the scope of their own work to be able to become an expert is good, but is not necessary, such as: I use Windows every day, However, you never know how to adjust the maximum number of network connections by editing Group Policy or the registry, but almost all of the optimization tools have this capability.

2. Tools

• Tools to connect to a remote Linux server: Mremoteng official website seems to be a wall, can be in the Chinese army celestial pole such as download
• Tools for executing the Linux command line on Windows: Cygwin or Git bash if neither of you is faking it, my advice is to install a git bash, which is basically enough and easy to setup.
• Text Editor: VI, occasionally under windows with notepad++ because the general case of the log file will have a dozens of or even hundreds of m, with the VI direct view comparison directly, and open relatively fast.
• Text analysis: Excel, but this time write something at home, so I'll try to use the wps form in Jinshan WPS later

3. Analyze Log3.1 Search text

Analysis log first is to find the log you need, in general, we need the log is so positioned

1. Find the corresponding line according to the exception information or request information
2. The relevant log in the context is found based on information such as the time, thread name, and keywords of the line

Let's talk about how the above two steps should be done separately.

3.1.1 Find the corresponding line, and some of the files

In general, when we start analyzing log, we have this information to assist us in our search.

• Parameters when an error occurs
• Exception information when an error occurs
• Time
• If there is no information, just know the error, then we can try to search "error", "Warn", "Exception" and the Like

3.1.1.1 If you know that log is in a specific file, I usually open the file using the VI read-only mode (assuming the file name is Access.log)

Vi-r Access.log

Go in and let it show the line number:

: Set Nu

Jump to the first line:

Gg

or a

1G

Skip to the last line:

0G

or a

G

Skip to Line 11th:

11G

Search for strings from the head of the file Hello World:

:/hello World

Search string forward from end of file Hello World:

:? Hello World

Skip to the next place that matches your search criteria:

N

Skip to the previous location that matches your search criteria:

N

VI inside with go abbreviation letter G Jump to jump, the letter n is the abbreviation of next Exit VI command is:

: q!

3.1.1.2 If you don't know which file it is, I usually use the grep command as an example: print the search results to the screen, and flip the page with the SPACEBAR

Grep-nr "Hello World"./*.log | More

Print search results to file HelloWorld.log

Grep-nr "Hello World"./*.log > HelloWorld.log

3.1.2 Find Context

Follow the above steps to find clues, if the interference information is not much, you can browse up and down. So if there is more interference information, I generally use the following methods to obtain the information I need without disturbing the log.

• Filter the required log according to the thread name and keyword of the request I usually use the grep command above to do
• All logs in that period are obtained based on a parsed start and end time (such as the entry of a Web request and the writeback time of the resulting information). I usually open with VI first, then find the line number of the first line of the desired content and the line number of the last line, and then use the SED command to get a few of these lines. The result of this command is to save the contents of line No. 200 to 266 in Source.log to the Result.log file.

  Sed-n 200,266p source.log > Result.log

3.2 Log Analysis In addition to finding clues in log, sometimes it is necessary to do batch analysis of the log, I have encountered such as analysis of the time of the occurrence of an error, the relevant ID of the error, and so on. Need to use some statistics or something. 3.2.1 Simple I generally extract each scene into a single file, then use the WC (don't laugh, it's the abbreviation for Word count) command to calculate the number of rows, as shown below

Wc-l Result.log

Complex I usually use Excel or Jinshan wps for file analysis. 3.2.2 using Excel or Kingsoft wps for file analysis

1. First, start a new worksheet
2. and copy the text you want to analyze.
3. Then use the * * * * (menu: Data-to-columns) function to split the log file into a single table
4. And then use the filtering sort PivotChart to analyze

3.3 Other Useful commands

• Open a log file and watch it update continuously as the system runs

Tail-f Abc.log

• Monitor changes to multiple files at the same time

Tail-f./*.log

• Look at the 20 lines at the beginning of a file

Head-20 Abc.log

• See 50 lines at the end of a file

Tail-50 Abc.log

Java Programmer's Path to Logging (3/3)-How to parse log