Golang Example 001

This is a creation in Article, where the information may have evolved or changed.

Part 01 Target

Filter the log files and split them;

Part 02 Understanding and thinking

This is the job assigned to me by the internship company when I am about to fade out of the bird. There are several log files, very large, in G units. There's a lot of unnecessary information that needs to be filtered out. In addition, the log is recorded together, when analyzed, there is no reason, so, the request according to the date, split, the same day the log for a file, named after the date file.
The focus of programming is the use of regular expressions and file manipulation .

Part 03来 a few lines to see

Because you need to construct a regular, you have to know the format of the file. Directly open the file is an unwise choice, for me, accurate point, for my computer, open large files, my little laptop can not bear such a burden. In a Linux environment, a command is actually available:

head -n 100 # 显示前面100行；tail -n 100 # 显示最后100行；sed -n '5,10p' filename # 这样可以只查看文件的第5行到第10行；

is not very convenient. However, I want to practice the Golang, so it took a little time to write a small program:

//Auth:reluxer//Time:2015-07-23//Name:preview.go///Description: Busy, wrote two functions PackageMainImport("Bufio"    "FMT"    "IO"    "OS")Const(Viewlinenum = +)funcReadFileByLine002 (filenamestringError {F, err: = OS. OpenFile (filename, os. O_rdonly,0660)ifErr! =Nil{returnERR}deferF.close () SC: = Bufio. Newscanner (f) forI: =0; i < Viewlinenum; i++ {SC. Scan () fmt. Println (SC. Text ())}returnSc. ERR ()}funcReadFileByLine001 (filenamestringError {F, err: = OS. Open (filename)ifErr! =Nil{returnERR}deferF.close () buf: = Bufio. Newreader (f) forI: =0; i < Viewlinenum; i++ {line, err: = buf. ReadString (' \ n ')ifErr! =Nil{ifErr = = Io. EOF {fmt. Println ("EOF") Break}returnERR} FMT. Print (line)}return Nil}funcMain () {filename: = os. Args[1] Err: = ReadFileByLine002 (filename)ifErr! =Nil{FMT. PRINTLN (ERR)}}

After that, I see the format of the file is this:

10.88.111.101 - - [18/Jul/2015:00:00:02 +0800] "GET /topics/2014-11-18/2269791.html HTTP/1.1" 200 1557110.88.111.101 - - [18/Jul/2015:00:00:02 +0800] "GET /topics/2014-12-26/2419017.html HTTP/1.1" 200 1517610.88.111.101 - - [18/Jul/2015:00:00:03 +0800] "GET /topics/2012-10-18/3535945.html HTTP/1.1" 200 1525010.88.111.101 - - [18/Jul/2015:00:00:03 +0800] "GET /topics/2014-11-10/2245185.html HTTP/1.1" 200 15250

Part 04 Filter Bar

Know the file format, the next step is to filter out miscellaneous logs, mainly the. css,.js,.jpg,.png,.gif. The code for this section is as follows:

 PackageMainimport ("Bufio"    "FMT"    "IO"    "OS"    "RegExp") const (Regruler = ' (\.js HTTP) | ( \.jpg HTTP) | (\.png HTTP) | (\.css HTTP) | (\.gif HTTP) ' Func Process (FILESRC, Fielestorestring)Error{var fsrc, Fdes *OS. File var errErrorFSRC, err =OS. OpenFile (FILESRC,OS. O_rdonly,0660)ifErr! =Nil{returnERR} defer fsrc. Close ()ifFdes, err =OS. OpenFile (Fielestore,OS. o_wronly|OS. o_create|OS. O_trunc,0666); Err! =Nil{returnERR} defer fdes. Close () SC: = Bufio. Newscanner (FSRC) Reg: = RegExp. Mustcompile (Regruler) forSc. Scan () {line: = sc. Text () Match: = Reg. FindString (line)ifMatch = =""{if_, Err =io. WriteString (Fdes, line+"\ n"); Err! =Nil{returnErr}}}returnSc. ERR ()return Nil}func Main () {filename: =OS. args[1] Filestorename: ="No-css-jpg-png-gif-js_log"    ifERR: = Process (filename, filestorename); Err! =Nil{FMT. PRINTLN (ERR)return} FMT. Println ("OK")}

To my surprise, the program's running time is super long, long I saw a set of flowers thousand bones, it is not over.
In addition, this step is under Linux, is also a knock command on the completion of the matter. Linux is so powerful that it has no edge.

Part 05 split, Time is standard

In this case, the file is still too large and disorganized, so ah, the time to split the file. Well, you said split it, I'm fine.
The code is as follows:

 PackageMainimport ("Bufio"    "FMT"    "IO"    "OS"//"Path"    "RegExp") const (Regruler = ' \[(? p<day>\d{2})/(? p<month>[a-za-z]{3})/(? p<year>\d{4}): \d{2}:\d{2}:\d{2}\s\+\d{4}\] ') func Process (filesrcstring)Error{var fsrc, Fdes *OS. File var errErrorPremonth: =""Preday: =""Preyear: =""Curmonth: =""Curday: =""Curyear: =""Filedes: =""FSRC, err =OS. OpenFile (FILESRC,OS. O_rdonly,0660)ifErr! =Nil{returnERR} defer fsrc. Close () SC: = Bufio. Newscanner (FSRC) Reg: = RegExp. Mustcompile (Regruler) forSc. Scan () {line: = sc. Text ()//FMT. Println (line) Match: = Reg. Findstringsubmatch (line)//FMT. PRINTLN (Match)//ifMatch = =Nil{        //ifCurday = =""&& Curmonth = =""&& Curyear = =""{        //ifFdes, err =OS. OpenFile ("Unkonow.txt",OS. o_wronly|OS. o_create|OS. O_append,0666); Err! =Nil{        //returnERR//}//}//FMT. Println ("Here")        // }Else{ifMatch! =Nil{ forI, Name: = Range Reg. Subexpnames () {//ignore the whole regexp match andUnnamed groupsifi = =0|| Name = =""{Continue} switch name {case"Month": Curmonth = match[i] Case"Day": Curday = match[i] Case"Year": Curyear = Match[i]}}//FMT. Println (Filedes)ifCurday! = Preday | | Curmonth! = Premonth | | Curyear! = preyear {fdes. Close () Filedes = Curyear +"-"+ Curmonth +"-"+ CurdayifFdes, err =OS. OpenFile (Filedes,OS. o_wronly|OS. o_create|OS. O_append,0666); Err! =Nil{returnErr}}} _, Err =io. WriteString (Fdes, line+"\ n")ifErr! =Nil{//FMT. Println ("Here") Fdes. Close ()returnERR} preyear = curyear Preday = Curday Premonth = curmonth//Match: =//fmt. fprintf (OS. Stdout,"%s\n", scanner. Text ())} fdes. Close ()returnSc. ERR ()return Nil}func Main () {filename: =OS. args[1]//FMT. PRINTLN (filename)//ERR: = ReadFileByLine001 (filename) Err: = Process (filename)ifErr! =Nil{FMT. PRINTLN (Err)} FMT. Println ("OK")}

Part 06 is over.

Here it is, basically. It's time to finish. Say, feeling.

• Before the code less a few lines, and then, my little laptop can not bear, motherboard overheating, automatic shutdown, really scared the father.
• The process of programming is one side of Baidu, while code, but also a small harvest.
• Learning a language has to be practiced and learned in practice.