Packetbeat Protocol Extension Development tutorial one

Original link: http://elasticsearch.cn/article/48

Packetbeat (Https://www.elastic.co/products/beats/packetbeat)
is an open-source network capture and analysis framework, built-in a lot of common protocol parsing, such as HTPP, MySQL, thrift and so on. But there are a lot of network protocols, how to extend an own protocol, this article will show you how to extend the implementation of your own protocol on the basis of packetbeat.

Development environment:
1.Go language
Packetbeat is written in the go language, with high performance and ease of deployment, for more information on the Go language, please visit: https://golang.org/.
2.Git
Source management, I believe we are more familiar with.
3.Tcpdump
*nix packet analysis, optional, for debugging.
4.Mac this one set
Windows is too hurtful, not recommended.
5.IDE
Recommend idea, others as long as you can.

This tutorial is to introduce you to write an extension of the SMTP protocol, SMTP is the protocol we use to send mail, encryption is more troublesome, for convenience, this tutorial uses unencrypted name transmission of the SMTP protocol, the default corresponding port is 25.

A. Source check-out
Log on to GitHub to open https://github.com/elastic/beats

Fork to get your own warehouse, like mine: https://github.com/medcl/packetbeat

#Create corresponding directory
mkdir -p $GOPATH/src/github.com/elastic/
cd $GOPATH/src/github.com/elastic
#Check out source code
git clone https://github.com/elastic/beats.git
CD beats
#Modify the official warehouse as the upstream source and set your own warehouse as the origin source
git remote rename origin upstream
git remote add origin [email protected]:medcl/packetbeat.git
#Get the latest upstream code. If it's just fork, don't worry
git pull upstream master
#Check out a branch called smtpbeat to develop this feature
git checkout -b smtpbeat
#Switch to the packetbeat module
cd packetbeat
#Get dependency information
(mkdir -p $GOPATH/src/golang.org/x/&&cd $GOPATH/src/golang.org/x &&git clone https://github.com/golang/tools.git )
go get github.com/tools/godep
Compile and compile
Make

Compiled files: Packetbeat is in the root directory
Now let's test it.
Modify ETC/PACKETBEAT.YML, add enabled:true under Elasticsearch below output, default is not enabled, and if your Elasticsearch has shield installed, For example, my elasticsearch user name and password are Tribe_user, oh, forgot to say, our elasticsearch run in this machine.
The detailed configuration of the packetbeat.yml can be found in: https://www.elastic.co/guide/en/beats/packetbeat/current/packetbeat-configuration.html

output:
  elasticsearch:
    enabled: true
    hosts: ["localhost:9200"]
    username: "tribe_user"
    password: "tribe_user"

You can now run the command to start Packetbeat, which will listen to all the built-in protocols, such as HTTP, DNS, etc. by default.

./packetbeat -e -c etc/packetbeat.yml  -d "publish"

  

	
   


  

Introduce the commonly used parameters:
-N Dry run mode, no actual output storage log
-e console Output debug log
-D displays only the logs for the corresponding logger

OK, we open a few pages, the console will have the corresponding output, as follows:

2015/12/29 14:24:39.965037 preprocess.go:37: DBG  Start Preprocessing
2015/12/29 14:24:39.965366 publish.go:98: DBG  Publish: {
  "@timestamp": "2015-12-29T14:24:39.709Z",
  "beat": {
    "hostname": "medcls-MacBook.local",
    "name": "medcls-MacBook.local"
  },
  "bytes_in": 31,
  "bytes_out": 115,
  "client_ip": "192.168.3.10",
  "client_port": 53669,
  "client_proc": "",
  "client_server": "",
  "count": 1,
  "direction": "out",
  "dns": {
    "additionals_count": 0,
    "answers": [
      {
        "class": "IN",
        "data": "www.a.shifen.com",
        "name": "sp2.baidu.com",
        "ttl": 333,
        "type": "CNAME"
      }
    ],
    "answers_count": 1,
    "authorities": [
      {
        "class": "IN",
        "data": "ns1.a.shifen.com",
        "expire": 86400,
        "minimum": 3600,
        "name": "a.shifen.com",
        "refresh": 5,
        "retry": 5,
        "rname": "baidu_dns_master.baidu.com",
        "serial": 1512240003,
        "ttl": 12,
        "type": "SOA"
      }
    ],
    "authorities_count": 1,
    "flags": {
      "authoritative": false,
      "recursion_allowed": true,
      "recursion_desired": true,
      "truncated_response": false
    },
    "id": 7435,
    "op_code": "QUERY",
    "question": {
      "class": "IN",
      "name": "sp2.baidu.com",
      "type": "AAAA"
    },
    "response_code": "NOERROR"
  },
  "ip": "192.168.3.1",
  "method": "QUERY",
  "port": 53,
  "proc": "",
  "query": "class IN, type AAAA, sp2.baidu.com",
  "resource": "sp2.baidu.com",
  "responsetime": 18,
  "server": "",
  "status": "OK",
  "transport": "udp",
  "type": "dns"
}
2015/12/29 14:24:39.965774 preprocess.go:94: DBG  Forward preprocessed events
2015/12/29 14:24:39.965796 async.go:42: DBG  async forward to outputers (1)
2015/12/29 14:24:40.099973 output.go:103: DBG  output worker: publish 2 events

Then Elasticsearch should have data in it, let's see:

curl http://localhost:9200/_cat/indices\?pretty\=true -u tribe_user:tribe_user
yellow open packetbeat-2015.12.29  5 1   135  0 561.2kb 561.2kb

At this point, the packetbeat source build success, our entire development process has run through, the next section formally began to introduce the SMTP protocol extension.

Packetbeat Protocol Extension Development tutorial one