ELK Packetbeat Deployment Guide (15th)

Original link: http://www.ttlsa.com/elk/elk-packetbeat-deployment-guide/

Packetbeat is a real-time network packet analysis tool that integrates with Elasticsearch to provide monitoring and analysis systems for applications.

Packetbeat decodes application-layer protocol types such as HTTP, MySQL, Redis, and so on, by sniffing through network traffic between application servers, correlating requests and responses, and documenting the fields that make sense for each transaction.

Packetbeat can help us quickly discover problems with back-end applications such as bugs or performance issues, and it's quick to fix troubleshooting.

The protocols currently supported by Packetbeat are:

• HTTP
• Mysql
• PostgreSQL
• Redis
• Thrift-rpc
• Mongodb
• Dns
• Memcache

Packetbeat can insert related transactions directly into Elasticsearch or Redis (not recommended) or Logstash.

Packetbeat can be run on an application server or on a separate server. When running on a single server, you need to get network traffic from the Mirror port on the switch or from the eavesdropping device.

After decoding the seventh layer of information, Packetbeat associates the response associated with the request, which is called a transaction. For each transaction, Packetbeat inserts a JSON-formatted document into the Elasticsearch. The analysis can then be presented through the Kibana.

Installation

Configure the Beats Yum source first, as described earlier.

1	# yum Install Packetbeat

Configuration

Choose which NIC you want to sniff network traffic from, by default, all network interfaces.

1234	interfaces: # Select on which network interfaces to sniff. You can use the ' any ' # keyword to sniff in all connected interfaces. device: any

In the Protocol section, configure the port so that Packetbeat finds the protocol corresponding to each port. If you use a non-standard port, you need to add it. Multiple ports are separated by commas.

123456789101112131415161718192021

protocols: # Configure which protocols to monitor and on which ports is they # running. You can disable a given protocol by commenting off its # configuration. http: ports: [8080, 8081, 8002< c18>] memcache: ports: [11211] mysql: ports: [3306] redis: ports: [6379] pgsql: ports: [5432] Thrift: ports: [9090]

Defining Elasticsearch Services

123456789101112131415

output: elasticsearch: # Uncomment out the This option if you want to the output to Elasticsearch. the # Default is False. enabled: true # Set the host and port where to find Elasticsearch. host: 192.168.1.42 Port: 9200 # Uncomment this option and set it to true if you want to store the topology in # Elasticsearch. Default behavior if this setting are left out of the # Config file is equivalent-setting "Save_topology" to "false" #save_topology: false

Load Elasticsearch Index Template

Load the index template so that Elasticsearch knows which fields are analyzed in what way.

1	# curl-xput ' http://10.1.19.18:9200/_template/packetbeat ' [email protected]/etc/packetbeat/ Packetbeat.template.json

Start the service

1	#/etc/init.d/packetbeat Start

View data

Load the Kibana packetbeat Dashboard

This in the previous article, has been loaded. Loading is no longer repeated here.

Configuration options

Beats common Configuration Select the previous article has said. Let's talk about Packetbeat's own configuration: Interfaces, Protocols, Processes (optional).

Interfaces

Interfaces partial configuration Sniffer

1234567891011121314

# Select The network interfaces to sniff the data. can use the ' any '# keyword to sniff on all connected interfaces.interfaces: # on which device to sniff device: any # The maximum capture size of a single packet. snaplen: 1514 # The type of the sniffer to use type: af_packet # The size of the sniffing buffer buffer_size_mb:

Device

The network interface from which to capture traffic. The specified device is automatically set to promiscuous mode, which means that packetbeat can capture traffic from other hosts from the same LAN.

12	interfaces: device: eth0

On Linux, you can specify any device. When specified as any, the interface is not set to promiscuous mode.

To view the available devices, you can use the following command:

123456

# packetbeat-devices0: eth0 (No description available) 1: eth1 (No description available) 2: usbmon1 (USB bus number 1) 3: Any (Pseudo-device , captures on all Interfaces) 4: lo (No description available)

Device can be specified as the index of the returned list above, as

12	interfaces: device: 0

The expression is eth0. This is very useful in cases where the device name is very long.

Snaplen

The maximum size of the capture package. Default 65535. Sufficient to handle all network and interface types. If you sniff the physical network interface, the value is set to the MTU size. For virtual interfaces, it is best to use the default values.

123	interfaces: device: eth0 snaplen: 1514

Type

Packetbeat supports the following sniffer types:

• pcap, using the Libpcap library, can work on most platforms, but is not the quickest option.
• af_packet, using memory-mapped sniffing. Faster than Libpcap and does not require kernel modules, Linux-specific.
• pf_ring, use the Ntop.org project. This setting provides the best sniffer speed, but requires a kernel module, Linux-specific.

The default sniffer type ispcap。

123	interfaces: device: eth0 type: af_packet

On Linux, the option is recommended if you want to optimize Packetbeat CPU usage af_packet 和 pf_ring .

If used af_packet , you can adjust the behavior with the following options:

Buffer_size_mb

The maximum shared memory buffer size used between the kernel and the user space. Default 30MB. The larger the buffer, the lower the CPU utilization, but consumes more memory. Only af_packet valid for.

1234	interfaces: device: eth0 type: af_packet buffer_size_mb:

With_vlans

Packetbeat automatically generates a BPF to capture port traffic for known protocols. For example, to configure HTTP 80 and MySQL 3306, Packetbeat generates the BPF filter as follows:"port 80 or port 3306"。

However, if the traffic contains VLAN tags, the filter generated by Packetbeat will be invalid because offset is moved through four bytes. To solve this problem, enable the with_vlans option to generate a BPF filter like this:"port 80 or port 3306 or (vlan and (port 80 or port 3306))"。

Bpf_filter

Packetbeat automatically generates a BPF to capture port traffic for known protocols. For example, to configure HTTP 80 and MySQL 3306, Packetbeat generates the BPF filter as follows:"port 80 or port 3306"。

可以使用 bpf_filterOverrides the generated BPF filter, such as:

123	interfaces: device: eth0 bpf_filter: "net 192.168.238.0/0 and Port 3306"

This setting disables the automatic generation of BPF filters. If you use this setting, you need to keep the BPF filter synchronized with the ports defined in the Protocol section.

Protocols and processes configuration items, and the following.

ELK Packetbeat Deployment Guide (15th)