Recently, Linux redhat 6.5 app business system, send out a large amount of traffic, and constantly establish a TCP connection, the target address is an IP in the United States, estimated to be treated as a broiler, more tragic, directly to IDC room, firewall shows this APP server TCP connection A lot of 10W per second ，
Using commands from the server Sar-n DEV 2 10, there are a lot of problems with the contract, (below is normal, abnormal situation eth0txpck/s 10000 or so)
First speed limit or unplug the network cable:
Before you begin, clear the Eth0 all queue rules first
TC Qdisc del dev eth0 root 2>/dev/null >/dev/null
1) define the topmost (root) queue rule and specify the default category number
TC Qdisc Add dev eth0 root handle 1:htb default 20
TC class Add dev eth0 parent 1:classid 1:20 HTB rate 2000kbit
(1kb/s = 8kbit/s)
TC Command Format:
TC Qdisc [Add | change | replace | link] Dev dev [parent Qdisc-id | root] [handle Qdisc-id] Qdisc [Qdisc specific P Arameters]
TC class [Add | change | replace] Dev dev parent qdisc-id [classid Class-id] qdisc [qdisc specific parameters]
TC Filter [Add | change | replace] Dev dev [parent Qdisc-id | root] Protocol protocol Prio Priority FilterType [Filte Rtype specific parameters] Flowid Flow-id
Show
TC [-S |-d] qdisc show [Dev Dev]
TC [-S |-d] class show Dev dev tc filter show Dev Dev
View the status of TC
Tc-s-D qdisc Show Dev eth0
Tc-s-D class show Dev eth0
Remove TC Rule
TC Qdisc del Dev eth0 root
View Status:
Top
The CPU and mem are normal and do not see the abnormal process.
Yum Install-y tcpdump
Tcpdump-nn
Find a large number of IP addresses
Data between the native (192.168.35.145) and host 114.114.110.110
Tcpdump-n-I eth0 host 192.168.35.145 and 114.114.110.110
And the interception of all data into the server can be used in the following format
Tcpdump-n-I eth0 DST 192.168.35.145
Or the server has multiple IPs that can use parameters
Tcpdump-n-I eth0 DST 192.168.35.145 or 192.168.35.155
We crawl all incoming TCP packets into the server using the following format, you can refer to the next
Tcpdump-n-I eth0 DST 192.168.35.145 or 192.168.35.155 and TCP
Packets Out of this machine
Tcpdump-n-i eth0 src 192.168.35.145 or 192.168.35.155
Tcpdump-n-i eth0 src 192.168.35.145 or 192.168.35.155 and port! and TCP
Or, you can filter out better results by using a condition that can be either an OR and a combination.
The exception IP can be added to/etc/hosts.deny, or under Firewall settings
Then install a nethogs
Install Howto:
Download the latest epel-release rpm from (one 32-bit, one 64-bit connection)
http://dl.fedoraproject.org/pub/epel/6/i386/
http://dl.fedoraproject.org/pub/epel/6/x86_64/
Install epel-release RPM:
# RPM-UVH epel-release*rpm (remember yum clean all and Yum Makecache)
Install nethogs RPM Package:
# yum Install Nethogs
Executive Nethogs
Nethogs
Or
Nethogs eth0
If the NIC is bound to a nethogs bond0
-------------------------
RPM-UVH epel-release-6-8.noarch64.rpm
Yum Clean All
Yum Makecache
Yum Install Nethogs
Nethogs
--------------------------
Shows the current network usage of each process:
Press "M" key to switch to the statistics view to show the total network usage of each process
Press "CTRL + C" or "Q" to exit the monitor
Use Help:
[Email protected] ~]# nethogs--help
Nethogs:invalid option--'-'
Usage:nethogs [-v] [-b] [-D seconds] [-t] [-p] [device [device [device]]]
-V: Displays the version information, note the capital letter V.
-D: Delayed update refresh rate, in seconds. The default value is 1.
-T: Trace mode.
-b:bug Hunting Mode-means the tracking mode.
-P: Mixed mode (not recommended).
Device: The name of the device to monitor. Default is Eth0
When Nethogs is running, press:
Q: Exit
M: Toggle between Total and current usage mode
Find the process of a large number of packages, then kill, in the process of troubleshooting what the program, where the file path, delete the exception file.
At present, the problem solved, observe a period of time to see if the problem will also occur, the firewall will be the public IP access restrictions, only allow office location access, increase security.
Troubleshooting a large number of outsourcing issues to Linux servers
