Troubleshooting a large number of outsourcing issues to Linux servers

Source: Internet
Author: User

Recently, Linux redhat 6.5 app business system, send out a large amount of traffic, and constantly establish a TCP connection, the target address is an IP in the United States, estimated to be treated as a broiler, more tragic, directly to IDC room, firewall shows this APP server TCP connection A lot of 10W per second ,

Using commands from the server Sar-n DEV 2 10, there are a lot of problems with the contract, (below is normal, abnormal situation eth0txpck/s 10000 or so)


First speed limit or unplug the network cable:
Before you begin, clear the Eth0 all queue rules first
TC Qdisc del dev eth0 root 2>/dev/null >/dev/null

1) define the topmost (root) queue rule and specify the default category number
TC Qdisc Add dev eth0 root handle 1:htb default 20
TC class Add dev eth0 parent 1:classid 1:20 HTB rate 2000kbit

(1kb/s = 8kbit/s)

TC Command Format:
TC Qdisc [Add | change | replace | link] Dev dev [parent Qdisc-id | root] [handle Qdisc-id] Qdisc [Qdisc specific P Arameters]
TC class [Add | change | replace] Dev dev parent qdisc-id [classid Class-id] qdisc [qdisc specific parameters]
TC Filter [Add | change | replace] Dev dev [parent Qdisc-id | root] Protocol protocol Prio Priority FilterType [Filte Rtype specific parameters] Flowid Flow-id

Show
TC [-S |-d] qdisc show [Dev Dev]
TC [-S |-d] class show Dev dev tc filter show Dev Dev

View the status of TC
Tc-s-D qdisc Show Dev eth0
Tc-s-D class show Dev eth0

Remove TC Rule

TC Qdisc del Dev eth0 root

View Status:
Top
The CPU and mem are normal and do not see the abnormal process.

Yum Install-y tcpdump
Tcpdump-nn
Find a large number of IP addresses

Data between the native (192.168.35.145) and host 114.114.110.110
Tcpdump-n-I eth0 host 192.168.35.145 and 114.114.110.110
And the interception of all data into the server can be used in the following format
Tcpdump-n-I eth0 DST 192.168.35.145

Or the server has multiple IPs that can use parameters
Tcpdump-n-I eth0 DST 192.168.35.145 or 192.168.35.155

We crawl all incoming TCP packets into the server using the following format, you can refer to the next
Tcpdump-n-I eth0 DST 192.168.35.145 or 192.168.35.155 and TCP

Packets Out of this machine
Tcpdump-n-i eth0 src 192.168.35.145 or 192.168.35.155
Tcpdump-n-i eth0 src 192.168.35.145 or 192.168.35.155 and port! and TCP
Or, you can filter out better results by using a condition that can be either an OR and a combination.

The exception IP can be added to/etc/hosts.deny, or under Firewall settings

Then install a nethogs
Install Howto:
Download the latest epel-release rpm from (one 32-bit, one 64-bit connection)
http://dl.fedoraproject.org/pub/epel/6/i386/
http://dl.fedoraproject.org/pub/epel/6/x86_64/

Install epel-release RPM:
# RPM-UVH epel-release*rpm (remember yum clean all and Yum Makecache)

Install nethogs RPM Package:
# yum Install Nethogs

Executive Nethogs
Nethogs
Or
Nethogs eth0
If the NIC is bound to a nethogs bond0

-------------------------
RPM-UVH epel-release-6-8.noarch64.rpm
Yum Clean All
Yum Makecache
Yum Install Nethogs
Nethogs
--------------------------



Shows the current network usage of each process:


Press "M" key to switch to the statistics view to show the total network usage of each process


Press "CTRL + C" or "Q" to exit the monitor

Use Help:
[Email protected] ~]# nethogs--help
Nethogs:invalid option--'-'
Usage:nethogs [-v] [-b] [-D seconds] [-t] [-p] [device [device [device]]]
-V: Displays the version information, note the capital letter V.
-D: Delayed update refresh rate, in seconds. The default value is 1.
-T: Trace mode.
-b:bug Hunting Mode-means the tracking mode.
-P: Mixed mode (not recommended).
Device: The name of the device to monitor. Default is Eth0

When Nethogs is running, press:
Q: Exit
M: Toggle between Total and current usage mode

Find the process of a large number of packages, then kill, in the process of troubleshooting what the program, where the file path, delete the exception file.

At present, the problem solved, observe a period of time to see if the problem will also occur, the firewall will be the public IP access restrictions, only allow office location access, increase security.

Troubleshooting a large number of outsourcing issues to Linux servers

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.