Server Security-Ddos attack and defense

[Directory]
Background
2. Emergency response
3. Common ddos attacks and defense
Four root causes and counterattacks
Summary

Background

A few days ago, a website we operated suffered a ddos attack. Our website is a public welfare website that builds a platform between various vendors and white hats to transmit security issues and other information, we do not know why such a shameless attack will occur. Because we are not engaged in this type of attack, this attack technology is generally rough, so we have less discussion, however, since such an attack happened, we feel that we have learned something in this process after the attack and our thoughts on this attack can make this attack truly valuable, this is not just a waste of time.
In addition, we found that large enterprises all suffered from attacks, but we did not share much of the measures and lessons learned after the attacks, as a result, various companies are exploring their own experiences and are still stuck in the situation of an enterprise defending against attacks on the entire Internet. However, this attack is targeted at attackers and the next attack is targeted at attackers, in addition, there is no loss of technology or resources after the attack, which is also the cause of frequent and unscrupulous attacks.
Let's try to make some changes :)

2. Emergency response

After the attack, the first phenomenon was that our website could not be accessed, but we could still access the management interface. We logged on and simply executed the command:

Netstat-antp

We can see that a large number of links exist and are all in the ESTABLISHED state. Under normal conditions, our website traffic is not so high. If it is so high, we believe there is hope for information security in China, in fact, it is relatively simple to deal with such a situation. This is a layer-4 attack, that is, all ip addresses are real, because so far only the network connection resources of webserver are consumed, therefore, we only need to block these ip addresses at the network layer. It is very easy to use the following command:

For I in 'netstat-an | grep-I ': 80' | grep 'est' | awk '{print $5}' | cut-d: -f 1 | sort | uniq-c | awk '{if ($1> 50) {print $2 }}''
Echo $ I
Echo $ I>/tmp/banip
/Sbin/iptables-a input-p tcp-j DROP-s $ I
Done

Then, the task can be executed once a minute. Soon, the ban list of iptables is filled with a large number of blocked ip addresses. We simply counted some of the ip addresses with the largest number of connections from South Korea. To ensure the system performance, we increase the number of acceptable connections and the request rate for each connection to Nginx, and the system resumes normal operation.
The normal status continued until the next day, but after noon we found that there was another problem with access and the network was slow. ping showed that there were about 70% packet loss, after hard login to the system, we found that the system had very few normal TCP connections. To find out the cause, we captured the system:

Tcpdump-w tmp. pcap port not 22
Tcpdump-r tmp. pcap-nnA

We found that the attack has been changed from the application layer to the network layer. A large number of target ports are 80 udp and icmp packets, which are full of the network at an extremely fast speed, the size of a package is about 1 kb. The resources occupied this time are purely bandwidth resources. This problem cannot be solved even if the system limits it, but it does not matter, we can restrict network layer issues on the network layer. We only need to disable all non-TCP packets, such as UDP and ICMP, that reach our ip address on the network, however, we do not have our own servers and lack control over network devices. At present, it is supported by CERT of the Ministry of Industry and Information Technology. As you can see, we cannot coordinate operations temporarily, our service is very slow. Basically, we stopped the service. After a period of time, the attacker stopped the attack and the service was restored. Is it really wrong? However, at the same time, we have received help from many enthusiastic friends and obtained better network and server resources. Our capabilities in network resources have been greatly improved, alleviating this problem, I would like to express my gratitude to them.

3. Common ddos attacks and defense

Continue to stick to the 80 sec "Know it then hack it". Here we will briefly discuss the ddos attack and defense issues. Ddos is short for distributed denial-of-service attacks. Since a denial-of-service must be stopped for some reason, the most important and common reason is the limitation of server resources, this type of server has a wide range of resources. You can simply sort out the process of normal request completion:

1. Enter the requested address in the client browser.
2. The browser parses the request, including analyzing the dns to identify the remote server address to be reached.
3. After the IP address is specified, the browser tries to establish a connection with the server service, and the data packet that tries to establish the connection passes through the local network. The intermediate route finally reaches the target network and then reaches the target server.
4. After the network connection is established, the browser establishes different data packets based on the request and sends the data packets to a port on the server.
5. The port is mapped to the process. After the process receives the data packet, it performs internal parsing.
6. Request various resources within the server, including backend APIs and some databases or files.
7. After the logic processing is complete, the data packet is returned to the user's browser according to the channel established previously. The browser completes resolution and the request is complete.

The above points can be used for ddos attacks, including:

1. Do you still remember to access Baidu Sogou due to some famous client hijacking viruses? :)
2. A large Internet company has a dns hijacking event or a large number of dns requests directly attack the dns server. Here, some professional third-party dns services can be used to alleviate this problem, such as Dnspod.
3. Use the network resources required to establish a network connection to attack the server bandwidth so that normal packets cannot reach flood attacks such as udp, the cpu resources of front-end devices are consumed so that data packets cannot be effectively forwarded, such as icmp flood attacks and some fragmented packets, the resource required by the server to establish a normal connection, such as syn flood, or a large number of connections, cannot be initiated. For example, this TCP flood
4. Use some of the characteristics of webserver to launch attacks. Compared with nginx, apache processes a request more heavily.
5. Use some internal characteristics of the application to attack internal resources of the program, such as mysql and interfaces that consume large resources on the backend. This is a traditional CC attack.

This involves the concept of attack and defense, but if you understand the attack points and methods of the other party, the defense will become a simple process of resource competition, do not use the weakest part to defend against the most powerful parts of the family. You should start from the most appropriate part to solve the problem. For example, solving application-layer attacks on routers and other devices is not a good solution, likewise, it is impossible to try to solve network-layer problems at the application layer. Simply put, the goal is to allow normal data and requests to access our services, A complete defense system should consider the following aspects:

1. As the entry to user requests, good dns defense is required.
2. Bandwidth resources that match your value, and the application layer defense policy is configured on the core node to only allow network packets of your normal applications to enter, for example, block all data packets except 80
3. Machine clusters that support the value of your services can resist the pressure on the application layer. If necessary, you need to continue decomposing an http request and decomposing the connection creation process pressure into other clusters, there seems to be a general hardware firewall that can do this, and even break down normal http request parsing processes to ensure that normal requests reach the backend and malformed requests are eliminated, record and monitor the request frequency and other behaviors of normal requests. If an exception occurs, block the application layer here.

Each company has its own value assessment to determine the size of Security Investment. Each attack also involves the existence of benefits, just as defense for various reasons, such as lack of investment and imperfection in the implementation process, has inherent vulnerabilities, attacks also have inherent vulnerabilities, because each attack involves different links, each stage may be completed by a person of different levels. The resources, tools, and technologies he uses are not perfect. Therefore, defense is possible, I believe that the people who conduct DDOS attacks are a fixed industry and there will be some fixed groups. The technologies, tools, resources, and interest chains used in these attacks are relatively fixed, in contrast, each enterprise lacks communication. It is naturally difficult for an individual enterprise to confront an industry. If every enterprise can share its own experience in attacks, including the size and IP address distribution of botnets, the characteristics of attack tools, and even the ability to analyze the benefits and operators behind them, each attack will increase the overall defense capability of everyone, we are willing to do this to reduce the attacker's attack capability.

Four root causes and counterattacks

What I am confused about is that we cannot get the actual benefit from attacks. Why are there still attacks? I have heard that other companies are under attack, I think one of the reasons is that we do not get any benefit from an attack, but in fact the attacker does not lose anything. Whether it is a resource or a legal risk, he will not lose too much due to an attack, in contrast, service providers suffer too many losses, which is an imbalance in economics and we are in a weak position.
In general, we do not have any punitive measures for the attackers, but this time we think we can do something. We try to find out the attackers behind the scenes and even clear the botnet.
First, the attack originated from the application layer, so all ip addresses are real. After communication with CERT, we also found that these ip addresses are all from South Korea and the control end is not in China, because there was no communication with China during this period, even if it was changed to udp + icmp flood later, it was still the ip addresses of South Korea, which is very interesting, under normal circumstances, udp + icmp data packets can be forged, but not here, which is probably confirmed later.
These ip addresses are real ip addresses, and these ip addresses must remain in contact with attackers after the attack is completed. The general contact information is that dns domain names are needed for convenient control, in this case, if we can find this dns domain name, we may indirectly find out where the real hacker is. First, we quickly found out the machines with Port 80 opened in the attack ip address, because we are confident about the security problem on port 80, we should soon be able to learn the details behind these ip addresses (80sec name). We found that most of them are routers and some web vpn devices. We guess this attack is mainly for Korean individual users, however, individual users' machine operating systems are generally windows, so there may be large restrictions on sending data packets in higher versions, this explains why even udp + icmp attacks are mostly real ip addresses. After discovering these routing devices, we tried to go deeper and quickly used some weak passwords, such as admin/admin, to log in. As a result, all Internet users in the world are the same. admin/admin is a natural Portal.
After logging in to some routes, we found that there is a function in these routers to set their own dns, which means that all the dns requests below can be directed to our own dns server, this is useful for us to understand the details of the internal network, so we established a dns server and enabled the dns request logging function to record the details of all requests. We controlled the dns pointing of about 20 routers and successfully redirected them to our own servers.
The rest is simple data analysis. Before that, we can make the following guesses about the domain name controlled by the botnet:

1. This dns should be used to flexibly control the domain name cache Time TTL, which is generally not very long.
2 This dns should be requested on a regular basis, so there will be a large proportion of dns requests
3 This dns should exist for control, so the domain name should not obtain a high Access Index in the search engine and other places, which will be better determined in combination with the rules in 2, it is a natural contradiction.
4. This dns should be requested under each route

Through simple statistics, we can easily get the answer. We found some 3322 general malware domain names, but found that they are not what we need, because only a few machines can access them, after some time, we finally found that the access volume of a domain name is the same as that of naver (a portal in South Korea), workgroup001.snow ****. net, it seems that it is well managed for its own botnet. About 18 machines have accessed this domain name, and the host of this domain name is hosted in Singapore, TTL is 1800, that is, half an hour. This domain name does not exist in all search engines. It was registered by a Korean one year ago in godady, at the same time, when we access this domain name and direct it to 3389 of the host, a typical windows backdoor exists on it after shift 5. It seems that we have found it, isn't it? After subsequent observations, this domain name points to 127.0.0.1 after a period of time, and we are convinced of our answer. workgroup001.snow *****. net seems to be well managed by our Botnet :)
This is a typical ddos attack. After the attack, we obtained the list of hosts involved in the attack and the domain name and ip address of the control end. We believe that cert from China and South Korea are very interested in clearing the attack source, we have some losses, but attackers have also suffered (probably including a botnet and a control domain name, or even an internal legal investigation ), we are no longer unequal, aren't we?
Laiyuan:

EMail: jianxin # 80sec.com
Site: http://www.80sec.com
Date: 2011-2-10