Self-built CDN to defend against DDoS attacks (1): Build a persistent defense line

This topic is the content we shared in the OWASP Hangzhou region security salon at the end of 2013. Here we resummarized the overall content of this topic and formed a text version.

In this article, the case and response experience of DDoS come from the actual scenarios of a customer service system with a high market share, we analyze the costs, efficiency, and specific architecture design (selection, configuration, and optimization) to cope with different types of DDoS attacks through self-built CDN.

Background

The main business of the customer service system is to provide real-time dynamic text chat Based on Web pages. It is mainly used in various online product sales, online customer service, and other fields, with a total of 0.58 million users, active online users at the same time: about 0.12 million/day.

These application fields are usually highly competitive among industries, including gray and profiteering industries that cannot be justified online, leading to frequent DDoS attacks among competitors. However, marketing websites are often accelerated on a single side. In addition, the promotion timeliness is very strong and it is difficult to be thoroughly cracked down. As a result, some smart hackers cannot communicate with visitors by attacking the website's online customer service system, transactions are not allowed to achieve the purpose of malicious attacks. Therefore, the customer service system, which originally contributed to website marketing, has become the main target of attacks. Although it has been wronged, it has to face challenges.

The types of DDoS attacks we encounter include: Slow CC attacks and fatal large-volume attacks. The following describes the attack features, defense ideas, and some of our defense solutions.

Slow CC attack features

Attackers use a large number of Proxy Server IP addresses on the network and attack software to generate legitimate requests directed to the affected host.

This type of attack is low for attackers, and there are a lot of ready-made software on the Internet. The attack style is relatively "gentle and cautious", with the aim of increasing the number of spam requests, it consumes the normal application overhead of the server, such as CPU, memory, Nic pressure, or even network congestion, and then requests are unresponsive and no outbound traffic. This causes the website to slow down and makes the website inaccessible.

Defense ideas

For such attacks, two vulnerabilities can be exploited to prevent such malicious CC attacks. The key is to respond quickly.

First, because a large number of illegal requests are generated manually, the incoming traffic caused by the network increases abnormally (normally, the incoming traffic is small and the outgoing traffic is large). Second, there is an increasing process of attack strength. We need to make full use of this precious time, so that the machine can respond intelligently at the first time, and call the log analysis script for decision-making, so as to defend against or divert traffic.

There are multiple methods. Here we only list the two methods we use:

1. Use the traffic monitoring diagram of the monitoring software to trigger the log analysis script (zabbix is used as an example ):
   

2. Use the bash script to count incoming traffic. When an exception is found, call the corresponding log analysis script to implement blocking.

   #! /Bin/bash DEV = $1 # define the listener network card LIMIT = $2 # define the trigger threshold value WARN = $3 # define the alarm threshold value TIME = $4 # define the network card data collection frequency mobile_num = "13 xxxxxxxxxx" # define the mobile phone number LOCK = "/tmp /. exchange_proxy.lock "[-z $ DEV] & echo" $0 ethx limit_band (kbps) warn_limit (kbps) seconds "& exit 0 [-z $ LIMIT] & LIMIT = 800000 #800 kbps [-z $ WARN] & WARN = 900000 #900 kbps [-z $ TIME] & TIME = 10 #10 s send_fetion () {# define the Feixin alarm SMS interface} while :; do net_flood = 'ifconfig $ DEV | sed-n "8" P' rx_before = 'echo $ net_flood | awk '{print $2}' | cut-c7-'sleep $ TIME net_flood = 'ifconfig $ DEV | sed-n "8" p 'rx_after = 'echo $ net_flood | awk '{print $2}' | cut-c7-'rx_result = $ [rx_after-rx_before) /$ TIME] over_bw = $ [(rx_result-LIMIT)] if [$ over_bw-gt 0]; then BOOL = 'echo "$ rx_result> $ WARN" | bc' # determine whether it is an attack if [$ BOOL-eq 1]; then # confirm it is an attack, execute the policy and send the SMS send_fetion $ mobile_num "$ STR" else # The traffic exceeds the threshold. Send the SMS. Pay attention to send_fetion $ mobile_num "$ STR" fi sleep $ TIME done.

The filter script is used to enable the log analysis mechanism on the server to identify abnormal IP addresses, agents, URLs, or other signatures at the first time. The kernel layer uses iptables to filter malicious IP addresses, the application layer uses the http keyword of nginx for filtering and directly returns badcode 444 for interception.

Disadvantages

Whether at the kernel level or application level, the CPU and memory of the server itself are highly dependent. For example, iptables filtering has a high CPU pressure on the server and blocks more than 15 k IP addresses, the server is basically unavailable. When Nginx blocks HTTP requests, it will allocate memory and processing chain rules for each http request, so the memory resources are exhausted; as the traffic increases and the attack time continues, the network adapter is under heavy pressure and the resources are eventually exhausted.

Therefore, this solution is temporary.

Critical high-traffic attack features

This type of attack is generally based on tcp syn, icmp, and UDP (especially UDP packets, a single UDP packet can be large. The maximum attack traffic suffered by the customer service system is 16 GB, and the entire data center is affected. Attackers usually control a large number of bots or directly collude with servers and Bandwidth Resources in the IDC to attack the target traffic. In this case, the traffic will quickly occupy the network bandwidth of the server, and thus cannot respond to any user requests.

This type of attack requires a large amount of bandwidth resources. For the attacker, the cost is quite high, but the attack is "quick and accurate" to make the website completely unresponsive in a short time.

Due to this type of attacks, the traffic monitoring devices in the IDC will be aware of this phenomenon. IDC usually takes measures to block or even directly strip the attacked IP address, causing the target to commit suicide. This is undoubtedly the case for customers who need help.

Defense ideas

Defense methods against such traffic attacks include:

• Set up a hard Firewall
• Rent anti-DDoS nodes
• Rent CDN to distribute target traffic

Disadvantages

• Set up a hard firewall: the price of 2G hard defense on the market is about 10 W, and the cluster defense cost is even higher. Although the hardware-level defense performance is high, the traffic flood is also a hit, and the side effects cannot be underestimated.
• Rent anti-DDoS nodes: Anti-DDoS nodes are divided into defense bandwidth, defense traffic, and sharing exclusive. The combined prices of each package vary greatly, and the traffic distribution policies vary, when the traffic exceeds the traffic promised by anti-DDoS pro, the Defense fails or the money is added, but both of them have performance loss and side effects.
• Rent CDN to distribute target traffic: All CDN providers on the market are charged based on traffic. For websites that are frequently attacked by traffic, they have to pay for the attack traffic, which is really unpleasant.

Both purchased hardware and anti-DDoS resources and CDN acceleration are expensive, and resource utilization is low during idle hours. During attack peaks, the costs are limited when there is a large volume of traffic organized, also accompanied by side effects (see the principles of the Green Alliance black hole firewall), is not a long-term plan.

Vulnerable party

To sum up, no matter which choice we make, it is very painful.

We have been talking with the attacker for nearly a year. We have learned that this is a very complete industrial chain (upstream personnel have long lived abroad, and remote control commands and operations cannot be investigated at all ), they control a large amount of attack resources, and the attack resources themselves come from IDCs. In order to make quick profits, attackers also like and recommend this direct method to attack the target. When launching an attack, they can mobilize the bandwidth resources of multiple IDCs to combat the target (this phenomenon also reflects the nonstandard IDC Management in China ).

From this point of view, the attacked party is always in a weak position. With a weak architecture and extremely limited resources, it cannot resist powerful cluster resource attacks.

We have been thinking about the question: if we continue to invest these funds, what can we leave after the crisis or a few years? Therefore, we jumped out of the single-node defense and the idea of renting CDN, and combined with the advantages of the above solution, we switched to the self-built CDN solution.

Long-term Plan: self-built CDN

The advantages of self-built CDN are as follows:

• Bypass traffic cleaning (acne grows on others' faces)
• Full use of resources: Route acceleration is enabled when no attack is available, and node switching is performed when there is an attack (one thing is used for multiple purposes)
• With the increase in investment, the ability to defend against DDoS attacks is enhanced (long-term planning, high return on capital)

We will introduce how to build self-built CDN and how much it costs in the next article in the series.

Author Profile

Hu Haiyang (personal page), from the Hangzhou Linux user group. The website is named "Heart of the ocean", a System Architect and amateur contributor. It is devoted to the research and exploration of open source software and cutting-edge technologies.

Zhang Lei (Weibo, blog), from the Google developer community in Hangzhou. Focusing on the information security technology field, he has led a number of website Security Testing and intrusion forensics analysis projects in the banking/securities industry to provide security protection technical support for the four major banks. Currently, entrepreneurs are engaged in Internet Security Protection