A real experience in defending against large-scale DDoS attacks

A real experience in defending against large-scale DDoS attacks

Each website is vulnerable to network attacks. The only difference is how to build defense and how to alert and respond.

It is difficult to find real cases on the Internet to defend against hacker attacks. On the one hand, information disclosure may lead to litigation; on the other hand, disclosure of such information may lead to adverse financial consequences, so companies are reluctant to share relevant details. However, if we do not disclose these stories at all, other people may make the same mistake without knowing it. If we contribute to the establishment of a threat intelligence sharing system and share the real experience on the network security battlefield, We can make things better.

Party B is a software as a service (SaaS) company that provides webpage content management services for large and medium-sized enterprises. Its service customer Company A focuses on helping medical suppliers improve their operation and financial performance. Company A provides services to thousands of hospitals and healthcare institutions to manage billions of dollars in spending.

The scale of the DDoS attack to be analyzed is extremely large: During the attack peak, 86 million users simultaneously visited the website from more than 0.1 million hosts around the world (the parties contacted the FBI ). 39 hours after the attack, the defender struggled to win a defensive victory. The following is the event process:

Crazy attack

Company A's annual meeting will be held soon, with 15000 participants. On the night before the meeting, Party B received an alert. Company A's website server is carrying incredible network traffic. It should be noted that Company A is also an SaaS provider that provides customers with data content and analysis. Therefore, this access volume will greatly affect the service quality and reputation of the company. There is not much time, and a quick response is required.

All access requests come from valid URLs. Therefore, it is difficult to identify malicious traffic.

Attack sources spread all over the world: North Korea, Estonia, Lithuania, China, Russia, South America

60% of access traffic comes from the United States

Attackers can directly unassociate DNS with attack IP addresses.

At first, the defender used AWS (Amazon Web Service) Route 53, reconfigured some files, and immediately cut off the communication with these IP addresses. After successfully resisting the first few waves of attacks, the network seems to be back to normal, but it turns out that this is only the first wave, and the next is the more crazy attack.

In the evening of the same day, the attack was initiated again and directed directly to the DNS domain name, which means that the previous IP address Blocking Policy was no longer effective and the data traffic saw a sharp increase.

Give up or resist

Party B discussed with the Chief Information Officer of Company A and finally reached an agreement: decide whether to resist. As an SaaS company that provides continuous and reliable services, maintaining the company's reputation is the top priority. Both parties agree to share the estimated defense costs of tens of thousands of US dollars to fight for justice.

After careful consideration of the second wave of attacks, the defender realized that some mitigation measures could be taken immediately:

1. Company A's business is only for American customers, but A small portion of the traffic comes from abroad. Therefore, the defender quickly established some firewall rules that only allow traffic from the United States to pass. As a result, 40% of the attack traffic was rejected.

2. A network application firewall is added behind AWS (Amazon Web Service) Route 53 and some HA agents are configured. This can collect a large amount of login information for the FBI to facilitate post-event analysis.

3. Adjust the automatic traffic scaling configuration. Auto scaling adjusts the upper and lower thresholds based on the traffic size. Adjusting the lower threshold is much larger than the upper threshold, which means that the system will make a proper response when the traffic increases, but it will never reach the lower threshold. As a result, each running instance is permanently present in the service list, and the complete and lossless login information is recorded for analysis by the FBI.

Attackers can zoom in the attack intensity, and AWS can zoom in the defense intensity. The attacker further amplified the attack intensity, and AWS further amplified the defense intensity, which was repeated the next day. At the same time, Party B reports the situation to the director of the board of directors of Company A every hour.

At the peak of DDoS attacks, a total of 18 HA proxy servers with enhanced computer capabilities and 40 large Web servers were deployed. The Web server group is very large, because although the traffic outside the United States is blocked and the total data traffic is reduced by 40%, the remaining 60% of the connections from the United States also include valid URLs. Most connections are accessing dynamic services, which are not easy to capture.

Attackers target a very large global Organization. The defender deploys a highly scalable Web server group through the HA proxy firewall and Server Load balancer configuration. CloudFront is the global content delivery network of AWS. Then Route 53 and Route 53 are AWS's global redundant DNS platform. These devices work together to form a critical infrastructure that provides scalability and security at each layer.

Things started to change around 7 o'clock the next day. After the defense strength is enhanced, the attacker does not increase the attack intensity further. At this moment, the server is carrying 0.1 million attack connections from 86 million hosts around the world, and the traffic through AWS infrastructure reaches 20 GB per second.

Attackers can only initiate repeated attacks until they completely give up. Afterwards, the CEO of Company A told us that if their websites are hosted in their own data centers, there will be very few defense options, in addition, the attack may not be able to respond within eight hours.

Finally, calculate the defense cost. This time, we successfully completed the 36-hour defense through Amazon's network service, with the cost not exceeding $1500. If both parties share their shares, they will not be worth $750.

The following are the experiences of this large-scale DDoS attack and some strategies that can be used to strengthen the data center and protect the company's website:

1. Design, configure, and test your devices for DDoS attacks. Use the experience of your hosting service provider to perform these tests and make good use of their assistance.

2. Confirm what "normal" looks like in your network environment. In this way, you can set an alarm immediately when the network status changes to "abnormal.

3. alias your public domain name (alias) to the internal domain name. This allows you to quickly respond behind the scenes and make DNS corrections in real time without relying on third-party service providers.

4. Learn how to effectively make DNS corrections when possible threats. Exercise frequently.

5. DDoS attacks use hundreds of attack vectors to exhaust server resources. Traffic testing starts many parallel threads, which may make DDoS attacks more aggressive. Each test should run for at least three hours to maintain the response time, but there should be sufficient response intervals between the two tests. A clear license should be granted before the significance test, risk suspension, and service cancellation.

6. Do not use the CPU load as a metric for automatic amplification configuration. The best evidence of DDoS attacks is an increase in the number of access HTTP requests, so it is best to use the number of access connections as a metric to trigger alarms.

7. The scale of defense should increase rapidly, but decrease slowly. The ratio of increase to decrease should be set to OR. This will allow the system to respond quickly to the first wave of attacks without increasing the scale repeatedly. This is especially true when attackers adopt the kite-flying tactics, that is, they are killed after the retreat.

8. If you use AWS Elastic Load Balancing, activate the cross-zone Load Balancing option. This is the best choice for balancing the traffic of the backend server group, which will significantly reduce the load on the DNS facilities.

The security industry needs to work together to better understand the tactics, technologies, and attack processes of the enemy, so as to seize the lead in the fight against malicious hackers.

CS Forum

Pay attention to Cyber Security, interpret Trends and Prospects, focus on management policies and system guidance, and provide consulting and suggestions for Security planning and implementation of enterprises and institutions.