China Telecom Data Center 1: 192.168.0.8
China Telecom Data Center 2: 192.168.19.5
Telecommunications data center. 100.19.10
Background:
1. China Telecom Data Center 1 is a one-month-old China Telecom node with Mbps of dedicated bandwidth. The network connection is normal and has not been officially started.
2. China Telecom Data Center 2 and China Telecom Data Center 3 have been enabled for more than one year, with 10 m exclusive and M exclusive respectively. The network and bandwidth are both normal.
Problem Discovery:
1. Services in Data Center 1 should be officially launched, and large files must be transferred over m) to other data centers.
2. When transmitting data through SFTP, it is found that outgoing traffic is seriously abnormal, with a speed of dozens of K.
3. The basic operations, such as ssh, are still normal.
Network deployment:
The three machine rooms have the same structure.
650) this. width = 650; "onclick = 'window. open (" http://blog.51cto.com/viewpic.php? Refimg = "+ this. src) 'border =" 0 "alt =" simple graph "src =" http://www.bkjia.com/uploads/allimg/131227/03230K242-0.png "style =" width: 579px; height: 157px; "/>
Analysis process:
All tests are carried out directly between firewalls in three IDCs.
1. Test the network upload and download speeds between data center 1 and Data Center 2.
Data Center 1 --- Data Center 2]Upload exception]
Data Center 1 --- Data Center 2 download normal]
2. Test the network upload and download speeds between data center 1 and Data Center 3.
Data Center 1 --- Data Center 3]Upload exception]
Data Center 1 --- Data Center 3 normal download]
3. Test the network upload and download speeds between Data Center 2 and Data Center 3.
Data Center 2 --- Data Center 3 normal upload]
Data Center 2 --- Data Center 3 normal download]
Confirm that the problem lies in the newly activated IDC 1.
Analysis again:
1. Check the firewall system. No obvious exceptions exist. After the firewall is restarted, the fault persists.
2. Data Center 1 firewall and other servers in the data center upload and download at a normal speed of 20 Mb/s
Test related:
1. Test between data center 1 and Data Center 2
650) this. width = 650; "onclick = 'window. open (" http://blog.51cto.com/viewpic.php? Refimg = "+ this. src) 'border =" 0 "alt =" Data Center 1-Data Center 2 upload "src =" http://www.bkjia.com/uploads/allimg/131227/03230GS8-1.png "/>
650) this. width = 650; "onclick = 'window. open (" http://blog.51cto.com/viewpic.php? Refimg = "+ this. src) 'border =" 0 "alt =" Data Center 1-Data Center 2 download "src =" http://www.bkjia.com/uploads/allimg/131227/03230I3Z-2.png "/>
650) this. width = 650; "onclick = 'window. open (" http://blog.51cto.com/viewpic.php? Refimg = "+ this. src) 'border =" 0 "alt =" ping and tracert "src =" http://www.bkjia.com/uploads/allimg/131227/03230H0U-3.png "/>
Communicate with the IDC:
1. Send all test results between the three data centers to engineers in the data center for verification.
2. Problems found during the first off-duty time on Friday are sent to the past for verification.
3. We finally had time on Tuesday. We contacted the data center three times in a row to check and analyze the cause.
4. After several interactions, I finally confirmed the problem.
Cause:
1. The reason is very simple. The construction time of the data center is earlier, and the upper-layer network equipment has full duplex and half duplex problems, which must be specified by force.
2. After the data center sets the network connection to full duplex, the network immediately returns to normal, and the speed between data center 1 and Data Center 3 reaches 20 Mb/s.
Review later:
1. The problem occurred in the IDC five years ago. It took too long to forget it.
2. The data center 3 is the same as China Telecom. It has just been online for less than one year and has not been detected, so the dual-duplex problem is ignored.
3. Missing points: in fact, the NIC status of the firewall server can be directly found. The NIC status is half duplex.
This duplex problem also occurred in the Netcom data center four or six years ago.
Summary:
With the continuous upgrading of network devices, the problem of Half Duplex network negotiation has not been found for a long time and has almost been ignored. In some data centers that were built earlier, you still need to pay attention to the existence of such problems, especially when your firewall is customized, it is easier to find such problems.