NTT large-scale network faults

At about on January 26, May 15, NTT was deployed in 14 prefectures in the East Japan region, all NTT flets optical fiber networks and IP Phones using optical fiber networks are suddenly unavailable. In Japan, where IP phones are widely used to replace traditional phones, this is a very serious accident. Nearly 3 million of network users and 0.6 million of IP phone users are affected. After urgent repairs by NTT, the network was completely restored at 01:35, and the entire process lasted about 7 hours.

NTT immediately investigates the cause of the accident. According to the announcement at the conference on June 23, May 16, the cause of the accident started with a router located in front of Tokyo. The specific situation is that the vro fails on the evening of the 15th, so the staff will replace it with a backup vro. This device switch changes the network structure, so the other 4000 routers on the network will update the route table. So far, it is a normal maintenance step.

However, the update of this route table has a major consequence-a large-scale route update causes the router to reach the upper limit of its processing capability, so the whole IP network is paralyzed by the congestion that cannot be processed. After finding this problem, NTT immediately recovers by restarting the vro and other methods, and finally recovers all the data around midnight at half past one.

To avoid similar problems in the future, NTT released the following countermeasures: first, reduce router routing information to reduce load, reduce the number of Route entries managed by each vro from 10 thousand to 10 thousand. In addition, if the vro software version is older, resulting in poor processing performance, the processing capability is improved by updating the software version.

At the briefing, the Deputy President and senior officials of the NTT company apologized. However, the user will not be compensated or refunded for this accident, because the agreement stipulates that the fee will be refunded when the service is interrupted for more than 24 hours.

* ** The technical content blog has been transferred to codesoil. Welcome to visit.