The hurricane is at the next moment. How the CIO ensures the system operates properly

Unlike many CIOs, the struggle against natural disasters and extreme weather has become an integral part of Jane Rideout's work. She is the CIO of the Northrup Grumman Company and the dock system department. This year's Hurricane http://www.aliyun.com/zixun/aggregation/17798.html "> Katrina and last year's Hurricane Ivan attacked the dock system she was responsible for.

Not too much to prepare.

The docks are usually on the seashore, and the seaside may be hit by the storm at any time. One of the first priorities in fighting bad weather is to get ready, says Rachel.

Rico is responsible for the company's two shipyard information systems, one in Mississippi and the other near Avondale. Last year, Hurricane Ivan attacked the Gulf two times in a row, and as a result of adequate preparations, the hurricane did not cause damage. Rico successfully completed the company's planning mission and ensured the normal operation of the shipyard.

However, not every hurricane comes with enough time to prepare. This year's Hurricane Katrina played a joke on Rachel. In the morning of Friday, the hurricane changed the route of the forecast and turned to New Orleans. This made it less time for Rachel and her colleagues to prepare.

But she has a plan, and she plans to deal with the storm in almost the same way. In the face of this, she changed the way to back up a lot of important data. This time her staff sent the backup information to Dallas, which was previously sent to the mobile base station.

The backup data is sent to the company's Dallas data center because of the data damage that can be avoided by hurricanes. The hurricane left him short of time and had to move quickly. In the last storm, some important systems were backed up, allowing the machine to keep running. "We keep the mail system and the black Strawberry server running. "But it has paralyzed other machines, including a dedicated packet switch (PBX)," he said. ”

In addition, to protect the server from the rain that came in from broken windows or leaking roofs, he used plastic skin to cover a server that was not working.

In addition to this, the challenge for Rachel is how to provide enough power to keep the server running properly. Just last year, the information department of the company rented a standby generator, but when the storm needed to supply the shipyard's internal system, it was found to have run out of electricity. For Rachel, one lucky thing this year is to have a good generator. "We install the generators in the Avondale, and then it can be powered automatically," he said. "And in Biloxi, we need to control the generators manually," says UBS. "So the dock's defensive work is about the same." Northrup Grumman Information Services have done their best to protect the data center from Hurricane Katrina, and he hopes the progress will be as smooth as last year's.

Luck is just a coincidence.

However, things are not as smooth as hoped. The storm damaged the buildings of several shipyards, and, worse, the information building was one of them. When the storm hit the power station, Rico and her colleagues watched as the protected server was powered down. "The data center is in the far building. "Those buildings have 25 meters of draft inside." ”

The side of the data center building was blown completely, and Rico found things worse than she had feared, "we were hurt by the buildings." Some other machines have been damaged by the destruction of the building, and the information department needs to fix it first if it wants to get back to work. "Everything in the building has been destroyed. "We lost 200 servers, the network communications equipment was damaged and the public infrastructure was not working," he said. ”

As the storm subsided, Rachel began the rebuilding process. The first is to pick up a commissioner from Dallas, the commissioner to the data center to assess the severity of the damage before they can start other work. The existing offices are too bad to be used, but luckily they have a spare office. "The crew of the dock Systems team started working on a ship we were building," he said. She said her colleagues were busy working on the destroyer that was going to be completed. "There are already some Marines working on the ship," he said. She stressed that, although the ship had been delivered to the Navy, her colleagues had been welcomed by the crew and, with their help, trimmed the hurricane-hit dock.

However, there are still problems to ensure that the entire information infrastructure is backed up and functioning properly. The first thing to do is to re-establish communications to be able to communicate directly with the staff, which requires a certain level of communication network, says UBS. The staff of the Information Center use satellite telephones to communicate with each other so that the work below can be carried out.

The difficulty of rebuilding

Northrup Grumman launched the disaster recovery plan and started rebuilding the information center in Dallas. The first thing to do is to replace the damaged facilities in Mississippi, including servers, clients to communications equipment, and some infrastructure. When the first batch of equipment arrived at Dallas, the information department staff began to work.

Rico set a priority for the work to be done. From the location, first in the Dallas, and then elsewhere. On the system, priority needs to be restored to the mail system and the Black Strawberry System environment, followed by the data System (DDX), which is used to develop a new destroyer for the Navy, which will be used for a large project evaluation in a few days. Because it is a government project, it should be restored in a timely manner. Even for companies that need to be rebuilt after a hurricane, the government has not been given special treatment.

This poses a challenge to Rico. Now that the entire infrastructure has been compromised, it is much more difficult to restore everything to normal than a few servers. She needs a week to recover all the systems of the DDX project. The project team will need another week to find ways to work without relying on previous information facilities.

After launching an emergency plan and emptying two warehouses, Rachel realised that it would be too dangerous to stay on the coast. The host is now Dallas, and it is the only data center that can hold 200 servers and will not be shipped back for the time being. Fortunately, there is still information support for Rachel in Mississippi, so there is work to be done there. The new bandwidth is added to the data centers of the Mississippi and Dallas, ensuring a normal backup of both data.

But the lack of infrastructure continues to affect recovery efforts. "Our local network connection is also problematic. Originally, these network facilities were put in the dock. We had to move to a rented building where there was no internet facilities. "But she said there was a way to do that for this particular problem. "We're going to take advantage of the wireless networks that were completed before the storm. It took us 3 years to equip the dock with this wireless network. Now these wireless cards are finally coming in handy. ”

Man is the most precious.

Over the years, he has evacuated the dock two times with her information department colleagues and data center, and has been rebuilt afterwards. What would be the difference if she didn't? In this case, he argues, there is no doubt that it would be a very effective way to move quickly. However, before the hurricane occurred, she did not think of the need to find a transit.

He believes that the human factor requires more consideration and better planning. One of her colleagues was killed in the disaster, and One-fourth of her colleagues were homeless. "You must be moved when you think about what they do in spite of personal danger." She told the great price her colleagues had paid. "We must first care about the safety of our employees. ”

"In Dallas, we have 50 employees on the recovery server. "Rico thought of her colleague. "Some of them don't have a home here, which is really touching," he said. "So caring for her staff would be the top priority for Rachel. If the employee's heart and home are in Mississippi, she won't ask them to leave. "We won't let them lose their jobs because of this. But if they are willing to move to Dallas, we will agree. "Regardless of where the employees choose, they will try to meet their needs," he said. "They don't need to move. This is the benefit of technology. ”

Not every CIO has to fight hurricanes, but the principle that each CIO keeps the system running is the same.

How CIOs bid farewell to fire fighting

The business sector now relies more and more on it. When the system is normal, no one notices the CIO's importance and, if anything goes wrong, blames the CIO. In order to ensure the system is normal, as a CIO often a sense of trepidation. However, there are many factors affecting IT system operation, not only hurricanes, but also the misuse of viruses, hackers and even employees. These common problems keep the CIO on the run and fire everywhere.

In fact, coping with these daily accidents and the principle of dealing with hurricanes is the same. In order to cope with hurricane weather, the article took two measures to prevent and emergency rescue. In order to ensure the normal operation of the system, the CIO in peacetime to do a variety of preventive work, such as: timely patching, update virus database, the establishment of emergency measures. When the system is not functioning properly, the rescue strategy should be implemented in a reasonable order.

Summing up past experiences and anticipating possible future events, it is essential to ensure that the system is functioning properly and to reduce the number of CIO fires.