Experiences in Data Center Construction

Experiences in Data Center Construction

The construction of data centers is a complex process. From initial requirement collection to mid-term product selection to fund allocation in the later stage, to final test acceptance, each part has its own rules. Data center construction is a long-term process. It usually takes two to three years from requirement collection to final delivery, and the demand will change with time. How to make the data center meet the ever-changing needs during the construction process and enable a longer life cycle for the new data center is a real problem.

We are fortunate to have participated in the construction of a medium-scale financial data center. I have accumulated some small experience in the whole process of data center construction and shared it with you.

From the initial demand collection stage, there were many problems at the time. First, the collection of requirements and the delivery date of the data center are two to three years, and the demand will change at this time; from three years ago, there was only a need to deploy minicomputers, high-end storage, and rack-mounted servers. Now, Minicomputers are gradually replaced by rack-mounted Server clusters, and storage can also be medium-and low-end products; from the time we don't use blade servers to the time when they become popular. This changes the capacity of each cabinet and the ratio of non-standard server areas to Standard Server areas. Changes in requirements also require dynamic adaptation of the IDC design. In addition, how to choose products can ensure high availability while taking into account flexibility. Flexible device deployment is the best solution to these problems. How to make the primary data center area more flexible is solved by converting the dynamic changes in demand into three dimensions: distribution capacity, cooling, and integrated wiring. In this way, three problems are derived. I. How can I configure cabinet Distribution? How can I allocate low-density areas and high-density areas? Ii. How can cabinet cooling meet the changing capacity requirements? 3. How can Integrated Wiring meet long-term planning requirements? After these three problems are solved, the requirements for changing requirements are solved.

The three most critical problems are distribution. We also found the rail-type busbar distribution item during product selection. The rail-based busbar distribution system has changed the way I think about cabinet power distribution. The cabinet power distribution capacity we define is no longer defined by the capacity of each cabinet, the total capacity of each cabinet is defined. Take a Cabinet with a capacity of 3 kW as an example: a column contains 14 standard cabinets, and the capacity of each cabinet is 3 kW, now, I can configure two rail busbars of a capacity for each cabinet, select different bus plug-in units of different specifications based on the actual capacity of the Cabinet, and regard the total capacity of the Rail busbars as a pool, it can be allocated to a Cabinet with a capacity of 5 kW or even 8 kW. In this way, my distribution method is upgraded from the original cabinet-based configuration to the configuration in the column-based unit. In the past, we needed to distinguish between low-density and high-density zones, I am now collectively referred to as "hybrid density zone ". It is easy to achieve high and low density areas. In this way, my cabinets are more comfortable to accommodate equipment. The 2U-4U rack servers can be placed, the 4U blade servers can also be placed, and the minicomputers can also be placed. You only need to control the total distribution capacity of the entire cabinet.

The second problem is cooling. The effectiveness of cooling caused by changes in the unit density of cabinets. I used a closed cold channel to solve the problem in a unified manner. After the cold channel is closed, the capacity of a single cabinet cannot exceed 8 kW. If there is a local hot spot, I can solve it by adding a cooling channel floor fan. In this way, even if the number of servers that can be carried by each cabinet changes dynamically, there will be no refrigeration problems. The number of 2U rack servers and 4U rack servers also changes with the elastic change of the Cabinet capacity. Because the bus plug-in unit can have an electric hot plug-in, we can adjust the positions of cabinets of different density according to the CFD detection, so that the bus plug-in unit follows the cabinet, use the air supply system of the entire module.

The last question is, how many cables are reasonable? What is the proportion of copper fiber cables? There is no doubt that replacing copper with optical cables is a trend, and 40 GB is also the future direction. But with the same problem, the new data center must take into account both the network physical connection needs of old equipment and future development. We cannot run 40 Gbit/s on copper cables, nor 100 Mbit/s on optical cables. We cannot re-deploy the cables due to the addition of some devices. The convergence of cabling systems and network systems is essential, and the network architecture of the next-generation data center is also essential. Our final solution uses Cisco's Nexsas7000, 5000, and 2000 architectures as the skeleton and structured cabling as the muscle to build my data center network platform. Nexsas2000 is used as the Cabinet top switch in the cabinet to solve the problem of mixed optical and Copper connections in the cabinet. Nexsas5000 is used as the aggregation center in the column and Nexsas7000 is used as the module network core. You can use pre-end cables + Nexsas2000 to quickly deploy the number of cables to solve local problems.

To sum up our data center construction, as a medium financial data center, we have little room for flexibility while focusing on high availability, of course, modular construction has different understandings in different types of data centers. We are also pursuing the goal of creating a new data center with a longer life cycle. Fortunately, the product suppliers of the data center provide us with so many new products that can be selected, so that we can implement some ideas to improve the utilization rate of our data center and prolong the entire data center lifecycle in disguise.