The secret of Facebook's high density data center: arm instead of Xeon

Core tip: At present, some small-scale traditional data centers in China, Pue value may be as high as 3, which means that it equipment consumption of 1 watts per watt, the data center infrastructure will need to consume 2 watts of power. Dr. Peng's unique energy-saving cooling system will enable PUE to reach below 1.5.

China IDC July 15 reported: At present some small-scale traditional data center, Pue value may be as high as 3, which means that it equipment consumption of 1 watts per watt, data center infrastructure will need to consume 2 watts of power. Dr. Peng's unique energy-saving cooling system will enable PUE to reach below 1.5.

1, choose the appropriate data center air-conditioning system

Because the heat of the data center is very big and requires the basic constant temperature and humidity always running continuously, so the air-conditioning system can be suitable for its use of high reliability (general design has redundant standby), refrigeration capacity, small temperature difference and high wind volume.

The refrigeration efficiency ratio (COP) of air conditioning system is the ratio of refrigerating capacity to effective input power when the air conditioner is operating under rated operating conditions. The size of this value reflects the energy saving of different air-conditioning products. The higher the refrigeration efficiency of air-conditioning system, the smaller the power consumption of the product when the same refrigeration is used, the less the consumption of the air-conditioning product in the unit time.

New Mandatory http://www.aliyun.com/zixun/aggregation/30825.html "> National standard" Room air conditioner energy efficiency limit and energy efficiency grade (GB12012.3-2010) by Aqsiq, National Standards Committee issued on February 26, 2010, from June 1 will be implemented nationwide. This standard stipulates 14000W refrigeration capacity of the Civil air conditioner energy efficiency grade indicators and test methods, inspection rules, room air conditioner products New energy efficiency limit, energy-saving evaluation value.

2 centrifugal water-cooled air Conditioning cooling system

This is the first choice for a new generation of large data centers, characterized by large refrigeration and high energy efficiency across the system (general energy efficiency ratio between 3 and 6).

The working principle of water-cooled chiller is as follows:

Frozen water side: general chilled water backwater temperature of 12 ℃, into the freezer and refrigerant heat exchange after the effluent temperature of 7 ℃. Chilled water generally through the fan coil, modular air-conditioning units or water-cooled precision air conditioners to the IT equipment to provide air-conditioning. Because the data center refrigeration volume is big, the request temperature difference small air volume and the humidity need to control, generally uses the water-cooled precise air conditioner.

Cooling water side: The general cooling water Inlet temperature is 30 ℃, enters the condenser and the refrigerant does the heat exchange, the effluent temperature is 35 ℃. Cooling water is generally used by evaporative cooling towers to dissipate heat and cool down by evaporation.

Refrigerant side: refrigerant at low temperature and low pressure overheating state into the compressor, after compression becomes high temperature and pressure overheating state refrigerant. When the refrigerant enters the condenser at high temperature and high pressure, it passes the heat to the cooling water and condenses into a high-pressure medium temperature liquid refrigerant. High pressure medium temperature liquid refrigerant through expansion device, become low temperature liquid gas mixed state refrigerant. Low-temperature low-pressure liquid-gas mixture state refrigerant into the evaporator, and frozen water for heat exchange, frozen water in this place is cooled, while the refrigerant is absorbed by the heat and evaporation, after the low temperature and low pressure superheated vapor state into the compressor.

The water-cooled air-conditioning system of large data center is composed of the following five parts, the schematic diagram is as follows.

Centrifugal refrigeration unit, generally for the n configuration, there is a standby machine, installed in the cold station room.

Cooling tower, installed in the outdoor, generally a refrigeration unit corresponding to a group of cooling towers (easy to repair and ensure that the standby system is ready to stand). Cooling water through the cooling tower to cool, because the water evaporation in the atmosphere, so the design and installation of water treatment system to remove dirt, dust, except sodium magnesium ions and germs, otherwise it will greatly reduce the refrigeration efficiency. In addition, because the data center is running continuously throughout the year, also design winter icing prevention measures.

Annular chilled water pipe. Because the data center needs to run continuously, therefore the water and the backwater pipeline of the chilled waters must design the loop, the large data center may design the two grade or the three grade loop, thus may close the adjacent valve to carry on the on-line maintenance and the maintenance when the partial freezing water pipeline or the valve fails. To facilitate future maintenance, repair, replacement and expansion, it is necessary to install a considerable number of valves. In order to prevent leakage and improve the service life, we need to choose high-quality valves, some projects using high-quality seamless steel tubes, even stainless steel tubes. Chilled water pipes and cooling hoses are not allowed to pass through the engine room area. Sewer and leaking alarm devices need to be set up in the area where the water pipe passes. In order to save energy and prevent condensate, chilled water pipes and cooling water pipes are to adopt strict insulation measures. The typical chilled water circulation pipeline loop is shown in the following illustration.

Water-cooled precision air conditioner. In fact, is a small temperature difference of large fan coil, generally recommended to use the floor under the wind and ceiling. In order to ensure the absolute safety of IT equipment and ease of equipment overhaul, it is recommended to set up a physical independent air-conditioning equipment, around the dam, the ground to do waterproof treatment and installation of drainage pipelines, install leakage alarm equipment. A redundant configuration scheme with N or n+2 is recommended.

Pump。 The refrigeration and cooling water cycles are carried out through a pump. In addition to the use of variable frequency device, the pump should adopt larger diameter pipe, reduce pipe length and elbow, adopt large radius elbow, reduce pressure drop of heat exchanger and so on. Refrigeration room, water pump, cooling tower, plate heat exchanger and precision air-conditioning as far as possible designed to install in a similar height to reduce the pump head.

Because of the large power load of water-cooled air-conditioning system in large-scale data center, it is necessary to design a separate distribution room for water-cooled air-conditioning system.

As can be seen from the above, water-cooled air-conditioning system is more complex, the cost is relatively high, maintenance is also difficult, but to meet the large data center cooling and energy-saving requirements.

3. Free cooling technology and data center location

Free colling technology refers to the total or partial use of natural free cold refrigeration to reduce the energy consumption of compressors or freezers. Common free energy sources are:

In the north-central region of winter and even spring and autumn, outdoor air stores large quantities of cold

The water temperature of some sea, river and groundwater is low, storing large amount of cold

A large amount of cooling water is also stored in some areas

Compressed gas produces a large amount of cold during vaporization

At present, the commonly used free cold source is mainly winter or spring and autumn outdoor air. Therefore, if possible, the location of the data center should be in the cooler weather or longer period of low temperature. In China, the northern region is very suitable for the use of free refrigeration technology.

Data center in the low ambient temperature of the season, the outdoor air filtered directly into the computer room as a cold source, can also save a lot of energy, called air-cooled natural cooling. The principle of this natural cooling method is relatively simple, the cost is relatively low, but there are the following deficiencies:

It's not easy to keep the air clean. Although the quality of the filter can be ensured through high-quality air cleanliness, but because of the air volume is particularly large, the need for frequent cleaning and replacement, while the huge resistance also consumes considerable energy.

Humidity is difficult to control. Humidification and dehumidification are quite energy-consuming. If the use of simple industrial humidification equipment, the humidification of water needs to be highly purified (high cost), simple softening water can not meet the requirements (harmful to equipment, long time in the equipment inside the formation of a layer of white matter).

Temperature is too low, easy condensation and dehumidification. Therefore, the need for meticulous and strict insulation treatment.

For large data centers, because of the distance, the air volume is particularly large, this requires a very large air duct, the power consumption of the fan is also very large. The actual design and installation is also very difficult.

It is not possible to achieve natural cooling year-round, and the refrigeration mode in summer also requires the installation of separate air-conditioning equipment.

Therefore, it is not recommended to use air-cooled natural cooling method in large data centers because of high natural environment requirements.

The use of water-cooled air-conditioning system, when the outdoor ambient temperature is low, you can turn off the refrigeration unit, the use of plate heat exchanger for heat transfer, called water-cooled natural cooling. This reduces the time to open the cooler and reduces the amount of energy consumed. The wet bulb temperature below 4 ℃ can satisfy the completely natural cooling, in the wet ball temperature 4-10 ℃ but realizes the partial natural cooling. In Beijing, an average of 5 months or so in a year can be achieved completely natural cooling, 2 months or so can be achieved part of natural cooling. The energy-saving effect will be very obvious.

4, the use of variable frequency motor energy conservation

We know that the refrigeration capacity of air-conditioning systems is closely related to the environment, the higher the summer outdoor temperature, the lower the refrigeration capacity, so large data center air-conditioning system refrigeration is based on the worst (the hottest in summer) operating conditions (the refrigeration capacity of air-conditioning is generally lower than its ideal operating conditions under the rated value, when the building itself not only not heat, Instead of endothermic. ）。 In this way, most of the year air conditioning system running in the load is not full state. In addition, the it load from zero to full loads in large data centers also takes a considerable amount of time, typically between 1-3 years. Also, the energy consumption of it load is related to network traffic or running state, according to its application characteristic, the energy consumption of 24 hours each day is changing, the energy consumption of 365 days is also changing every year. For example, the game server in the morning load and energy consumption is relatively low, but in the evening is relatively high;

Therefore, we strongly recommend that all motors in water-cooled air-conditioning systems use a frequency conversion system, which can save a lot of energy, the increase in investment in the general savings in a year of electricity can be recovered (basically full load). It should be noted that in the selection of inverter, the requirements of the harmonic coefficient is generally less than 5%, otherwise it will adversely affect the power grid. For fans and pumps, the input power is proportional to the three times of the speed of these devices. For example, if the speed of the fan or the pump is 50% of the normal speed, only the same equipment is required to run at 100% rated speed of 12.5% of the theoretical power. Therefore, when the equipment is running in part load, the energy-saving potential of the variable speed device is obvious.

5, improve the temperature of chilled water to save energy

The standard chilled water temperature of chiller is 7-12 ℃, and the standard working condition of water-cooled air conditioner is that the chilled water temperature is 7-12 ℃. But the temperature range is a bit low for the data center, with the following two disadvantages:

This temperature is much lower than the normal operation of the data center in the 40% relative humidity of the dew point temperature, will be in the fan coil formed a large number of condensate, the need for further humidification to maintain the environmental humidity of the room. This dehumidification and humidification process is a very energy-consuming process.

The temperature of the chilled water is proportional to the efficiency of the chiller, i.e. the higher the temperature of the chilled water, the higher the efficiency of the chiller. According to the York Company's network of materials, the cooling water temperature every increase, the efficiency of chillers can be increased by about 3%.

At present, in the container data center and high power density cold water back plate refrigeration technology, the chilled water temperature is designed to 12-18 ℃, has been higher than the dew point temperature, completely does not dehumidification, also do not need humidification. The cooling capacity of water-cooled precision air conditioner will decrease after the temperature of chilled water is improved. The actual refrigeration ability needs the manufacturer to provide the computer selection software to determine, generally will drop 10% to 15%. However, as the chilled water temperature increased little or no basic dehumidification and humidification, coupled with the use of EC speed fan, the motor generated by the reduction of heat, The actual cooling capacity (cooling) of the whole water-cooled precision air conditioner is not much reduced.

6, choose energy-saving cooling tower equipment

The structure and size of the cooling tower itself determines how much energy is consumed. For the general high-rise office buildings, due to the installation of cooling tower location is limited, the general choice of small volume of cooling towers, in order to achieve the prescribed heat dissipation, can only increase the power of the fan, rely on strong exhaust wind to increase evaporation capacity.

If the installation site allows, please select a larger cooling tower to save energy. According to a company's cooling tower tender information, the same refrigeration volume, the benefit of the United States high recommended product motor power of 55KW (larger), Marley recommended products of the motor power for 120KW,BAC recommended products of the motor power of 90KW. It is obvious that the large volume of cooling towers is significantly more energy-efficient than smaller cooling towers. In the same conditions, such as refrigeration capacity, as far as possible to choose a small fan power cooling tower.

According to the fan shaft power and rotational speed of three square, the use of multiple cooling towers with inverter operation may be more efficient than a single cooling tower fan full speed operation. For fan selection, propeller-type fans are generally lower than the unit energy consumption of centrifugal fans.

7, closed cold channel to save energy

After years of practice and theoretical proof, the best air organization of the data center is under the floor of the wind, ceiling back wind, precision air-conditioning set up separate equipment, cabinet face to place cold and hot channel isolation and achieve cold channel closure (to prevent air-conditioning short circuit), schematic below.

Domestic telecommunications operators in the early data center of the use of air duct mode, the disadvantage is that the wind speed before and after the difference is very large, it is difficult to control the air volume of each outlet evenly. Although each outlet has a regulating valve can adjust the size of the air volume, but the maximum wind volume can not be adjusted in the high altitude adjustment inconvenient, general operation and maintenance are not able to adjust in a timely manner.

8, improve the temperature of the room to save energy

According to the national standard of the People's Republic of China gb50174-2008 "electronic Information System Room Design Code", Class A and Class B computer room temperature requirements remain in 23±1℃, that is 22-24 ℃. The ansi/tia-942-2005 "Telecommunications infrastructure Standard for Data Centers" stipulates that the cold channel temperature is 68-77 degrees Fahrenheit, that is, 20-25 ℃. The American Society of Heating refrigeration and air Conditioning Engineers (ASHRAE) 2008 the new cold channel temperature of the engine room has been increased from 20-25 ℃ in 2004 to 18-27 ℃.

9. Save energy by scientific operation and maintenance

Large data centers can greatly improve the energy efficiency of air-conditioning system through scientific operation and maintenance. The common scientific operation measures include the following aspects.

People turn off the lights and turn off the large-screen display, reducing the energy consumption generated by lighting and the corresponding increase in cooling capacity.

Minimize the running of the fresh air system. In the room when no one close the corresponding area of the fresh air system.

It is recommended that the cold channel temperature not less than 25 ℃. Periodically check the temperature of the cold channel to avoid excessive refrigeration causing energy waste.

Regularly inspect the floors to prevent air leakage in various ways.

The cold channel must be completely closed, the area without equipment on the cabinet must have a blind board, the cabinet area without load cannot install the ventilated floor. Every device change is checked.

The temperature of the chilled water can be increased as much as 12-18 ℃ under the conditions of refrigeration.

Under the conditions of meteorological environment, try to use all natural or semi natural free refrigeration.

Regularly check the data collected data and analog display instruments are consistent, if the data are found to be error, immediately replace the relevant acquisition equipment or instruments.

For independent room areas that are not yet in use, close all electrical and mechanical equipment, including valves for chilled water pipelines.

Regular overhaul and cleaning of the copper tubes and cooling towers within the chiller to ensure that they are working optimally.

For the low load of the room, if the precision air-conditioning does not have EC fan, it is necessary to close the corresponding air-conditioning, if the EC fan, through practice to explore more energy-efficient air-conditioning operating mode.

(Responsible editor: Duqing first)