Talking about the development of big data: Problems and challenges

Currently, almost all world-class Internet companies have extended their business reach to the big data industry.

Social platforms, e-commerce price wars, and portal competition all have their shadows. Big Data is changing from a hot word of technology to a social wave, affecting all aspects of social life.

　　What is big data? Big data, or massive data, refers to the massive amount of data involved that cannot pass the current mainstream software tools, obtain, manage, process, and organize information that helps enterprises make business decisions more actively within a reasonable period of time.(In The Big Data era written by Victor Mayr Schönberger and kennis cokeye, big data does not use a shortcut such as random analysis (sampling survey, 4 V features of big data: volume (massive), velocity (high speed), variety (Diverse), and value (value ). Let's take a look at the definition of "Big Data" in the four features defined in "Big Data time generation,We can probably perceive its value: large data volume, many data types, low data value density, and timeliness.

Along with the development of various portable devices, Iot, and cloud computing cloud storage technologies, all tracks of people and things can be recorded. The core network node on the mobile Internet is a person, not a webpage.In the context of the big data explosion, big data also faces many challenges.

 

　　Challenges from data storage:Big Data development is faced with data information from different places, different standards, large data volumes, multiple structural forms, real-time performance, and other diverse requirements. These problems undoubtedly increase the difficulty of data collection and integration. Therefore, we should modify the block and file-based storage system architecture design to overcome the existing problems.

　　Challenges from data security:The continuous growth of data brings about data security issues. First, big data is more easily discovered on the Internet because of its large targets. Second, big data has more sensitive and valuable data, making it more attractive to potential attackers. In addition, personal information exposure may also cause personal security problems.

　　Challenges from data display:Compared with data analysis, many users are more interested in displaying data results. The traditional method of outputting results in the form of text or displaying results directly on a computer terminal may be a good choice in the face of small data volumes, but it is not feasible for complicated massive data forms. This requires the introduction of visualization technology to visualize the final and even intermediate computing results. In addition, human-computer interaction or data origin technologies are also required, this allows users to better understand the origin of results while obtaining results.

Challenges from data cost control:For enterprises that are using the big data environment, cost control is a key issue. To control the cost, it means we need to make every device more efficient and reduce those expensive parts. Technologies such as deduplication have already entered the primary storage market and can process more data types, which can bring more value to big data storage applications and improve storage efficiency. In an environment with increasing data volumes, you can reduce the consumption of backend storage by a few percent. Nowadays, traditional boot drives used by data centers not only have high failure rates, but also have high repair and replacement costs. If it is used to replace the independent server boot drive of the data center, the reliability can be increased by up to 100 times. It is also transparent to the host system and can provide a unique boot image for each additional server, which simplifies system management, improves reliability, and achieves a power saving rate of up to 60%.

Challenges from data analysis:Data analysis is the core of the big data processing process, because the value of big data is generated in the analysis process, but it also brings great challenges. First, a large amount of data brings greater value while also bringing more data noise. Therefore, we must be more cautious when performing data cleansing and other pre-processing operations. If the cleaning granularity is too small, it is easy to filter out useful information, and the cleaning granularity is too coarse to achieve the desired cleaning effect. Therefore, we need to carefully consider and weigh between quality and quantity, at the same time, it is also a severe test of machine hardware and algorithms. Second, traditional data warehouse systems do not have high requirements for processing time, but they are required in many big data application fields.

　　The significance of big data is associated with the increasing popularity of Internet behavior. Extracting useful information from massive data is a very huge project and a major challenge facing the big data era. After several years of criticism, questioning, discussion, and hyping over big data, the development of big data remains a long way to go.

Talking about the development of big data: Problems and challenges