Information and network security need big Data security analysis

There is no doubt that we have entered the Big Data era. Human productive life produces a lot of data every day, and it produces more and more rapidly. According to IDC and EMC's joint survey, the total global data will reach 40ZB by 2020. In 2013, Gartner ranked big data as the top 10 trends in the future of information architecture. Gartner forecasts a cumulative output of $232 billion between 2011 and 2016.
Big data has long existed, but there has been insufficient basic implementation and technology to make a valuable dig of the data. As the cost of storage continues to fall, and as analytics continues to evolve, especially cloud computing, many companies have discovered the great value of big data: They reveal new trends that other tools cannot see, including demand, supply, and customer habits. For example, banks can have a more in-depth understanding of their customers, to provide more personalized customized services, banks and insurance companies can find fraud and deception, retail companies more accurately detect changes in customer demand, for different segments of the customer group to provide more targeted choices; pharmaceutical companies can use this as a basis for the development of new drugs, Track drug efficacy in detail and monitor potential side effects; security companies can identify more covert attacks, intrusions, and violations.
Current network and information security areas, are facing a variety of challenges. On the one hand, the enterprise and organization security architecture is increasingly complex, various types of security data more and more, the traditional analysis ability is obviously inadequate; on the other hand, the emergence of new threats, internal control and compliance in-depth, the traditional analysis method has many shortcomings, more and more need to analyze more security information, and to make decisions and responses more quickly. Information security also faces the challenge of big data.

　Big data for secure data
　The big data of safety data is mainly embodied in the following three aspects:
1) More and more data: The network has moved from gigabit to million, network security equipment to analyze the amount of data packets increased sharply. At the same time, with the advent of NGFW, the security gateway to the Application layer protocol analysis, the analysis of the data volume is greatly increased. At the same time, with the deepening of security defense, the content of security monitoring is continuously refined, in addition to the traditional attack monitoring, there are compliance monitoring, application monitoring, user behavior monitoring, performance testing, transaction monitoring, etc., which means to monitor and analyze more data than ever before. In addition, with the emergence of new types of threats such as apt, the whole-packet capture technology is gradually applied, and the problem of mass data processing becomes more and more obvious.
2) faster and faster: packet processing and forwarding speed is faster for network devices, and for Anguiping, event analytics platforms, the data source's event sending rate (eps,event per Second, number of events/sec) is getting faster.
3) More and more varieties: In addition to data packets, logs, asset data, security element information is also included in the vulnerability information, configuration information, identity and access information, user behavior information, application information, business information, external intelligence information and so on.
The big data of security data, naturally lead people to think about how to apply big Data technology to security domain.
　　traditional security analysis faces a challenge
　The rapid expansion of the number, speed and type of security data not only brings the problem of the fusion, storage and management of the massive heterogeneous data, but also shakes the traditional method of security analysis.
Most of the current security analysis tools and methods are designed for small data volumes and are unsustainable in the face of large data volumes. New attacks have emerged, with more data to be detected and the existing analysis technology overwhelmed. How can we perceive the network security posture more quickly in the face of the security element information of the day quantity?
Traditional analysis methods mostly adopt rules and features based analysis engine, must have the Rule Library and feature library to work, and rules and features can only describe the known attacks and threats, do not recognize unknown attacks, or is not yet described as a regular attack and threat. In the face of unknown attacks and complex attacks such as APT, need more effective analysis methods and techniques! How do you know the unknown?
Faced with the security data of the day, the traditional centralized security analysis platform (such as Siem, security management platform, etc.) also encountered a number of bottlenecks, mainly in the following aspects:
-High-speed collection and storage of massive security data becomes difficult
--the storage and management of heterogeneous data becomes difficult
-Small threat data source, resulting in limited system judgment
--the ability to detect historical data is weak
--The investigation of security incidents is too inefficient
--Security systems are independent of each other and work together without effective means
-fewer methods to analyze
-More difficult predictions for trending things, poor ability to early warning
--System interaction ability is limited, data display effect needs to be improved
Since the birth and establishment of intrusion detection technology in the 80 's, security analysis has been developing for a long time. At present, there are two basic development trends in information and network security analysis: Situational awareness security analysis and intelligent security analysis.
"Future information security will be situational-aware and adaptive," Gartner said in a 2010 report. Situational awareness is the ability to improve security decisions by taking advantage of the comprehensive analysis of more relevant element information, including asset perception, location awareness, topology awareness, application awareness, identity awareness, content awareness, and so on. Situational perception greatly expands the depth of security analysis, incorporates more security element information, pulls up the space and time range of analysis, and inevitably challenges the traditional method of security analysis.
Also in 2010, another Gartner report said, "Be prepared for the rise of enterprise Security Intelligence". In this report, Gartner puts forward the concept of security intelligence, emphasizing the need to integrate and correlate disparate security information from the past, and to integrate independent analytical methods and tools to create interactions that enable intelligent security analysis and decision making. The integration of information, the integration of technology will inevitably lead to the rapid growth of security elements information, intelligent analysis must require the machine learning, data mining and other technologies to apply to security analysis, and to make security decisions faster and better.
information and network security need big Data security analysis
The big data of security data, as well as the challenge and development trend of traditional security analysis, all point to the same technology-big data analysis. As Gartner made clear in 2011, "Information security is becoming a big data analysis problem".
As a result, the industry has the technology to apply big data analytics to information security-Big Data Security Analysis (BDSA), and some call it a big data analytics for security Security).
With the help of big data security analysis technology, we can better solve the problem of collecting and storing the information of the day quantity security elements, and with the help of the machine learning and data digging algorithm based on big data analysis technology, we can more intelligently understand the situation of information and network security, and respond to the new complex threats and unknown changeable risks more actively and flexibly.

Reprint: http://labs.chinamobile.com/mblog/631399_235858

Information and network security need big Data security analysis