The road to big data is thorny, but the opportunity is above all

As life grows richer, large numbers become more difficult to deal with, and because of the volume of data and the variety of data types, technicians have to overcome a lot of challenges and obstacles in the process of analysis. This article discusses why data becomes more complex and difficult to manage, and what challenges and obstacles we face when we analyze, consolidate, and store it, and, of course, what kind of opportunities the future brings with big data.

Big data is really big and complicated.

How big is the big data?

To take a simple example, go to a kid's birthday party. When you start, you send a tweet stating that the data comes along. Car on the halfway, stop refueling, payment decisively produced data. Buying birthday cards in supermarkets, scanning shopping cards and closing accounts also produced data. At a birthday party, take a photo, record a video, and create data when you publish on Facebook, Flickr, and YouTube. Messages sent during the party also produce data. Throughout the process, your phone generates data because it keeps sending GPS locations, and your car generates data because of constant tracking of fuel consumption. This shows that we produce a large amount of data in our daily behavior activities.

IBM learned that we set up about 2.5 quintillion (1 000 0003) bytes per day, while 90% of the total data volume was established over the past two years, while the volume of data increased exponentially. As corporate data capture increases, multimedia becomes popular, social media sessions increase, and more things are done using the Internet, the volume of data is incredibly fast.

How complex is the big data?

Large data is complex. Complex because of the diversity of data, including structured and unstructured data. The complexity of large data is also due to the speed of delivery and use, such as "real-time". Also, the complexity of large data is the volume of data. Previous home storage said MB and GB, is now talking about TB, and enterprises have already entered the PB unit.

Large Data market

Large data increases the need for information management business, such as Software AG, Oracle Corporation, IBM, Microsoft, SAP, EMC, and HP have paid 15 billion of dollars to software companies specializing in data management and analysis. In 2010, the industry itself was worth more than $100 billion trillion and grew at a rate of 10% a year-twice times faster than the entire software business.

Developed economies have made large data-intensive technologies more widely available. Worldwide, 4.6 billion of mobile terminals are generating data, and 10 to 2 billion people are accessing the Internet. Between 1990 and No. 2005, more than 1 billion people entered the middle class, and more rich people also contributed to the growth of information. In 1986, the World Telecommunication Network effective information interaction ability for 281 pb,1993 years for the 471 pb,2000 year 2.2 eb,2007 year for 65EB, and in 2013, the total amount of communication is expected to 667 EB.

Large data analysis

Large data requirements in a tolerable time to deal with large volumes of data processing special technology, large data analysis practitioners usually do not like shared storage, more inclined to direct-attached storage (checkmark Storage,das), High-speed SSD and high-capacity SATA disks are mixed in parallel internal processing nodes. The current shared-storage architecture san and NAS has been slow, complex, and expensive, and this type of architecture completely does not conform to the standards of performance, business servers, and low cost for large data technologies.

Real-time and near-real-time information delivery has become the defining feature of large data analysis, and avoiding delay is also one of the primary challenges of large data technology. The data would prefer to be stored in memory rather than on a mechanical hard disk attached to other terminal FC Sans. Also in large data scenarios, the requirements for profiling applications in San mode are much higher than other types of storage.

Of course, shared storage has its own advantages in large data analysis scenarios, but it has not been adopted by most large data practitioners since 2011.

Big data challenges and obstacles

Given the complexity, large data processing faces a number of challenges:

1. How do we understand and use unstructured data such as text or video?

2. How do we capture the most important part when data is generated and deliver it to the right person in real time?

3. Given the current volume and computational power of data, how to store, analyse and understand these data.

4. Lack of talent

The most discussed issue today is the lack of large data professionals, and thankfully many educational institutions have offered corresponding academic courses. And we also see some better phenomenon, enterprises and colleges and universities to jointly combat this talent scarcity problem, which is also the most effective way to train talent.

5. Other inherent challenges, privacy, access security, and deployment

Through EIU (Economist FDI Unit) and Lyris (digital marketing software provider) The latest report "Mind the Digital Marketing Gap" learned that 37% Marketing executives found great challenges in translating large data into decision making, while 45% said they did not have an effective large data analysis capability.

24% of marketers say they have been using large data technology to discover insights and develop marketing strategies, although most of them use data occasionally for feasibility analysis and personalized customer communications.

Other hurdles include a lack of funding (43% of respondents), too much emphasis on digital tools and social media, increased channels and lack of human resources (about 33% of respondents).

Big Data Opportunities

Although there are still many challenges in the application of large data technology, the opportunities that exist are far beyond these challenges. Large data has become an absolute weapon of innovation, competition and productivity gains, and we can use large data to answer previously unresolved problems. We can use large data to gain insight and knowledge, identify trends and increase productivity, gain competitive advantage and create more value for the world economy.

(Responsible editor: The good of the Legacy)