Matt Turck: Big Data industry information map and some thinking

Editor's note: This article comes from Pan (@ Star key V: Akavir), who worked on data quality analysis at Thomson Reuters for nearly 7 years, compiled from Matt Turck, the state's big data in 2014:a Chart, Matt Turck has served as managing director of Bloomberg Ventures and is now a partner in Firstmark capital.

It's been two years since I first tried to draw a picture of a large data ecosystem for prosperity, and there has been a lot of dramatic changes in the big data industry. I should have made an update on this picture long ago, and now I'm done.

From the VC point of view, I would like to talk about this picture and the big data industry some ideas:

Getting crowded: Entrepreneurs flock to the industry, and VCs invest large sums of money into startups that seem to have a chance of success, and as a result, the industry is becoming more and more crowded. Some categories such as databases (whether NoSQL or newsql) and social media data analysis are maturing, and the start of acquisitions or elimination (Twitter's acquisition of Bluefin and Gnip could mean that the trend has already begun in the area of social media data analysis).

For the latter, although space still exists, it seems that the early big wind investment bets were placed on infrastructure (infrastructure) and analysis (analytics), leading to higher standards of success. However, this does not mean that VC funds will stop flowing into these areas.

For some areas, the number of companies has apparently reached the upper limit that a map can hold. I'm sure there are some good companies we haven't been able to get into, maybe we didn't find it, maybe because it wasn't enough, I'm very sorry about it, and I want you to give us feedback and comments on the companies that should be included in the comments.

is still in its early stages: overall, the market is at an early stage of development. In the past few years, some of the favoured companies have failed (such as drawn to Scale), and some companies have exited early (e.g. PRECOG, Prior knowledge, Lucky Sort, Rapleaf, nodeable, karmasphere, etc.) , and some of the outcomes are slightly better (e.g. Infochimps, Causata, Streambase, Paraccel, Aspera, Gnip, Bluefin Labs, BlueKai).

At the same time, some companies seem to be doing more and more, and getting huge amounts of spectacular wind bets (such as MongoDB has financed more than $230 million trillion, Palantir financing nearly $900 million trillion, cloudera nearly 1 billion dollars). Some big companies are aggressively taking deals (Oracle buys BlueKai, IBM buys Cloudant), but on the whole, most companies are far from successful IPOs and investors ' success (though Splunk and tableau do). In many categories, startups and big companies compete with each other, but they don't appear to be market leaders.

Marketing is a reality: after all these years of indiscriminate marketing, is big data still the focus? In the next few years, perhaps big data will no longer be a hot topic for the media, but it is crucial for big data markets, as companies begin to turn large data projects from experimentation to full-scale deployments.

While this means that the profits of some big data providers will grow rapidly, they will also be a litmus test of whether big data will bring the value it has advertised. At the same time, with the rapid rise of the "IoT" industry, the data will accelerate as the tide increases, further boosting demand for large data technologies.

Infrastructure: Hadoop seems to have laid out its key part as a whole large data ecosystem, with some competitors still in the area that may be further developed and integrated. Sprak is another open source framework based on the Hadoop Distributed File System (HDFS), which seeks to fill the weaknesses of Hadoop, provide faster data analysis and a good programming interface, and is now attracting a lot of attention (some indication that it's doing well).

Some topics, such as real-time data processing, remain the most important, and new themes are emerging (such as the next generation of tools for processing, transforming, and cleaning data, including Trifacta, Paxata, and Datatamer). Whether or not the enterprise data will actually be put into the cloud (public cloud or private cloud), and if so, how long it will happen is another big topic. Many people think that Fortune 500 will continue to put data (and data processing software) in the engine room for the next few years. A group of cloud services +hadoop startups think that in the long run, all the data will eventually be put into the cloud.

Analysis tools: From the number of startups and VC investment, this area is the most active. From the Excel tabular user interface to timeline animation and 3D animation, startups offer a wide range of data analysis tools and user interfaces, and different customers do have different needs, so there is probably still plenty of room for development in this area.

There are also different strategies for promoting products-some startups are more focused on data scientists, who are not much but growing fast. Others, on the contrary, sell automated solutions to general business users, completely ignoring the existence of data scientists.

Big Data applications: As previously predicted, big data is slow but really moving toward the application level. This chart lists some exciting startups-they are essentially based on big data technologies and tools (we can't list all the relevant companies here, of course). Some companies offer horizontal applications-such as marketing systems based on large data, customer relationship management systems, and fraud screening solutions.

The financial and advertising technology industry has been the leader and the first fan of Big data promotion, even before big data is called Big data. Slowly, large data are extended to all walks of life, such as the medical and biochemical industries (especially gene research) and the education industry. It's just getting started.