Big data coming, companies are not ready?

Large data technology is exciting, innovative and powerful. Large data technology can definitely bring business analytics to a new level ... But not now.

The skills and best practices of business Intelligence (BI) have accumulated for many years in 1000 major global companies and countless smaller companies, and the relational database management system (RDBMS) is up to dozens of years. The products in these categories have excellent tool technology, manageability, and fault tolerance, providing interfaces for non-developer design, with carefully created data models that represent a great deal of input over the years.

At the same time, Hadoop is usually used on the command line, controlled by command mapreduce code, MapReduce code must be written in Java, and a file system (HDFS) controlled by a single and easily corrupted name node. While some browser-based tools are emerging, and similar hive technologies provide a layer of connectivity for BI tools, we are still in the technology of the 90 's. In this area, the enterprise is not yet fully prepared, not even close to full preparation.

Insight

Because of the strategies and logic involved in capturing, sharing and purifying data, translating data into information has become a long-term struggle in many areas. In solving this conundrum, big data is no better than business intelligence. Data "Big" only increases the managed data coverage, but it will make data analysis more complex.

One of the advantages of large data is that it has a more flexible approach that can be defined during polling/analysis, eliminating some of the complexities of managing data. However, the management of unstructured data tools is relatively immature, and enterprise data experts are conceptually not accustomed to this.

The long-term potential of large data is good, so it should shorten its innovation cycle. But in the short term it is not feasible.

Small Business usage and strategy

If I were a fast-food entrepreneur with 5 McDonald's franchises in a mid-sized city, it was not obvious to me how to use Hadoop and MapReduce to get more customer access. If I had a large network company, a large financial services company, a manufacturing-related enterprise or a large retailer with a large number of clicks, markets, and data, the big data would be more appealing to me.

I do think that small businesses should now open their big data strategy. If they are always online (most), they will even have a lot of click Data, and use the reprint operation, you can start to accumulate a large number of store video (can reveal the shopping habits, the effect of the store layout and product affinity). Data can help everyone, and when they no longer discard data, they become big data.

ROI of large data

In the Internet industry, big data can be profitable to attract attention and to monetize accordingly. In manufacturing, large data can benefit from reducing or eliminating assembly-line downtime (through predictive analysis of equipment failures). In the area of financial services, large data can make services better and more efficient, thus enabling better business strategies. Media companies can sell more ad pages. E-commerce companies can sell more products.

But these companies have one thing that ordinary business companies do not have: the return on investment is clear enough to allow these companies to exclude entry barriers into large data areas. Is the business team smart, budget and attractive enough to generate the necessary Hadoop experts, statisticians and data specialists to achieve attractive ROI? Probably not. In off-the-shelf products as well as professional services, large data values must be very good, cheap and mature enough to attract customers to buy.

Broader IT strategy impact

Big Data definitely has the potential to change the overall IT strategy of the enterprise. This is because large data involves more content. For example, Hadoop uses direct-attached storage and commercial hardware, which is extremely disruptive to our common enterprise deployments with storage networks, expensive servers, and equipment.

Hadoop may also give the enterprise more emphasis on Java skills, while reducing the focus of SQL skills. The clustering approach used by Hadoop may also accelerate the spread of hybrid presets/cloud strategies: it is easier to drive data to a preset server, but the resilience of cloud computing is more effective when it comes to solving the intermittent instructions of large clusters.

Skills shortages

Math, statistics, and data modeling skills are required, which is a problem. Many universities are now starting to set up courses in the fields of analytical and data science to address this problem. As I mentioned earlier, Java programming skills will be useful, even for data-oriented work, rather than for developer positions. For the strength of the enterprise, the most important and most difficult to find is the expertise and skills in these areas combined with talent. This is the formula for success, and it can be very difficult to recruit people who meet the requirements.

Which industries benefit?

Similarly, the Internet, media, financial institutions, online retailing and manufacturing industries will benefit the most. Supply chain enterprises, parts distributors can certainly join the benefit of the team. Medical research, management or payment/insurance operations can also benefit. Marketing organizations in these industries can derive great benefits from large data.

I think every organization has big data, not just organizations that don't have monitoring, data retention, or businesses and institutions that don't evaluate the costs and benefits of changing the operating model to large data.

Large Data and cloud

The Hadoop commercial hardware and the on-demand clustering approach have a great affinity for cloud computing models. In general, elasticity is one of the characteristics of both. On the other hand, uplink bandwidth is still a limiting factor for large data in the cloud. It is easier to process and maintain a cloud database (including Hadoop Distributed File system files) than to migrate a large amount of data and create a database. This is another area that will eventually change and remove obstacles.

Big Data Challenge

Data quality is a very big challenge. Data management is also a broader issue. In both cases, the rapid growth of unstructured data increases the difficulty of data integration. At the same time, many large data technologies have not yet matured and are potentially flawed. As a result, many companies are still in the development phase of large data. Large data technology must be easier, project management skills more extensive, large data can really become mainstream.

CEO and CFO

I think many CEOs know big data at a higher level, so they want to get big data. However, their management team must have a more nuanced understanding of large data and implement large data recommendations. Looking at this risk, I don't think we've achieved big data. Large data only more simple, managers fully familiar with, can be universal.

In many companies, the decision power of business intelligence is the CFO team. If large data becomes a successor to BI, then the CFO will be considered to maintain this right. However, large data project owners may come from various departments of it and the enterprise. Getting acquainted with technology and taking first-hand information may be a prerequisite for the success of the project. Financial-sector data is relatively fragmented – possibly containing a PB-scale general ledger – but I haven't met yet. So the CFO seems unlikely to be a big data decision-maker.

Five year forecast

Big data may be the culmination of a current hype (maybe not), but it's definitely not a fad. According to my experience, there is very little data correlation. Regardless of commercial application development and the corresponding transaction database requirements, or spatial analysis, various predictive analysis and other collected views on large data, we discuss useful and important techniques.

Typically, new data technologies start with innovation and breakthrough technology, then become mainstream and major technologies, and ultimately become day-to-day technologies, not failures and disappearing. I think that in the future big data will become the mainstream of the enterprise. The future is likely to be five years, depending on whether large data is able to weather its fragmented small industry phase during this phase.

Before a technology is widely used by enterprises, the technology must be very mature, even a bit boring. Large data will reach the level of widespread use of the enterprise, but it has to overcome some obstacles first.

(Responsible editor: The good of the Legacy)