First-line users talk about the biggest challenges to big data

Klout Company is using data analysis to understand people's influence on social networks, said Virendra Vase, chief technology officer, "We are basically on social networks, gathering all the information to help people understand and recognize their influence." ”

The job involves processing 12 billion of data signals from more than 400 million users a day, and Klout's large Data environment includes Open-source platform Hadoop, as well as HBase, Hive, Elasticsearch, Scala, Storm, Node.js, and other tools.

Virendra Vase

Rachel Higham

Vase recently talked about some of the big data lessons that Klout learned. Brian Barnes, vice president of the company's chief Enterprise architect Rachel Higham,tenet Healthcare, Inc. ACE Group, also shared their big data skills, tips and problems. Now let's look at what they share:

The amount of data complicates testing, storage, and computing

The sheer volume of data makes testing a challenge. Handling unstructured data formats and data exported by social media (such as 140-character meager) is also not a simple matter. When it comes to storage, more data is not always better. "We have to figure out what kind of data we need to keep and what data we need to archive," he cautioned. In the end, it's not about data, it's about how we analyze data. ”

Technical maturity is a challenge

Vase warns that many large data technologies are still under development. While these basic technologies are becoming more sophisticated, the tools used for management and configuration are still in their infancy, leaving IT professionals to cope with the gap. "We need to develop attacks from a management perspective, from a workflow perspective, from a configuration perspective," he says. ”

Modular

For Klout, the modular infrastructure is important because it allows the IT team to handle business-priority changes and provide business transparency. He reiterated the need for businesses to invest in management and productivity tools. "This is where our 20% and 25% Engineering resources focus-productivity tools and workflow management." ”

Trying to find talent

Large data tools are developing rapidly, and vendor support is not fully in place, making it a challenge to find large data people. "It's hard to find talent and we can only hire good engineers and train them." ”

Large data experts work with business experts

We often talk about getting big data experts to work closely with product experts and business experts, but that's easier said than done, but it's still difficult to implement, but both sides need to know the other side.

ACE Group has been adept at using large data analysis to uncover risk and fraud patterns, identify new business opportunities, and understand customer sentiment. "I think the best lesson we have learned is to communicate our intentions, visions, and success stories within the enterprise," says Higham.

Define a vision, education leadership

"We spend a lot of time finding areas where big data can be used, for example, in our underwriting and claims process, and explain to the business unit," Higham said, "This gives us two proof of concept, now we have applied to more areas, and further use of some large data tools and technologies. ”

Set up a steering group

The ACE Group's Steering Committee is responsible for leading the company's large data project, which is surprisingly not entirely technical. "It includes 4 technicians and about 20 business leaders. ”

At the beginning of the current, equipped with data and talent

"One of the key things we invest in is to prepare the skills and resources before we start the project, without which we will not be able to provide value to the business," says Higham. ”

Spread the talent within the enterprise

The ACE group's large data specialists are dispersed across the enterprise, "we have built our core competencies across the enterprise, with new skills in the enterprise, including statisticians, data-screening personnel, data analysts, process specialists, etc." This collaboration can help us collect resources, share skills, and evolve and innovate. ”

Do not underestimate vendor management or system integration

For Vanguard Tiyatien Bae, the best way to do large data analysis is to use the 8-vendor approach, "We use 8 vendor solutions because there is no vendor-supplied solution." "Vendor management is a challenge, and integrating all the different systems is also a challenge."

Individual assessment of ROI

The second largest health care network in the United States Tenet Healthcare uses large data analysis to improve and validate the effectiveness of marketing campaigns. When big data is successful, everyone wants to be part of it. "When you deploy a large data project, you start bringing new revenue to the business, and then the other teams in the business jump out and say it's the revenue they bring." "For their team, the CFO can help solve the problem by providing a separate return on investment for large data projects," he said.

(Responsible editor: The good of the Legacy)