Dealing with large data challenges: management and oversight

Large data challenges require management oversight

For some organizations, one of the biggest challenges in managing and analyzing large datasets is to search for valuable information that can bring business benefits and decide which data can be discarded.

For example: UPMC, a Pittsburg healthcare network that has more than 20 hospitals and more than 50,000 employees, has seen rapid growth in data storage in recent years. William Costantini, the deputy director of the company's integrated operations center, believes that it is largely because employees are afraid to delete any information.

"The biggest problem at the moment is figuring out what you can clear and what you can't clear up because everyone is afraid to take responsibility and be sued," Costantini said. Everyone is afraid to discard any information or remove any content. At the same time, everyone wants to be careful to reduce the amount of data. ”

In addition to large data adjustments, organizations face an increasingly popular "data sandbox" that allows data analysts to explore and experience a subset of information that typically comes from an external data warehouse. Analysts also say the company needs to keep a close eye on the sandbox to ensure that they do not form incompatible data "chimneys".

In addition, databases and Hadoop installations used to store large, non transactional forms of data, are typically set up by an independent IT department application developer. "This is done by people outside, and usually it focuses on different tools," Adrian said at the BI Summit. Having been properly managed may be too generous a statement. ”

He added that Gartner's view is that organizations that can integrate those data types into a unified information management infrastructure will outperform those that do not.