The State of Enterprise Data Integration in 2020: Over 80% of Enterprises Believe that Data Integration is Essential

· More than 80% of corporate business operations leaders said that data integration is critical to ongoing corporate operations.

  · Today, 67% of companies rely on data integration to support analytics and BI platforms, and 24% of companies plan to achieve this goal in the next 12 months.

  · 65% of enterprise organizations tend to use cloud platforms or hybrid clouds to deploy data integration solutions.

  · Manufacturing is one of the three industries where data integration is the top priority today.

The above conclusions come from Dresner Consulting Associates' "2020 Client Access Reqd", which is the latest article in its "Wisdom Of Crowds®" series of research. Based on data integration usage and deployment trends, product and supplier data, present the status and trends of data integration.

Key insights from the 2020 data pipeline market research include:

  · Enterprises currently attach great importance to data integration, especially to support analysis and business intelligence (BI) plans throughout the enterprise, and will prioritize the deployment of data integration. Enterprise organizations rely on reports and dashboards for analysis and BI. Data delay affects the value of data obtained by enterprises from existing systems, which is an important factor driving enterprises to deploy data integration.


· 80% of business operations leaders (including those from service delivery, manufacturing, and supply chain) said that data integration is critical to their success. Dresner's research team found that this is not surprising, because a large number of data and workflow integration points are required to operate a business effectively and competitively. Executive management puts data integration at a higher priority than IT, reflecting the rapid adoption of real-time reporting and monitoring applications. The use of these applications is currently exploding in manufacturing and supply chain companies. The head of operations also stated that they have the least difficulty in accessing the required analysis content (from Dresner Consulting Associates' 2019 Data Catalog Market Research Report).


· Operations usually rely on data integration and data pipelines to support their data aggregation and graph creation needs. The reliance of financial services on search analysis has greatly promoted their demand for data integration and pipelines. Companies' ongoing machine learning projects also rely on the support of data integration.


· Manufacturing, government, and healthcare industries place the most emphasis on data integration. In the nine industries studied, data integration is critical or very important for most leaders in each industry. According to data from the McKinsey Global Institute, manufacturing is currently the industry with the most data, generating an average of 1.9 petabytes of data per year. Among them, most of the data is generated during the supply chain, procurement, factory operations, and compliance and quality management stages.



· Compared with other industries, medical organizations pay more attention to advanced data conversion technologies and functions. Government organizations have given high priority to embedding data quality and conversion technologies in their data pipelines (including data enrichment, data coordination, fuzzy matching and data deduplication in data pipelines). Financial service leaders and their organizations give relatively high priority to embedded statistical functions.


· The leading data integration use case in the enterprise is to achieve more effective data cleansing and transformation workflows to improve data warehouse reports and dashboards. The next important use case is ad hoc query, discovery, and exploratory analysis. Nearly 25% of business leaders said that it is not important to refactor or copy existing data warehouses. Data science and enhanced analytics are the least important use cases. Many data scientists use in-memory data integration/transformation functions, and end-user data integration/preparation tools are not sufficient.


· Education, retail and wholesale, and manufacturing industries have the highest satisfaction with data integration solutions. Among them, leaders from the education sector expressed the highest overall satisfaction with their data integration investments. The financial services industry executives are the least satisfied with data integration solutions. The financial services industry has historically faced the most demanding data integration challenges, and some interviewees said they were completely dissatisfied with their data integration solutions.

· Enterprises prefer to deploy data integration in the cloud, which is nearly twice as much as internal deployment. 65% of enterprise organizations prefer to deploy data integration solutions on cloud platforms or hybrid clouds. The Dresner research team found that many organizations also use hybrid data integration services that can collect, transform, and deliver data locally or in the cloud. In the #BIWisdom Twitter chat last Friday, Howard Dresner shared the latest data from an ongoing survey that is tracking the impact of COVID-19 on cloud adoption. He found that the enterprise organizations most affected by the pandemic are three times more likely to adopt cloud BI and two times higher than the overall sample. Data integration based on cloud deployment is the new normal for enterprises.



· Leaders from the advertising, consulting, legal, telecommunications, and transportation industries make multiple data sources and targets the highest priority for their data integration strategy. The government has the least diversity of data sources and targets, and gives priority to the data sources/targets of relational databases and document databases. Analytical databases dedicated to BI and analysis are the first choice for the technology industry. Manufacturing leaders (compared to other industry leaders) give higher priority to NoSQL databases, most likely due to the use of non-SQL data types in the machines and log files related to the manufacturing process.


· Relational databases dominate the data sources and targets of data integration and pipelines in enterprise organizations. Large enterprise organizations with more than 10,000 employees list Spark, Hadoop and related big data technologies as high priority/important technologies. Compared with other sized enterprise organizations, medium-sized organizations composed of 101 to 1,000 employees prioritize analytical databases (such as memory databases, column databases) as data sources and target integration points.