Rapid Time to Value Is the Main Driver for Analytics Projects, yet Making Data Available for Insights Is a Barrier for 90% of Enterprises

Matillion, the leading provider of data transformation software for cloud data warehouses (CDWs), and IDG Research have released findings of an IDG Research MarketPulse survey, “Optimizing Business Analytics by Transforming Data in the Cloud.” The research exposes the challenges companies face and the strategies they use to prepare data for BI and analytics, with faster time-to-value for implementing analytics projects rising as the main driver for migrating to a cloud approach.

The survey polled more than 200 IT, data science, and data engineering professionals at North American organizations with at least 1,000 employees. The top takeaways include:

Enterprises struggle to use data volumes for actionable insights.

• On average, data volumes are growing at a rate of 63% a month
• 12% report that their data volumes are growing at 100% or more per month
• More than 20% of those surveyed report drawing from 1,000 or more data sources
• Responsibility for exploiting data is divided between IT groups (55%) and business units (45%)

Read More: AI And The Future Of Content Management

Enterprises are tapping the cloud to simplify and scale data management.

• While less than one-quarter (23%) of respondents have completely centralized their BI and analytics teams, virtually all plan to tap the cloud for data management, migrate or continue to migrate data to the cloud over the next 24 months
• 90% have already placed some data in CDWs
• 37% of organizational data is in CDWs, 35% is in on-premises data warehouses, and 25% is in offsite, non-CDWs
• Three CDWs dominate: Amazon Redshift (used by 54% of the respondents), Google BigQuery (50%), and Snowflake (26%)
• The top reason enterprises migrate their data to cloud platforms is faster time-to-value for implementing analytics projects

Despite broad and growing use of CDWs, they aren’t a panacea; CDW adoption alone does not address all data analytics needs.

• More than 90% said it is challenging to make data available in a format usable for analytics
• Respondents cited several obstacles slowing their data analytics projects, including a lack of necessary data granularity; manual coding of data pipelines; and difficulty connecting with multiple data sources

Read More: Radpay’s CryptoClick Is an Innovative New Patent-Pending Blockchain-Based Solution for Ecommerce and Digital Wallets

Part of the challenge is in how enterprises are using legacy ETL processes in a modern data environment.

• Many struggle with the sequence and process of data transformation; more than one-third (37%) manually code data into the necessary format before loading it into BI and analytics tools
• Only 28% load data into the cloud and then use the cloud platform to automate the transformation process
• Data portability (45%) and scalability (46%) top the list of perceived benefits of a modern approach to data transformation

“Without the right data management strategy, growing data volumes go from a competitive advantage to a company-wide struggle to make it useful,” said Matthew Scullion, CEO of Matillion. “Our research underscores the demand and the urgency to use the flexibility and scalability of the cloud – not just to store data, but also to accelerate insights through data transformation.”

Read More: How Artificial Intelligence Is Regulating Live Video Streams