90 Percent of Data Experts Seek to Alleviate Workload Increases from Fragmented Pipelines and Overwhelming Business Demands

Matillion, the leader in data productivity, and Vanson Bourne have released findings of a survey in a new report, “Data Productivity: A Survey of Data Experts.” The survey data offers insights from data team practitioners and leaders on how they manage the increasing complexity of unprecedented amounts of data, team workloads and overall performance at work.

The survey polled more than 900 experts across the United States and the United Kingdom to determine the impact of data and business demands on data teams. Survey results illustrate that data teams are overextending themselves to meet business demands.

• 84% of respondents described the volume of their workload as exceeding their capacity
• 90% reported an increase in their workload over the last year, a challenge that will only grow if not addressed soon

AiThority: How to Get Started with Prompt Engineering in Generative AI Projects

Additional key points from the survey include:

Teams spend too much time at work integrating, collecting and transforming data rather than performing strategic work.

• For nearly 40% of respondents, this work takes between a day and a week to perform per project
• 34% of respondents said this process takes 3 to 5 hours to complete per project — a sizable chunk of the typical 8-hour workday
• Employees using slow, inflexible pipelines can’t get the right information when they need it

Data professionals spend too much time pulling from and unifying a high volume of data sources to build their data pipelines, which only delays the transformation process.

• 41% of respondents reported that their company uses 51-100 data sources
• 22% of respondents reported that their company uses 101-200 data sources
• 6% of respondents reported that their company uses 201-500 data sources
• The sheer number of sources these experts use becomes a burden because they drastically increase their workload

Employee burnout hasn’t reached disastrous levels, but it’s on the rise, and it’s leading to challenges with talent retention for employees who enjoy their work.

• More than a third of respondents described feeling at least some burnout
• 19% of respondents described feeling very burned out
• 74% of employees responded that they feel very motivated at work.
• Companies must find ways to alleviate data team members’ workload if they want to avoid employee burnout

Read More: The Practical Applications of AI in Workplace

Low-code and no-code tools are emerging as a primary strategy for companies to transform business-ready data.

• Sophisticated, visually guided low-code environments help create accurate, maintainable and self-documenting code
• These solutions make it possible for team members with less coding expertise to pull and transform necessary business data
• Cost-effective and agile, these solutions also help end tribal knowledge, which can alleviate the challenges associated with employee turnover

“This research highlights the data productivity pitfalls that modern data teams experience on a day-to-day basis,” said Ed Thompson, CTO and co-founder of Matillion. “By taking note of these results — and partnering with organizations like ours to find solutions – businesses can empower their data teams to perform and prioritize impactful projects rather than the transformation of data; assure key decision makers with precision in their business decisions; and make it simple and cost-effective to deliver the right data to the right person at the right time.”

Thompson continued, “At a time where change is rapid, complexity is growing and the appetite for the right insights at the right time is voracious, addressing data productivity issues changes the game. If organizations don’t take these challenges seriously, not only will they cost money and time in lost resources, but will expose the talent pipeline to employee burnout and turnover.”

AiThority: How AI Can Improve Public Safety

[To share your insights with us, please write to sghosh@martechseries.com]