In October 2018, TDWI and Talend asked over 200 architects, IT and Analytics managers, directors and VPs, and a mix of data professionals about their cloud data warehouse strategy in a survey conducted in October 2018. We wanted to get real answers about how companies are moving to the cloud, especially with the recent rise of cloud data warehouse technologies. For instance, we wanted to know if a move to cloud data warehouse (CDW) is seen as a key driver of digital transformation. Which use cases are driving CDW adoption? Does a cloud data warehouse help companies become more data-driven?
We heard the feedback and divided the results in our latest report with TDWI in three key areas including progress, challenges, and what’s next.
Complex Cloud Data Warehouse Environments
Survey respondents noted that their cloud data warehouses need to do complicated work. They have to cross hybrid environments as well as accommodate a larger organizational shift to the cloud. In addition to all that, respondents wanted their cloud data warehouse to work for functions throughout the company, not just a select few technical teams.
The cloud data warehouse environment is getting more complex. The majority (36%) of the respondents indicated that they would be deploying their CDW in a hybrid environment.
CDW business use cases are spread throughout the organization. Interestingly, 62% of respondents in the process of implementing CDWs want them to complement a data lake for analytics.
Challenges for Successful Data Warehousing in the Cloud
However, there are still major road blocks for organizations to adopt CDWs successfully and the challenges and go beyond the CDW themselves.
Getting data into the CDW is only the beginning. Besides the complexity of getting various data ingested into a CDW, there are many more major challenges. The top challenges indicated by the survey respondents are governance (50%), integrating data across multiple sources (>40%), and getting data into the warehouse (38%). Some respondents told us “Cloud databases have their limitations, and on-premises will never go away completely, so the different environments just complicate everything”; and “We have hundreds of legacy systems without good master data. A cloud data warehouse does not fix a data structure problem”.
Meanwhile, the needs organizations have to perform data analytics in a CDW are increasingly complex. Companies require a number of additional processing and methodologies, the top 3 including in-memory processing (>35%), supporting structured and unstructured data (>35%), and integration with 3rd party analytics tools (>35%).
Survey respondents also indicated that they want data processing both BEFORE and AFTER data is loaded into a CDW. And the most common transformations & integration needs are:
Organizations today are fully on board with adopting a cloud data warehouse, but recognize that what a cloud data warehouse must do has evolved with changes in cloud computing, automation, machine learning, and other important trends. A CDW is no longer seen as an end in itself, but rather a stage in the data-driven journey, which has to involve managing a data lifecycle, ensuring data quality, providing a data governance framework, among other considerations.
To find out more details on the survey report, click here to read the full report.