In the last few years, we’ve see the concept of the “Cloud data lake” gain more traction in the enterprise. When done right, a data lake can provide the agility for Digital Transformation around customer experience enabling access to historical and real-time data for analytics
However, while the data lake is now a widely accepted concept both on-premises and in the cloud, organizations still have trouble making them usable and filling them with clean, reliable data. In fact, Gartner has predicted that through 2018, 90% of deployed data lakes will be useless. This is largely due to the diverse and complex combinations of data sources and data models that are popping up more than ever before.
Migrating enterprise analytics on-premises to the cloud requires significant effort before delivering value. Cognizant just accelerated your time to value with a new Data Lake Quickstart solution. In this blog, I want to show you how you can run analytics migration projects to the cloud significantly faster, deliver in weeks instead of months, with lower risk using this new Quickstart.
Cognizant Data Lake Quickstart with Talend on Snowflake
First, let’s start by going into detail on what this Quickstart solution is comprised of.
The Cognizant Data Lake Quickstart Solution includes:
- A data lake reference architecture based on:
- Snowflake, the data warehouse built for the cloud
- Talend Cloud platform
- Amazon S3 and Amazon RDS
- Data migration from on-premises data warehouses (Teradata/Exadata/Netezza) to Snowflake using metadata migration
- Pre-built jobs for data ingestion and processing (pushdown to Snowflake and EMR)
Data Lake Reference Architecture
How It Works
- Use Talend to extract data files from on-premises (structured/semi-structured) and ingest into Amazon S3 using a metadata-based approach to store data quality rules and target layout
- Store data on Amazon S3 as an enterprise data lake for processing
- Leverage Talend Snowflake loader to move files to Snowflake from Amazon S3
- Run Talend jobs on execution connecting to Snowflake and process data