In a typical/traditional data warehouse solution, the data is read into ETL memory, processed/transformed in the memory before loading into the target database. With the growing data, the cost of compute is also increasing and hence it becomes vital to look for alternate design. Welcome to pushdown query processing. The basic idea of pushdown is that certain parts of SQL queries or the transformation logic can be “Pushed” to where the data resides in the form of generated SQL statements....

Talend 7.1 brings in many new features, one such feature which needs a special discussion is containerization, which opens the doors to design and implement new Architectures namely Microservices. With a Maven plugin the Studio is now able to Build and Publish standard Jobs as Docker images into a Docker registry. The Talend Jobs which are built as Docker images provide us portability and ease of deploy...

Data integration is one of the hardest things developers have to do, but the talented members of the Talend User Group have lots of ideas on how to make it easier. Stitch, now part of Talend, recently hosted the inaugural meeting of the...

Introduction Talend Data Catalog provides intelligent data discovery that delivers a single source of trusted data into a centralized data catalog. Tale...

Recently, Databricks introduced Delta Lake, a new analytics platform that combines the best elements of data lakes and data warehouses in a paradigm it calls a “lakehouse.” Delta Lake expands the breadth and depth of use cases that Databricks customers can enjoy. Dat...

Last night it occurred to me that everything in the last three parts of this blog series ( Part 1 / Part 2 / Part 3 ) had...

Hello and welcome to Part 3 of my best practices guide on context variables! Before I get started, I just want to inform you that this blog builds on concepts discussed in Part 1 and Part 2. Read those before you get started.  Let’s say that we decide that we want our Talend Jobs to be able to ru...

First off, a big thank you to all those who have read the first part of this blog series!  If you haven’t read it, I invite you to read it now before continuing, as ~Part 2 will build upon it and dive a bit deeper.  Ready to get started? Let’s kick things off by discussing the implicit context load. The Implicit Context Load The Implicit Context Load is o...

A question I was regularly asked when working on different customer sites and answering questions on forums was “What is the best practice when using context variables?” My years of working with Talend have led me to work with context variables in a way that minimizes the effort I need to put into ongoing maintenance and moving them between environments. Thi...

In our previous blog, we walked through a simple job moving data from a CSV file into a Snowflake data warehouse.  In this blog, we will explore some of the more advanced features of the tMap component. Similar to the last blog, you will be working with customer data in a CSV file and writing out to a Snowflake data warehouse; however, you will also be joining your cu...