In a typical/traditional data warehouse solution, the data is read into ETL memory, processed/transformed in the memory before loading into the target database. With the growing data, the cost of compute is also increasing and hence it becomes vital to look for alternate design. Welcome to pushdown query processing. The basic idea of pushdown is that certain parts of SQL queries or the transformation logic can be “Pushed” to where the data resides in the form of generated SQL statements....

Talend 7.1 brings in many new features, one such feature which needs a special discussion is containerization, which opens the doors to design and implement new Architectures namely Microservices. With a Maven plugin the Studio is now able to Build and Publish standard Jobs as Docker images into a Docker registry. The Talend Jobs which are built as Docker images provide us portability and ease of deploy...

Data integration is one of the hardest things developers have to do, but the talented members of the Talend User Group have lots of ideas on how to make it easier. Stitch, now part of Talend, recently hosted the inaugural meeting of the...

Introduction Talend Data Catalog provides intelligent data discovery that delivers a single source of trusted data into a centralized data catalog. Tale...

Think about your experience when you book a hotel room, order a taxi, or purchase something online. You reach the best offer in a few clicks and you get additional guidance with ratings so you can predict the quality of the goods or services you’re buying. It’s really helpful to have all that additional information. So why don’t we get a similar experience when consuming data? Well, now you can. Welcome to Talend Data Inventory. This is our new cloud native appl...

Last night it occurred to me that everything in the last three parts of this blog series ( Part 1 / Part 2 / Part 3 ) had...

Hello and welcome to Part 3 of my best practices guide on context variables! Before I get started, I just want to inform you that this blog builds on concepts discussed in Part 1 and Part 2. Read those before you get started.  Let’s say that we decide that we want our Talend Jobs to be able to ru...

First off, a big thank you to all those who have read the first part of this blog series!  If you haven’t read it, I invite you to read it now before continuing, as ~Part 2 will build upon it and dive a bit deeper.  Ready to get started? Let’s kick things off by discussing the implicit context load. The Implicit Context Load The Implicit Context Load is o...

A question I was regularly asked when working on different customer sites and answering questions on forums was “What is the best practice when using context variables?” My years of working with Talend have led me to work with context variables in a way that minimizes the effort I need to put into ongoing maintenance and moving them between environments. Thi...

In our previous blog, we walked through a simple job moving data from a CSV file into a Snowflake data warehouse.  In this blog, we will explore some of the more advanced features of the tMap component. Similar to the last blog, you will be working with customer data in a CSV file and writing out to a Snowflake data warehouse; however, you will also be joining your cu...