In a typical/traditional data warehouse solution, the data is read into ETL memory, processed/transformed in the memory before loading into the target database. With the growing data, the cost of compute is also increasing and hence it becomes vital to look for alternate design. Welcome to pushdown query processing. The basic idea of pushdown is that certain parts of SQL queries or the transformation logic can be “Pushed” to where the data resides in the form of generated SQL statements....

Talend 7.1 brings in many new features, one such feature which needs a special discussion is containerization, which opens the doors to design and implement new Architectures namely Microservices. With a Maven plugin the Studio is now able to Build and Publish standard Jobs as Docker images into a Docker registry. The Talend Jobs which are built as Docker images provide us portability and ease of deploy...

We all know that data is important and that becoming a data-driven enterprise is critical to future enterprise success. But recent events threw into sharp relief just how critical data is to business. Google announced its intention to buy Looker for $2.6 billion dollars. Several days later, Salesforce announced that it would be...

Data integration is one of the hardest things developers have to do, but the talented members of the Talend User Group have lots of ideas on how to make it easier. Stitch, now part of Talend, recently hosted the inaugural meeting of the...

…and how to fix them! We’ve asked our team of Talend experts to compile this top ten list of their biggest bugbears when it comes to jobs they see in the wild – and here it is! 10. Size does matter Kicking off our list is a common problem – the size of individual Talend jobs. Whilst it is often convenient to contain all similar logic and data in a single job, you can soon run into problems when building or deploying a huge job, not to mention trying to debu...

Let me start by thanking all those who read the first part of our blog series on converting legacy ETL to modern ETL! Before I begin the second part of this three-blog series, let’s recap the three key aspects under consideration for converting from legacy ETL to modern ETL. Will the source and/or target systems change? Is this just an ETL conversion from their legacy system to modern ETL like Talend? Is the goal to re-platform as well? Will the target system cha...

Last night it occurred to me that everything in the last three parts of this blog series ( Part 1 / Part 2 / Part 3 ) had...

Hello and welcome to Part 3 of my best practices guide on context variables! Before I get started, I just want to inform you that this blog builds on concepts discussed in Part 1 and Part 2. Read those before you get started.  Let’s say that we decide that we want our Talend Jobs to be able to ru...

First off, a big thank you to all those who have read the first part of this blog series!  If you haven’t read it, I invite you to read it now before continuing, as ~Part 2 will build upon it and dive a bit deeper.  Ready to get started? Let’s kick things off by discussing the implicit context load. The Implicit Context Load The Implicit Context Load is o...

A question I was regularly asked when working on different customer sites and answering questions on forums was “What is the best practice when using context variables?” My years of working with Talend have led me to work with context variables in a way that minimizes the effort I need to put into ongoing maintenance and moving them between environments. Thi...