The energy industry supplies electrical power to consumers from a variety of sources, including gas-based and hydroelectric plants, as well as nuclear and coal-based power plants. As temperature, economic and political events occur along with changes in demography, preferences and technology, shifting demand and supply interact to form prices in competitive energy markets.  Supply and demand need to be managed on a daily second-by-second basis, otherwise blackouts may occur. In thi...

Introduction In my last blog I described how to achieve continuous integration, delivery and deployment of Talend Jobs into Docker containers with Maven and Jenkins. This is a good start for reliably building your containerized jobs, but the journey...

As a Customer Success Architect with Talend, I spend a significant amount of my time helping customers with optimizing their data integration tasks – both on the Talend Data Integration Platform and the Big Data Platform. While most of the time the developers have a robust toolkit of solutions to address different performance tuning scenarios, a common pattern I notice is that there is no well-defined strategy for addressing root causes for performance issues. Sometimes not having a strateg...

Deploying a successful technology solution, especially in data management, takes more than just installing software and writing a job (or multiple jobs… thousands in some cases), and running those jobs. If you’re taking on a new data management initiative, deploying using containers and serverless technology, migrating from traditional data sources to Hadoop, or from on-premises to...

In the last few years, microservices or microservice architecture has become a popular reference in IT due to its benefits and the flexibility this architectural style brings. Before we get into working with microservices and Talend, we should review the basics of microservices or a microservice architecture. I...

This is the first of a series of blogs on how to architect, engineer and manage performance.  In it, I’d like to attempt to demystify performance by defining it clearly as well as describing methods and techniques to achieve performance requirements. I'll also cover how to make sure the requirements are potentially achievable. Why is Performance Important? To start, let's look at a few real-world scenarios to illustrate why performance is critical in toda...

As enterprises move towards massively scaled interconnected software systems, they are embracing the cloud like never before. Very few would dispute the notion that the cloud has become one of the biggest drivers of change in the enterprise IT landscape and that the cloud has provided IT a powerful way to deploy services in a timely and cost-effective manner. However, the tremendous benefits that you’ve tapped into the cloud has to be balanced against the need to adopt, configure a...

When it comes to Data Matching, there is no ‘one size fits all menu’. Different matching routines, different algorithms and different tuning parameters will all apply to different datasets. You generally can’t take one matching setup used to match data from one distinct data set and apply it to another. This proves especially true when matching datasets from different regions or countries. Let me explai...

The challenge today for big data is that 85% of on-premises based Big Data projects fail to meet expectations and over 2/3 of Big Data potential is not being realized by organizations. Why is that you ask? Well, simply put on-premises “Big Data” programs are not that easy.  Your data engineers will need to be proficient in new programming languages and architectural models while your system admins will need to learn how to ...

In the past blogs, we have learned how to install Talend Open Studio, how to build a basic job loading data into Snowflake, and how to use a...