The 2019 Gartner Magic Quadrant for Data Quality Tools : Talend named a leader

Continuous Integration made easy in the Cloud with Talend

Continuous Integration made easy in the Cloud with Talend

  • Thibaut Gourdel
    Thibaut Gourdel is Technical Product Marketing Jr at Talend since 2017. His area of interest includes cloud technologies, containerization, serverless computing and data stream processing.

In this blog I would like to highlight a new capability from our latest 7.2 Studio and its integration with Talend Cloud in the Summer ‘19 release. While this feature could be seen as a light improvement aiming to ease the life of Talend users, it’s in fact much more than that!

While Talend has been delivering CI/CD capabilities for a very long time, the continuous integration tools landscape is evolving fast. Here at Talend we have kept pace to allow you to use the latest technologies and speed up your CI/CD pipelines faster than ever. That is why Talend has introduced the Zero install CI feature.

Simply put, you won’t have to manually install the Talend CommandLine. The CommandLine is the application that, along with Maven, assume all CI/CD related tasks such as builds, tests, and publications within the Talend product.

Why it is a game changer?

This new capability has much more impact than just facilitating the setup of CI/CD pipelines. It allows us to now take advantage of cloud hosted environments for our builds. Before I explain what these environments are and what they can bring to us, let me remind you how we used to consider continuous integration. Most of the times, we would dedicate one or more servers to CI tasks, meaning that we would have to provide physical servers or virtual machines (on-prem or in the cloud) to achieve our builds and tests. We would also need to manage, update, patch, and monitor these environments daily.

Talend Continuous Integration Continuous development

Talend Continuous Integration, Continuous Delivery and Continuous Deployment

 

While this situation was suitable and worked great, the rise of new cloud native services such AWS CodeBuild or Azure Pipelines introduced a new way to perform these tasks. Instead of providing our own machines, these new CI/CD tools give us the possibility to use on-demand environments hosted and managed by these cloud service providers.

AWS calls them build environments whereas Azure DevOps uses the term Microsoft-hosted agents. In both cases, these environments will take care of the builds and tests for you when requested. No servers to setup. You pay as you go. These environments can be virtual machines or containers that spin up on-demand and come with everything you would expect to build artifacts such as Maven, npm, Gradle or even Docker.

Undoubtedly, the Zero Install CI feature allows us to take advantage of these on-demand environments by not requiring the manual setup of a Talend CommandLine beforehand. We can leverage one of these environments and let it automatically install the Talend CommandLine on the fly and build our artifacts right afterwards.

Another benefit of the Zero-Install CI is that it is also much easier to leverage Kubernetes or any other container platform for your build executions. Indeed, most of the tools allow you to now execute your builds in containers to simplify and uniformize your CI/CD pipelines.

Great, what are the benefits?

The benefits of such architectures are multiple.

  1. No more CI software to install, you can stay within your favorite cloud provider console.
  2. Installation, maintenance and upgrades are taken care of for you. No need to worry about disk space or network slowness anymore.
  3. You get brand new environments each time you run a pipeline. Your builds are immutable and consistent between two builds.
  4. You scale up and down automatically depending on your build workload. Pipelines are run independently and concurrently. No more builds are left behind in the queues!
  5. And finally, you are charged based on the time to complete your builds. No more paying for unused servers.

 

How does it work in practice?

Technically, we have integrated within our Maven plugin a new mechanism to pull the Talend CommandLine from a given location. Instead of indicating where you have installed your CommandLine, you now specify a URL from which it can be pulled and installed for you. As you need to pull it, you will have to host it. A simple HTTP server will be enough, or you can choose to host it in Amazon S3 or Azure Blob Storage. The only thing needed is access from your build workers. For the Zero Install CI, we chose to distribute the CommandLine as a P2 repository. They are very common in the Eclipse space and composed of artifacts and metadata. They are independent from Maven repository but fit well in the Maven ecosystem.

Talend CI/CD Zero Install

Figure 1: CI/CD workflow with hosted build environments

As previously mentioned, there are several services giving you the possibility to use hosted build environments on-demand. Here is a non-exhaustive list:

 

Of course, you don’t have to stick to hosted build environments to use the Zero Install CI. If you want to still use Jenkins with your very own provisioned machines it will help you install Talend CommandLine in an easy and efficient manner. In a static mode like this, the Zero Install CI will only be actioned the first time, while the following build will then continue to use the same CommandLine.

Conclusion

Talend CI/CD capabilities are fully compatible with all CI/CD tools providing Maven support, which is one of the most popular standard package management and building tools. With the Zero Install CI, we are going even further. No more complex install and configuration, and you can take advantage of hosted build environments in your favorite cloud native CI/CD tool.

 

Join The Conversation

0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *