An Introduction to Apache Airflow and Talend: Orchestrate your Containerized Data Integration and Big Data Jobs


In my last blog I described how to achieve continuous integration, delivery and deployment of Talend Jobs into Docker containers with Maven and Jenkins. This is a good start for reliably building your containerized jobs, but the journey doesn’t end there. The next step to go further with containerized jobs is scheduling, orchestrating and monitoring them. While there are plenty of solutions you can take advantage of, I want to introduce an effective way to address this need for containerized Talend jobs in this blog.

What Challenges are we Addressing?

When it comes to data integration or even big data processing you need to go beyond simple task scheduling. Indeed, you may want to run several jobs sequentially or in parallel in order to monitor them in an efficient manner. This is what we call workflow management. Even though this is not a new topic in this industry, we now need to consider that we are dealing with containers and their ecosystem.

Speaking of the container ecosystem, the rising adoption of Kubernetes shows us that more and more companies are betting on containers (as we do at Talend). As a result, our main challenge is to be able to provide a solution that can adapt to this fast-growing environment but also respond to our existing needs.

Why Apache Airflow?

According to Apache Airflow website:

Apache airflow is a platform for programmatically author schedule and monitor workflows.

In short, Apache Airflow is an open-source workflow management system. It allows you to design workflow pipelines as code. It uses Python which is a very popular language for scripting and contains extensive available libraries you can use. The value that Apache Airflow brings is: native management of dependencies, failures and retries management. Everything is easily configurable, and Airflow provides a great graphical interface to monitor your workflows. Like most of its competitors (such as Luigi or Pinball), it offers scalability and resilience over your workflows.

To sum up, Apache Airflow meets all the expectations we would have for a workflow management system. However, a key reason to choose Apache Airflow is the community behind it. A number of services integration have been already developed which considerably extends the capabilities of Airflow and its future is promising with a strong roadmap and a great number of contributors.

How to Use Apache Airflow with Containerized Talend Jobs

Before we begin, please be aware of the following requirements needed to follow our example:

The goal in this article is to be able to orchestrate containerized Talend Jobs with Apache Airflow.

  1. First, we are going to build 3 jobs as Docker container images.
  2. These jobs will run in a Databricks
  3. Then we will create an Apache Airflow workflow to create a Databricks cluster, run our jobs and eventually terminate our cluster.

The workflow is represented in the following schema:

1)    Talend 7.1 and Docker Containers

First, let’s drill in the new Talend 7.1 release. You now have two different ways to package your Talend jobs into Docker containers. First, you can use Talend CI/CD capabilities as introduced in Talend 7.0.

The 7.1 new release improves and eases the configuration to build your container images. Please read the documentation to see the updates. Talend has introduced a new feature allowing you to build your jobs as Docker images directly from Talend Studio. Let me show you how easy it is:

Right-click on any Standard or Big Data job:

It will build your job in a Docker image using your Docker daemon either locally or using a remote Docker host.

You can also publish it directly to a Docker registry:

Whether you are using the CI/CD Builder or the Studio to build Docker images you are now all set up to orchestrate your containerized jobs.

2)    Serverless Big Data with Databricks

Talend 7.1.1 new release introduces Databricks support. Basically, Databricks is a managed service for Spark available on AWS or Azure. You can start a Spark cluster in a matter of minutes and your cluster can automatically scale depending on the workload making it easier than ever to set up a Spark cluster.

First, let’s orchestrate a workflow around Databricks.

From the Databricks website:

Databricks unifies data science and engineering across the Machine Learning lifecycle from data preparation, to experimentation and deployment of ML applications.

Databricks is targeting different types of users on their platform such as data engineers, machine Learning engineers, data scientist and data analysts. While data engineers deploy their Spark jobs and build ML models, data scientists analyze and interpret, through notebooks, the data made available for them. In this context, it is becoming even more important to develop ETL and Big Data jobs and be able to orchestrate, schedule and monitor them in order to fasten the model creation and data availability iterations.

In the light of this, the use of Talend to operationalize and Apache Airflow to orchestrate and schedule becomes an efficient way to address this use case.

To illustrate my point, I chose the following workflow example:

  • Create a Databricks Cluster
  • Copy files from AWS S3 to Databricks DBFS
  • Run two Databricks Jobs packaged in containers (train a model and test this model)
  • Stop the Databricks cluster once the jobs are done

I have 3 different jobs. The first one is a standard data integration job that copies a file from AWS S3 to Databricks DBFS (Databricks file system):

The second one trains a machine learning model using a decision tree.

As you can see we have defined three context variables. These variables will be my only setup needed to run a job on a Databricks cluster.

In my case:

DATABRICKS_TOKEN=you can create one under user settings.

My last job is the testing of the previously trained model. It returns a confusion matrix based on the testing dataset.

Finally, you need to specify these three parameters in your job Spark configuration, and you are all set. I won’t go into the details of the job because the purpose of this article is to show how to orchestrate containers, but they are simple big data processing jobs training a machine learning model on a dataset and then testing this model on a different dataset. If you are interested in this use case, you can find a detailed description here as these jobs are available in our Talend Big Data & Machine Learning Sandbox.

As an example, once built either through Continuous Integration/Continuous Delivery (CI/CD) or from the Talend Studio you can then run your job as follow:

docker run trainmodelriskassessment:latest \
--context_param DATABRICKS_CLUSTER_ID="1114-095830-pop434" \
--context_param DATABRICKS_TOKEN="{TOKEN}"

 At this point, we now have three containerized Talend jobs published in a Docker registry allowing us to pull and run them anywhere.

3)    Apache Airflow

Let’s get started with Apache Airflow. If you have never tried Apache Airflow I suggest you run this Docker compose file. It will run Apache Airflow alongside with its scheduler and Celery executors. If you want more details on Apache Airflow architecture please read its documentation or this great blog post.

Once it is running, you should have access to this:

As you can see I have created one DAG (Directed Acyclic Graph) called databricks_workflow. Please read the description of Apache Airflow concepts if needed. In short, this is where you define the tasks, their execution flow and scheduling settings. Please find my Python file here defining my workflow.

In the Python file above you can see we have several tasks:

  • create_clustercreate_cluster_notify
  • s3_list_files
  • run_job (for each file listed in previous task) + run_job_notify
  • train_modeltrain_model_notify
  • test_modeltest_model_notify
  • terminate_clusterterminate_cluster_notify

Each of these tasks requires an operator. We have added a notification along with all the tasks using the Slack Operator that allow us to send Slack messages to follow the workflow advancement.

Example of notifications sent by Apache Airflow to Slack

We use the Python Operator for create_cluster and terminate_cluster tasks. The Python Operator simply calls a Python function you can see in the file. The train_model and test_model tasks use the ECS Operator that allows us to run a Docker Container in an ECS Cluster easily.

In our case, we use the containerized Databricks Jobs we earlier built, and we specify the 3 parameters to target our newly created Databricks cluster. This workflow associated graph is rendered as follows:

As you can see we use S3 List Operator to list all the files in a S3 bucket and then we use our containerized job to copy each of these files into Databricks DBFS in parallel. We dynamically pass the parameters with Apache Airflow to the container at runtime.

Then we run our other containerized jobs to train and test the machine learning model. In fact, only the training_dataset.csv and testing_dataset.csv files are used but to clearly show you the loop that Apache Airflow achieves, I have added 2 files in the bucket that are not used.


Thanks to Talend containers capabilities associated with an orchestrator such as Apache Airflow you can easily create complex and efficient workflows. Apache Airflow is still a young open source project but is growing very quickly as more and more DevOps, Data engineers and ETL developers are adopting it.

The above example shows you how you can take advantage of Apache Airflow to automate the startup and termination of Spark Databricks clusters and run your Talend containerized jobs on it. It potentially can reduce your cloud processing cost profile and help you monitor your data pipelines more efficiently. Monitoring the success and performance of your workflows is also a core concern for ETL, Data engineering teams. As you have seen you can track all the tasks with email or Slack notifications as well as get logs depending on the services you use. This is very useful to track errors and be more agile with your processes.

Join The Conversation


Leave a Reply