The best part of holiday travel: Flight tracking with Talend Pipeline Designer

Nobody likes air travel during the holiday season, especially when it involves long queues at the airport security checkpoints or delayed departures thanks to unfavorable weather. Of course, once onboard, then you have to deal with a crying baby nearby or an eager conversationalist at your side. Oh, the agony!

Luckily, amidst the unpleasantries associated with air travel, there is still consolation for those of us who have opted out of the air fair and have instead chosen to sit idly by at our computer to watch the air traffic chaos unfold from a safe distance.

When a new tool is released, the full end-to-end examples are hard to find, and yet, this is one of the best ways to get hands on experience. “We learn by example,” we used to say. And what better way to learn than with a cool real-time example. With Talend Pipeline Designer, now available since April ’19, I will show you how you can build a detailed air traffic tracking dashboard in real-time from the comfort of your spacious armchair. Sounds cool, huh?

What I am about to show you will dive deeper into the awesome capabilities of Talend Pipeline Designer and I am hoping that by following it, you will get a good overview of the product. Usually, when I want to learn a new tool I always start with a guided tutorial or blog, such as this.

Disclaimer: A getting started guide is always available in the documentation and I highly recommend you give it a look before continuing on with this blog.

How to Access Pipeline Designer?

There is no download required to build your data pipelines—the entire interface is web-based, and the product is hosted in the cloud. If you are a current Talend Cloud customer, ask your administrator to activate Pipeline Designer (it comes free with all Talend Cloud developer licenses!). If you aren’t a Talend Cloud customer, please give Pipeline Designer a try during a 14-day trial, it takes less than two minutes to get started.

<< Sign up to the free trial >>

Talend Pipeline designer

What are we going to achieve?

The goal here is first to ingest data in real-time from an open API providing us current aircrafts information. Thanks to Talend Pipeline Designer we will process the incoming data along the way in order to aggregate and transform these and eventually display aircraft positions on a nice visual dashboard.

Note: This post is written specifically for beginner-level users. However, having a grasp on technologies such as Apache Kafka and Elasticsearch is needed.

Before we start, what do we need?

As you may have understood, we are going to process data in real-time which means in a streaming mode as opposed to batch mode. To do so we will use a message broker as a source to ingest data, and I am calling out Apache Kafka! Now to store and display our airplanes information we are going to use Elasticsearch as a destination along with Grafana on top of it. In this article I won’t go into much detail about how these tools work, so don’t hesitate to look at their documentation if needed. However, be reassured, this tutorial is very detailed to make sure you can achieve it yourself without expertise on any of these tools.

To sum up, Talend Pipeline Designer will ingest data from a source, in our case Kafka, and push the processed data to a destination, Elasticsearch. The following requirements are needed:

  • Zookeeper + Kafka 0.10.0.x to 1.1.x
  • Elasticsearch v6.5
  • Grafana (latest version preferred)
  • Talend Pipeline Designer (sign-in for a free trial here)

Talend Pipeline designer

To help you, we have left at your disposal a docker-compose file to deploy Kafka, Elasticsearch and Grafana anywhere you want if you have Docker installed. You only need to make sure the pipeline will be able to access them. See next section to understand where your pipeline will be executed.

With the docker compose file provided, you can run the following command in the same folder on an accessible machine:

docker-compose up

Alternatively, you can use a managed Kafka service for Kafka and a managed Elasticsearch service in the cloud.

Where the pipeline will be executed?

With Talend Pipeline Designer you have multiple ways to run your pipelines. If you subscribe to a free trial the easiest way to go is to use the cloud engines provided to you. These are engines that are managed by Talend and you don’t need to setup anything. If you want to run your pipeline in your own environment (VPC or even your laptop) you can install what we call a Remote Engine for Pipelines. Please look at the documentation to install a remote engine.

Quick tip: The fastest way to deploy a Remote Engine for pipelines is to use the CloudFormation for AWS or the ARM Deploy form for Azure.

Where does the data come from?

Good question! We will use the OpenSky Network open API based on the ADS-B protocol. First, ADS-B is a technology used by aircrafts to periodically broadcast their position. It allows air traffic control stations to track airplanes in addition of their regular radars. Luckily these signals are not encrypted for non-military aircrafts. Consequently, a community of several individuals has built a network of antennas around the world to gather in real-time most of the ADS-B signals and openly offer them through an API. You can have a look at the data fields available here. To ease the ingestion of the data we have prepared a script that periodically pushes the data from the API to a Kafka topic. This is a NodeJS script, so you need to install npm and nodejs to run it. As this can be cumbersome and if you like Docker as much as I do, just pull the following Docker image and I will explain how to run it later.

docker pull talendinc/flight-tracker

For your information, Talend Pipeline Designer works natively with Avro. To put it simply, this is a serialized format in binary data. It allows way much compact data transmission meaning a better throughput, but the drawback is the need of specifying a schema. Indeed, as the data is in binary, we need a schema in order to retrieve the different fields and values from the documents. Therefore, the data requested from the open API will be converted into Avro to speed up our pipeline!

Now let’s get our hands-on Pipeline Designer!

In the next part of this blog you will have to replace {KAFKA_IP:PORT} and {ELASTICSEARCH_IP:PORT} with your own relative IP addresses and ports.

1) Data source

Once you have Talend Pipeline Designer at your fingertips we are going to start by creating a connection for Kafka. To do so click on “Connections” on the left menu and press the button “Add a connection.”

Fill the form with the following and test the connection.

Talend Pipeline designer

The next step is to create a Dataset. A Dataset is a particular set of data from a connection. For example, for MySQL it would be a table or for AWS S3 it would be a bucket and file. However, for a message broker such as Apache Kafka it is a topic. Create it as follows:

Talend Pipeline designer

You copy-paste the Avro schema:



If you have already tried to look at the sample, of course, you have found that the sample is empty. This makes sense because your Kafka topic is not populated. Let’s use the Docker image I made you pull previously:

docker run -e KAFKA_TOPIC=flights -e KAFKA_HOSTS=< KAFKA_IP >:9092 –e CITY=Paris talendinc/flight-tracker

This will send aircrafts detected around Paris to the Kafka queue. You can change the name of the city. It works with the biggest cities in the world.

Please wait a bit and then try again to get the preview. You should get some records in your sample:

Talend Pipeline designer

We have our data source, let’s continue.

2) Data destination

As mentioned in the introduction, we chose Elasticsearch as a destination. The reason why Elasticsearch is a good fit is that it is particularly effective for time-series data. In our case we are going to get aircraft data on a periodic basis with a time date. In addition, Elasticsearch can be easily associated to visual dashboard tools such as Kibana or Grafana. In the context of this blog we chose to use the latter.

Before creating a connection to Elasticsearch in Talend Pipeline Designer we are going to prepare our Elasticsearch indices. An index is what we could compare to a table for a regular SQL database. Elasticsearch can be managed through its RESTful API. As a matter of fact, let’s create an index called “aircrafts” and specify its mapping. To do so, perform the following request:

curl -i -X PUT \
   -H "Content-Type:application/json" \
   -d \
' { 
}' \
 'http:// {ELASTICSEARCH_IP:PORT}/aircrafts'


Now that your Elasticsearch instance is ready, you can create a connection and a dataset for it in Talend Pipeline Designer.

Talend Pipeline designer

Talend Pipeline designer

3) Data Processing

We are now going to create the pipeline. Click on Pipelines on the left menu and click on “Add a pipeline”. You should get an empty canvas where you can add a new source. That is great because you have created one in the previous section. So, click on that box and select your dataset “Kafka Flights”. You should get your sample data available, check that you get it all right for the following steps as we want to view the live preview feature of Talend Pipeline Designer in action!

a). Window

The first component we are adding is a Window. The aircrafts data is pushed to Kafka every five seconds by the script (provided previously). This is streaming data, so if we want to aggregate it, we need to set time boundaries in order to achieve the aggregations. That’s why we are going to add a fixed window of 5 seconds. Please have a look at the documentation to understand the windowing notions such as fixed, sliding and session windows. In our demonstration we only want aircrafts position updates in the last 5 seconds time frame which correspond to a fixed window.

Talend Pipeline designer

b). Aggregate

Now that the data flow is cut in time frame windows, let’s aggregate the aircraft data in each of them. The goal of this aggregate is to remove duplicates. In fact, you could get the same aircrafts twice in the same 5-second window. To avoid that, we “Group By” icao24 and origin_country. In the case of several identical aircrafts found in the 5-second window, we want to list the different positions. As operations we want:

MAX of time_position

LIST of latitude and longitude

You should get something like the following screenshot:

Talend Pipeline designer

c). Field Selector

This is where we are going to see one of the greatest Talend Pipeline Designer feature. The Field Selector is a pretty simple component. You can select which fields you want to keep as output of this component.

But where it becomes interesting is the availability of Avpath, (an expression language to select, update, filter Avro data). As I was telling you before, Talend Pipeline Designer natively supports Avro format. That is why you can use Avpath (an equivalent of Jsonpath or XPath) in Talend Pipeline Designer. The Talend Avpath documentation is available here and you can take a look at its usage in the examples provided. In our case it’s going to help us select the latest position of an aircraft in the case we have multiple positions for the same one. We simply use the “[-1]” from Avpath, and it selects the latest element of the list. This works in our example because we know that the Open API assures a chronological order of the positions.

Talend Pipeline designer

d). Python

The last step requires the Python component which allows us to use custom Python snippet codes to fulfill our custom needs in Talend Pipeline Designer. As you know, our goal is to display aircraft positions on a world map in Grafana. To achieve this, we are going to use the Worldmap Panel plugin of Grafana that only accepts Geohash positions. A Geohash is a string that represents geolocalization in the world. It is particularly useful because it can also represent rectangular surfaces depending on the precision used to encode the position. In our case we use it as coordinates. Paste the following code is the Python component (Flat Map mode):


def encode(latitude, longitude, precision=12):
    __base32 = '0123456789bcdefghjkmnpqrstuvwxyz'
    __decodemap = { }
    for i in range(len(__base32)):
        __decodemap[__base32[i]] = i
    del i

    lat_interval, lon_interval = (-90.0, 90.0), (-180.0, 180.0)
    geohash = []
    bits = [ 16, 8, 4, 2, 1 ]
    bit = 0
    ch = 0
    even = True
    while len(geohash) < precision:
        if even:
            mid = (lon_interval[0] + lon_interval[1]) / 2
            if longitude > mid:
                ch |= bits[bit]
                lon_interval = (mid, lon_interval[1])
                lon_interval = (lon_interval[0], mid)
            mid = (lat_interval[0] + lat_interval[1]) / 2
            if latitude > mid:
                ch |= bits[bit]
                lat_interval = (mid, lat_interval[1])
                lat_interval = (lat_interval[0], mid)
        even = not even
        if bit < 4:
            bit += 1
            geohash += __base32[ch]
            bit = 0
            ch = 0
    return ''.join(geohash)

output = json.loads("{}")
output['id'] = input['id']
output['country'] = input['country']
output['date'] = input['date']
output['location'] = encode(input['lat'], input['lon'])


Talend Pipeline designer

4) Run the pipeline

Now add your Elasticsearch dataset as the destination of your pipeline and run it!

As explained before you can send real-time data to your Kafka queue with this Docker image:

docker run -e KAFKA_TOPIC=flights-e KAFKA_HOSTS=<IP_KAFKA>:9092 –e CITY=Paris talendinc/flight-tracker


Talend Pipeline designer

5) Dashboard

Now open your Grafana tool and import this dashboard. Make sure to connect the data source to your Elasticsearch instance. The dashboard is setup to get the last 5 minutes of data in Elasticsearch and refresh every 5 minutes. You can see the aircrafts are moving quickly and you can guess the path they follow.

Talend Pipeline designer


As described in this article, you can create data pipelines, batch or streaming, in a matter of minutes with Talend Pipeline Designer. In our case we focused on streaming processing with Kafka as a source and Elasticsearch as a destination. It allows us to get aircrafts position in real-time that we lightly transform and store to follow the aircraft live positions. Building such a pipeline by code would have taken so much more time and yet Talend Pipeline Designer requires no code to create efficient and powerful Spark processing pipelines.

If you are interested in seeing more don’t hesitate to sign up for free to a 14-days trial. You’ll be able to reproduce this example and create your own pipelines!


Join The Conversation


Leave a Reply