DOWNLOAD : The Cloud Data Integration Checklist from TDWI

Overview: Talend Server Applications with Docker

Overview: Talend Server Applications with Docker

  • Michaël Gainhao
    I've been at Talend since 2009. Previously, I worked as a Professional Services Consultant on various project across the globe with Talend and have worked on all Talend Products (DI/ESB /MDM/Big Data /Cloud). I grow an experience on Cloud and containers. Today, I work as a Customer Success Architect to provide customers guidance on Talend architecture and future-proof implementation.

Talend & Docker

Since the release of Talend 7, a major update in our software, users have been given the ability to build a complete integration flow in a CI/CD pipeline which allows to build Docker images. For more on this feature, I invite you to read the blog written by Thibault Gourdel on Going serverless with Talend through CI/CD and Containers.

Another major update is the support of Docker for server applications like Talend Administration Center (TAC). In this blog, I want to walk you through how to build these images. Remember, if you want to follow along, you can download your free 30-day trial of Talend Cloud here. Let’s get started!

Getting Started: Talend Installation Modes

In Talend provides two different installation modes when working with the subscription version. Once you received your download access to Talend applications, you have a choice:

  • Installation using the Talend Installer: The installer packages all applications and offers an installation wizard to help you through the installation.
  • Manual installation: Each application is available in a separate package. It requires a deeper knowledge of Talend installation, but it provides a lighter way to install especially for containers.

Both are valid choices based on your use case and architecture. For this blog, let’s go with manual installation because we will be able to define an image per application. It will be lighter for container layers and we will avoid overload these layers with unnecessary weight. For more information on Talend installation modes, I recommend you look at Talend documentation Talend Data Fabric Installation Guide for Linux (7.0) and also Architecture of the Talend products.

Container Images: Custom or Generic?

Now that we know a bit more about Talend Installation, we can start thinking about how we will build our container images. There are two directions when you want to containerize an application like Talend.  Going for a custom type image or a generic image:

  • A custom image embeds part of or a full configuration inside the build process. It means that when we will run the container, it will require less parameters than a generic image. The configuration will depend of the level of customization.
  • A generic image does not include specific configuration, it corresponds to a basic installation of the application. The configuration will be loaded at runtime.

To illustrate this, let’s look at an example with Talend Administration Center. Talend Administration Center is a central application in charge of managing users, projects and scheduling. Based on the two approaches for building an image of Talend Administration Centre:

  • A custom image can include:
    • A specific JDBC driver (MySQL, Oracle, SQL Server)
    • Logging configuration: Tomcat logging
    • properties: Talend Administration Centre Configuration
    • properties: Clustering configuration
  • A generic image
    • No configuration
    • Driver and configuration files can be loaded with volumes

The benefits and drawbacks of each approach will depend on your configuration, but :

  • A custom image:
    • Requires less configuration
    • Low to zero external storage required
    • Bigger images: more space required for your registry
  • A generic image
    • Lighter images
    • Reusability
    • Configuration required to run.

Getting Ready to Deploy

Once we have our images, and they are pushed to a registry, we need to deploy them. Of course, we can test them on a single server with a docker run command. But let’s face it, it is not a real-world use case. Today if we want to deploy a container application to on-premise or in the cloud, Kubernetes has become de facto the orchestrator to use. To deploy on Kubernetes, we can go with the standard YAML files or a Helm package. But to give a quick example and a way to test on a local environment, I recommend starting with a docker-compose configuration as in the following example:

 

version: '3.2'

services:

  mysql:

    image: mysql:5.7

    ports:

    - "3306:3306"

    environment:

      MYSQL_ROOT_PASSWORD: talend

      MYSQL_DATABASE: tac

      MYSQL_USER: talend

      MYSQL_PASSWORD: talend123

    volumes:

      - type: volume

        source: mysql-data

        target: /var/lib/mysql

  tac:

    image: mgainhao/tac:7.0.1

    ports:

    - "8080:8080"

    depends_on:

      - mysql

    volumes:

      - type: bind

        source: ./tac/config/configuration.properties

        target: /opt/tac/webapps/org.talend.administrator/WEB-INF/classes/configuration.properties

      - type: bind

        source: ./tac/lib/mysql-connector-java-5.1.46.jar

        target: /opt/tac/lib/mysql-connector-java-5.1.46.jar

volumes:

  mysql-data:

The first MySQL service, creates a database container with one schema and a user tac to access it. For more information about the official MySQL image, please refer to: https://hub.docker.com/_/mysql/

The second service is my Talend Administration Centre image, aka TAC, a simplified version as it uses only the MySQL database. In this case, I have a generic image that is configured when you run the docker-compose stack.  The JDBC driver is loaded with a volume like the configuration.

In a future article, I’ll go in more details on how to build and deploy a Talend stack on Kubernetes. For now, enjoy building with Talend and Docker!

 

 

Join The Conversation

0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *