Talend & Docker
Since the release of Talend 7, a major update in our software, users have been given the ability to build a complete integration flow in a CI/CD pipeline which allows to build Docker images. For more on this feature, I invite you to read the blog written by Thibault Gourdel on Going serverless with Talend through CI/CD and Containers.
Another major update is the support of Docker for server applications like Talend Administration Center (TAC). In this blog, I want to walk you through how to build these images. Remember, if you want to follow along, you can start your free trial of Talend Cloud here. Let’s get started!
Getting Started: Talend Installation Modes
In Talend provides two different installation modes when working with the subscription version. Once you received your download access to Talend applications, you have a choice:
- Installation using the Talend Installer: The installer packages all applications and offers an installation wizard to help you through the installation.
- Manual installation: Each application is available in a separate package. It requires a deeper knowledge of Talend installation, but it provides a lighter way to install especially for containers.
Both are valid choices based on your use case and architecture. For this blog, let’s go with manual installation because we will be able to define an image per application. It will be lighter for container layers and we will avoid overload these layers with unnecessary weight. For more information on Talend installation modes, I recommend you look at Talend documentation Talend Data Fabric Installation Guide for Linux (7.0) and also Architecture of the Talend products.
Container Images: Custom or Generic?
Now that we know a bit more about Talend Installation, we can start thinking about how we will build our container images. There are two directions when you want to containerize an application like Talend. Going for a custom type image or a generic image:
- A custom image embeds part of or a full configuration inside the build process. It means that when we will run the container, it will require less parameters than a generic image. The configuration will depend of the level of customization.
- A generic image does not include specific configuration, it corresponds to a basic installation of the application. The configuration will be loaded at runtime.
To illustrate this, let’s look at an example with Talend Administration Center. Talend Administration Center is a central application in charge of managing users, projects and scheduling. Based on the two approaches for building an image of Talend Administration Centre:
- A custom image can include:
- A specific JDBC driver (MySQL, Oracle, SQL Server)
- Logging configuration: Tomcat logging
- properties: Talend Administration Centre Configuration
- properties: Clustering configuration
- A generic image
- No configuration
- Driver and configuration files can be loaded with volumes
The benefits and drawbacks of each approach will depend on your configuration, but :
- A custom image:
- Requires less configuration
- Low to zero external storage required
- Bigger images: more space required for your registry
- A generic image
- Lighter images
- Configuration required to run.
Getting Ready to Deploy
Once we have our images, and they are pushed to a registry, we need to deploy them. Of course, we can test them on a single server with a docker run command. But let’s face it, it is not a real-world use case. Today if we want to deploy a container application to on-premise or in the cloud, Kubernetes has become de facto the orchestrator to use. To deploy on Kubernetes, we can go with the standard YAML files or a Helm package. But to give a quick example and a way to test on a local environment, I recommend starting with a docker-compose configuration as in the following example:
version: '3.2' services: mysql: image: mysql:5.7 ports: - "3306:3306" environment: MYSQL_ROOT_PASSWORD: talend MYSQL_DATABASE: tac MYSQL_USER: talend MYSQL_PASSWORD: talend123 volumes: - type: volume source: mysql-data target: /var/lib/mysql tac: image: mgainhao/tac:7.0.1 ports: - "8080:8080" depends_on: - mysql volumes: - type: bind source: ./tac/config/configuration.properties target: /opt/tac/webapps/org.talend.administrator/WEB-INF/classes/configuration.properties - type: bind source: ./tac/lib/mysql-connector-java-5.1.46.jar target: /opt/tac/lib/mysql-connector-java-5.1.46.jar volumes: mysql-data:
The first MySQL service, creates a database container with one schema and a user tac to access it. For more information about the official MySQL image, please refer to: https://hub.docker.com/_/mysql/
The second service is my Talend Administration Centre image, aka TAC, a simplified version as it uses only the MySQL database. In this case, I have a generic image that is configured when you run the docker-compose stack. The JDBC driver is loaded with a volume like the configuration.
In a future article, I’ll go in more details on how to build and deploy a Talend stack on Kubernetes. For now, enjoy building with Talend and Docker!