VIRTUAL TRAINING : Step-by-Step to Big Data Integration

Automation Made Easy: Building Jenkins Jobs with Jenkins DSL

Automation Made Easy: Building Jenkins Jobs with Jenkins DSL

  • Lokesh Poojari Gangadharaiah
    Lokesh Poojari G is a Customer Success Architect at Talend. His field of expertise include Integration technologies , Project management & other programming languages. He has over 15 yrs work experience spanning organizations like Accenture, Thomson Financials, MetricStream etc in GRC, Telecom, Investments, Mobility & Media domains. He is Certified Scrum Master and has worked in the roles Senior Solution Architect, Delivery Manager, Product Manager & Business Analyst for various IT programs prior to joining Talend.
  • April 24, 2018

Recently, I've seen a lot of articles and conversation around DevOps. DevOps is a set of practices that automates the processes between software development and IT teams, in order to enable be shorter development cycles, increased deployment frequency, and more dependable releases all in conjunction with business needs.

Companies are increasingly adopting DevOps practices utilizing agile methodologies along with automation tools like Jenkins, Bamboo, TeamCity etc. Companies that have started using Jenkins for setting up continuous integration, delivery and deployment usually build their jobs using the Jenkins UI. They initially love it because it is a wonderful system for managing builds, but as diverse groups within the company start adopting this method, they are faced with some problems like:

  • Maintenance suffering due to scale of the number of Jenkins jobs
  • Inconsistency of jobs due to copy/paste & unintended divergence
  • Reusability and complexity
  • No backup of jobs
  • No disaster recovery methods

The Jenkins job-dsl-plugin attempts to solve these problems by allowing jobs to be defined with the absolute minimum effort in a programmatic format. We will look at a sample script to achieve continuous deployment in the Talend Administration Center using the Jenkins DSL plugin.

Creation of Continuous Delivery Job in Jenkins

The Jenkins "Job DSL / Plugin" is made up of two parts: The Domain Specific Language (DSL) itself that allows users to describe jobs using a Groovy-based language, and a Jenkins plugin which manages the updates and ultimately the Jenkins jobs which are created and maintained as a result.

Step 1:

Assuming you have installed the DSL plugin from the “Manage Plugins” section, the script for a continuous delivery job to automatically create tasks and deploy in the Talend Administration Center (TAC) looks something like this:

Depending on the automation required, this can change to any specifc Software Development Lifecycle (SDLC) requirements set up within each company. In the above script, apart from setting up the configuration for GIT, TAC and your respective credentials, you can pass the Groovy Script through a managed files plugin that is needed for:

  • Parsing the JSON request file (task_list.json)
  • Calling the MetaServlet API to create tasks in TAC through the BASE64 encoded message
  • Deploying the jobs in the JobServer along with creating the optional triggers

 Step 2:

Next, create your Jenkins job using the freestyle project style to run your DSL scripts. This is usually termed as SEED job which uses the script that was created above.

Step 3:

Run the above SEED Job to create the Jenkins Job needed for continuous delivery as shown below.

At the end this process, you will have the job needed to create Talend tasks automatically in TAC and deploy them to JobServers.

Conclusion:

The Jenkins DSL plugin provides maintainability to enterprise scale environments. You can further enhance the design by storing the SEED scripts in GIT or any SCM tools and execute it so that any changes made to the job can be tracked with history. This way even if you lose your Jenkins job you can recreate the same from GIT. Finally, the DSL plugin allows you to build script templates around regular design patterns so that there is consistency in the way teams build or deploy their Jobs. As always, comments and feedback are appreciated. Let me know what you think! Until next time. 

Join The Conversation

1 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

  1. Deepali Asodekar says:

    I want to connect Talend to Einstein Analytics (Salesforce). Could you please help me for this.