REPORT : The Forrester Wave™: Big Data Fabric, Q2 2018

Conducting Effective Talend Job Design Reviews – A Primer

Conducting Effective Talend Job Design Reviews – A Primer

A common practice in any development team is to conduct code reviews, or at least it should be.  This is a process where multiple developers inspect written code and discuss its design, implementation, and structure to increase quality and accuracy.  Whether you subscribe to the notion of formal reviews or a more lightweight method (such as pair programming), code reviews have proven to be effective at finding defects and/or insufficiencies before they hit production.

In addition, code reviews can help ensure teams are following established best practices.  The collaboration can help identify new best practices to follow as well along the way!  Not only that, regular code reviews allow for a level of information sharing and an opportunity for all developers to learn from each other.  This is especially true for the more junior developers, but even senior developers learn a thing or two from this process.

While you aren't writing actual code with Talend (as it is a code generator), developing jobs do share many characteristics with line-by-line coding.  After all, Talend is a very flexible platform that allows developers to build jobs many ways.  All the benefits of code reviews still apply, even if you are only reviewing job designs, settings, and orchestration flow.

The "Why" of Talend Job Reviews

If you had to summarize the goals of doing Talend job reviews into a single word, I would say that it is ‘Quality’. Quality in the tactical sense that your reviewed jobs will probably perform better, will tend to have fewer defects and be likely much easier to maintain over time. Quality in the more strategic sense that over time, these job reviews will naturally improve your developer’s skills and hone best practices such that future jobs also perform better, have fewer defects and are easier to maintain even before hitting job reviews! 

Have I sold you on Talend job design reviews yet?  Attitudes vary on this, after all there is a certain amount of ego involved with jobs that developers build... and that's ok.  It's important however to focus on the positive; treat reviews as an opportunity to learn from each other and to improve everyone's skills.  It's important to always be thoughtful and respectful during the process to the developers involved.  Depending on your team culture, paired reviews may work better than formal full team reviews.  I do recommend meeting face-to-face, however, and not creating offline review processes as there is a lot to be gained by collaborating during these reviews.  Be pragmatic.  Solicit the team for input into how to improve your code review process.

Quantitative & Qualitative

When reviewing Talend jobs, it's important to think both Quantitatively and Qualitatively.

Qualitatively speaking, you want to adapt and adopt best practices.  If you haven't read Talend’s recommended best practices, then I highly recommend doing so.  Currently the four-part Blog series can be found here:

 

In these documented best practices, we discuss some qualities of good job design.  Recommendations on how to make your jobs easy to read, easy to write, and easy to maintain.  In addition, you'll find other foundational precepts to building the best possible jobs.  Consider these:

  • Functionality
  • Reusability
  • Scalability
  • Consistency
  • Performance
  • And several others

Choosing and balancing these precepts are key!

You might also find interesting the follow-on series (2 parts so far) on successful methodologies and actual job design patterns.

I think you’ll find all these blogs worth the read.

Ok, great, now how should we quantitatively review jobs?  In code review projects, often there are metric tools that teams may use to assess the complexity and maintainability of your code.  Did you know that Talend has an audit tool that includes much of this?  You'll find it in the Talend Administration Center (TAC) here:

Here you will find all sorts of really great information on your jobs!  This is a very often overlooked feature of TAC, so be sure to check it out. I recommend reading our Project Audit User Guide.

Talend Project Audit provides several functions for auditing a project through investigating different elements in Jobs designed in a Studio. Talend Project Audit reviews:

  • Degree of difficulty of Jobs and components used in jobs
  • Job error management
  • Documentation and Job versioning
  • Usage of metadata items in Job designs
  • Layout-related issues in Job and subjob graphical designs
  • Job analysis

I've been asked in the past if it is possible to use standard code metric tools with Talend.  The answer is yes, although with some caveats.  Code metrics tools can be run after the code generation phase of the build, so if you could perhaps create a Jenkins job for doing this or inject the metrics right into your standard builds.  It's likely that a complex job in Talend would generate more complex code.  

However, keep in mind that you are maintaining this code on a graphical user interface with prebuilt and reusable components.  The complexity and maintainability metrics are fine to compare between jobs however, it is not always apples-to-apples with hand-coded programs.  For that reason, I generally recommend using our Audit functionality.

7 Best Practice Guidelines for Reviewing Jobs

Now that I've (hopefully) convinced you of the concept of Talend Job reviews, I want to conclude with some best practices and guidelines as you start to put this concept into practice. 

  1. Capture metrics and when you find areas of jobs to change, count them.
    • Use t-shirt size classification.
    • It's important to try and quantify the impact of your review sessions.
    • Track your defects and see if your review efforts are paying off with fewer defects making it past the review process. Capture how long your reviews take.
  2. Don't review too many jobs at one time.
    • In fact, don't even review too much of the same job at one time if it is complex.
    • Take your time to understand the job design and patterns.
  3. Keep your jobs well described and annotated to ease the review process.
  4. Label your components, data flows and subjobs.
  5. Document poor patterns and create a watch list for future job reviews.
  6. Create and share your discovered best practices.
  7. Be sure to include a feedback mechanism built into your process so that the recommended changes and defect resolution can be implemented on schedule.
    • Talend job reviews should be part of your development process, and should not be rushed in as an afterthought.

Above all, keep it positive! Until next time. 

Join The Conversation

0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *