5 Hidden Gems in Talend Spring ’18 (Talend 7.0)
Talend Spring ’18 (Talend 7.0) was Talend’s biggest release yet. With improvements to Cloud, big data, governance and developer productivity, Talend Spring ’18 is everything you need to manage your company’s data lifecycle. For those that missed the live webinar, you can view it here. But as highlighted in the webinar, I realize that everyone is not going to Serverless today, or is looking to ingest streaming data … so what are some of the things in this release that most Talend users will start using immediately? Aka the hidden gems that did not make the headlines.
#1 Hidden Gem – Unified database selection
For this scenario, you wired your database components into an integration pipeline, e.g. Oracle and Salesforce into AWS Redshift, and then a few months later you want to use a different database like Snowflake. But of course, this needs to be done across the many pipelines that you’ve implemented. With Talend Spring ’18, several unified database components, labeled tDB***, have been added into the Talend Studio as an entry point to a variety of databases. Instead of deleting the Redshift component, adding the Snowflake component and configuring the Snowflake connection, you just click on the Redshift component, a drop-down list appears, and you select Snowflake. Big time savings!
#2 Hidden Gem – Smart tMap fuzzy auto-mapping
For this scenario, you are using your favorite component, tMap. You select two data sources to map, and as long as the input and output column names are the same, the auto-mapping works great! But most of the time since different developers created the tables, the column names are different, e.g. First_Name or FirstName or FirstName2 or FirstName_c (custom field in Salesforce). With Talend Spring ’18 and Smart tMap Fuzzy Auto-mapping, tMap uses data quality algorithms (Levenshtein, Jaccard) to do fuzzy matching of the names, saving you time as similarly named columns are matched for you. Just image when dealing with tables with hundreds of columns how much faster this will be!
Input column names
Output column names
Auto-map works Talend 6.5
Auto-map works Talend 7.0
#3 Hidden Gem – Dynamic distribution support for Cloudera CDH
For this scenario, you just installed the latest release of Talend 7.0, which has Cloudera CDH 5.13 support, and you want to use some of the new features in Cloudera CDH 5.14. Bummer! Talend Spring ’18 enables you to quickly access the latest distros without upgrading Talend through a feature called Dynamic Distributions (technical preview). Maven is used to decouple the job from the Big Data Platform version, making it quicker and easier to adopt new Hadoop distributions, so you can switch to the latest release of a new Big Data distribution as soon as it becomes available. Initial support is provided for Cloudera CDH (our R&D team are working on others J )
How does it work? When a new version of Cloudera CDH is released by Cloudera, Talend Studio will automatically list available versions. If you want to use it within your Talend jobs, then Studio will download all the related dependencies. This is an option in the project properties of the Studio (not in TAC or TMC) as this is only needed at design-time.
Not only are you able to use the features more efficiently, this update can also save you days of administration and upgrade time.
#4 Hidden Gem – Continuous Integration updates
For this scenario, your DevOps team operates like a well-oiled machine firing on all cylinders. While projects are getting completed faster, there were some Talend CI/CD command inconsistencies and if you changed a referenced job, you would need to recompile all of the connected jobs. With Talend Spring ’18, the CI features have been rewritten to use Maven standards and CI best practices.
You can now do incremental builds in the Talend Studio, so only the updated job is rebuilt, not all of them. Other features include broader Git support including Bitbucket Server 5.x and Nexus 3 support for the Talend Artifact Repository, standard Maven commands application integration (technical preview), and the ability to easily extend the build process through Maven Plug-ins and custom Project Object Models (POMs). Translation .. we have seen a 50%-time improvement in the build process! How cool is that?
#5 Hidden Gem – Remote Cloud Debug
For this final scenario, we look at the process of debugging Talend Cloud integration jobs. Everything works in Talend Studio, and the next step was to publish to Talend Cloud and test in the QA environment. Talend Spring ’18 provides a free test engine and the ability to remotely debug jobs (data integration or big data) on either Talend Cloud Engines or Remote Engines. The new feature allows Integration Developers to run their pipelines from the studio onto the cloud engine or remote engine and see logs and debug designs locally, which increases productivity by cutting test and debugging from minutes to seconds.
Now that you know all the hidden gems that Talend Spring ’18 can bring you, go put more data to work today. Enjoy!