DOWNLOAD : The Cloud Data Integration Checklist from TDWI

4 Ways You Should be Using the Talend tMap Component

4 Ways You Should be Using the Talend tMap Component

  • Mark Balkenende
    Mark Balkenende is a Sales Solution Architects Manager at Talend. Prior to joining Talend, Mark has had a long career of mastering and integrating data at a number of companies, including Motorola, Abbott Labs and Walgreens. Mark holds an Information Systems Management degree and is also an extreme cycling enthusiast.

At Talend one of my “Shadow IT” jobs is reporting on the component usage. If you have ever used Talend Studio (either the open source Talend Open Studio or the commercial version you most likely know the component tMap

It is the most used component by a long shot. Why? Simply put, it’s because it is extremely versatile and useful.  When I was first asked to write this article I thought, “Why not, this will be easy.”  But, as I started actually going about picking out use cases I quickly realized that there are so many more than just 4 to choose from! I actually challenge everyone to respond and tell me below what you think the top features are of the tMap component. I would love to hear from you.

I will start out by listing the most obvious but most needed and then get into some more advanced uses.  (I will try and sneak a couple in together so I can have more than 4, hopefully, our editor doesn't catch it.)

Editors note: I did catch it, but that’s just fine Mark.

#1 Mapping, of course. 

The tMap's most basic use is to map inputs to outputs.  This can be as simple as Source fields to Target fields of the data integration job.  It can also be from some other input components like aggregators, matching or data quality components. 

With the tMap you can also limit the fields mapped from left to right, basically filtering unneeded columns.  You can create new columns coming out of the tMap, say for example adding sequence keys, or concatenating multiple input columns into a new column, for example address fields into one column to make a single mailing data field.   This leads into the next big use case I want to cover….

#2 Expression Builder. 

Within the tMap on any column or variable you can open the expression builder wizard where you get access to hundreds of Talend functions, and if you can't find a Talend function that meets your needs then you can fall back on a native Java function (don't fear, if you don't know Java just Google it).

If you happen to know some Java, you can easily build custom Java routines which will then be available within the expression builder.   Also, expression builder allows you do complex math on multiple fields if needed. You can extract parts of dates, do data conversions, even case statements to build in “what if” logic.  The same conditional statements can be used to determine if a row should pass through the tMap at all, acting as a filter.  As you can see with the tMap Expression Builder you get great transformation powers.

#3 Lookups. 

The tMap is where you can do what many Data Integration specialists refer to as “lookups” on data. For those unfamiliar, this is basically joining data from one source to another source.  The tMap has a lot of functionality on lookups, like inner join or outer join, reject if join is not found, cache the lookup data and much more.  Lookups are critical to data transformation process as you often need to pull in reference data or get expanded views of records. 

With Talend the lookup source can be anything that can be sourced into a Talend job.  This has almost endless possibilities.  To illustrate, let’s imagine a multi-cloud scenario quickly. Let’s say you have customer data in AWS on S3 and some other critical data on Azure Blob storage. in a single Talend job using tMap and lookups you can easily join the two sources together and write your data anywhere you need, like say Google BigQuery just to be crazy!

#4 Route Multiple Outputs. 

The tMap can only have one input (not counting Lookups) but you can have multiple outputs with any number of columns as outputs on each stream. 

This becomes a fast and powerful way to route errors down a different flow or to just duplicate the data flows to different streams one flow can go to an aggregation component while another output could go direct to your target outputs and a third have a conditional statement looking for errors.  All this becomes extremely useful as your data flows become complex with multiple outputs, error processing and conditional outputs.

Conclusion

There you have it, a quick and (hopefully) helpful introduction to our most popular component in Talend. If you want some more tMap knowledge, let me know in the comments below and I’ll spin up my next article around some more advanced mapping functionalities. Happy connecting!

 

Join The Conversation

0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *