Open Integration Meets Metadata With The New Talend Metadata Bridge
Open Integration Meets Metadata With The New Talend Metadata Bridge
So, what does it bring to Talend developers, data architects and designers?
Metadata is data about data. Business metadata generally includes information like the definition of business objects (such as a customer), its attributes (for example, a customer ID), the relationships between objects (a contract related to a customer), the business rules that apply to that information (an active customer has at least one open contract), the roles in regards to that information, etc. It brings clarity to information systems, making them more useable and accessible as self-services by business users, and it brings auditability too, a key capability especially needed in heavily regulated industries.
Technical metadata is created by any tool that deals with data: databases, data modelling tools, Business Intelligence tools, development tools, enterprise applications, etc. In fact, metadata is a core capability for solutions and platforms that can bring a high level of abstraction to the IT technical layer, for example, for visual programming or Business Intelligence.
Talend is a perfect example. Metadata is at the cornerstone of our visual design capabilities. So Metadata is not new to Talend. What Talend Metadata Bridges adds is the ability to exchange Talend’s Metadata with Metadata from other tools. In addition, the Excel Data Mapping tool allows for the exposing and authoring of Talend’s data transformation capabilities such as mappings and transformations directly into Excel.
Faster design with the Talend Metadata Bridge
In many organizations, developers, application designers and data architects may not use the same tools when designing, implementing or maintaining systems. Designers may use tools that provide a very high level of abstraction but don’t deal with the technical details: they may use data, objects or process modeling tools, like CA ERwin Data Modeler, Embarcadero ER/Studio, SAP Sybase Power Designer, IBM Infosphere Data Architect, etc. Developers use other tools like a database, an ETL, a Business Intelligence tool, etc. The lack of integration between the tools leads to inefficiencies during the implementation phase.
What Talend Metadata Bridge does is seamlessly integrate Talend with higher-level tools. It can also reverse-engineer existing Talend data jobs into the modeling tools and keep them in sync during the project life cycle. In addition, it not only synchronizes data models with Talend’s physical models, but it also synchronizes metadata across all tools because of its ability to export the metadata across databases and BI tools.
The aforementioned modeling tools are very good at designing and managing data models and data relationships inside a system. However, they don’t provide similar capabilities to manage the relationships between systems, which are the typical problem that you are addressing when you are using Talend. Although Talend Studio provides a high level of abstraction to those data integration processes, some stakeholders involved in the design of a system may still find it too complex for their design job.
This is where our new Excel Bridge for data mapping comes into play. It is an Excel add-in, delivered as part of the Talend Metadata Bridge that allows designing mappings with simple data transformations between data sources and targets, in a simple spreadsheet. Designers will enjoy it for prototyping, documenting, auditing, or applying quick change to the transformation process, directly from the Excel interface they are familiar with. The Excel add-in includes a new “ribbon” with helper functions to format the sheet. It also provides drop-down lists in the cells for easy access to the source or target metadata. Through this new tool, collaboration between the designers and the developer becomes a matter of import-export, eliminating the traditional specifications / implementation / acceptance cycle. The developer deals with the connectivity and other technicalities of the job, while the designer or a subject-matter expert, uses Excel as a frontend to complete the mappings.
So what are the benefits of the Talend Metadata Bridge? It brings reduced implementation times and maintenance costs, increases data quality and compliance through better documentation and information consistency, and improves agility for change.
At the same time, it empowers designers and business users with simple authoring capabilities for mappings and transformations in Talend, accelerates development by using common formats for specifications and development, and avoids delays at runtime for a quick fix in case of unforeseen changes.
Re-platforming and ETL offloading
Data platforms are at the core of any information system. Changing the core in a system often seems as a daunting task, which is a reason why data platforms don’t change much over time. But, there are times when change is needed. It happened twenty years ago when relational databases outperformed their alternatives for information management. It happened more recently, but to a much lesser extent, when came alternatives to the traditional relational databases, such as open source or datawarehouse appliances. And now, as we are engaging the Big Data trend, we see a new generation of open-sourced innovative databases like NoSQL and data management environments such as Hadoop that can manage more volume, variety and velocity of information at a fraction of the cost of their predecessors.
But re-platforming may appear as a risky and costly project that may hamper the benefits of the new technologies. There is a need for accelerators and a well-managed approach to address this challenge. With this respect, Talend Metadata Bridge enhances Talend capabilities to handle migration projects. The bridge’s metadata connectors complement the existing Talend data connectors to automatically create the data structures in the new environment before moving the data itself. It allows therefore renewing your platforms without losing your previous design and implementation investments, and preserves existing development standards such as naming conventions.
When used in conjunction with Talend Big Data, it also dramatically streamlines the offloading of ETL processes to Hadoop: Existing ETL jobs can be converted into native Hadoop jobs that run without even requiring a proprietary run-time engine. In this scenario, the Talend Metadata Bridge can replicate the metadata from the legacy system to the one needed in the new Big Data platform. Note that it is an accelerator, but not a magic box: re-platforming is a project that needs a well-defined approach and methodology. This is something that we are investigating with System Integrators in the context of a new conversion program.
And that’s not all…
Metadata Management holds a lot more promise. Beyond the capabilities mentioned in this article, the Talend Metadata Bridge will drive new best practices in our Talend community, as we envisioned through the feedback of some of the experts that participated in our early adoption program. In addition, the Talend Metadata Bridge provides the foundation for future capabilities within the Talend Unified Platform. Stay tuned...