Data Migration

When upgrading to a new version of a database or application, or when switching to a new system, data needs to be preserved in the new system. The purpose of data migration is to transfer existing data to the new environment. It needs to be transformed to a format suitable for the new system, while preserving the information present in the old system.

Get more information on Talend´s open source solutions for data migration now.

 

Why Data Migration?

Many circumstances exist where an organization needs to migrate applications or perform a data migration on databases. The data migration might be as simple as an upgrade to a new version of the system. Or it might involve switching to a new database or application. After a merger or acquisition, oftentimes redundant applications are retired, but the data they contain needs to be preserved in the surviving system. After an open source data migration, the old system and the new system may also need to co-exist.

 

Challenges of Data Migration

Data migration is often overlooked as a minute aspect of applications or systems migration. There are however numerous challenges for a successful open source data migration.

• Migrations often involve high volumes of data - in some cases, the migration concerns all the transaction history of the organization. The open source data migration involves processing large amounts of individual data.
• By nature, open source migrations happen in heterogeneous environments, with very different source and target data structure. Often, legacy systems are involved. And data structures can be poorly documented. Complex mappings and transformations are required, with aggregations, calculations, etc.
• In many cases consistency needs to be maintained between the old and the new systems, after the data has been migrated. This is for example the case when multiple applications work against the same databases, but they don´t get migrated at the same time. Or when a new system is gradually phased in with users. In these cases, there may be a need to perform complex bi-directional synchronization between the old and the new systems.

 

Open Source Data Integration Solutions for Data Migration

Talend´s open source data integration solutions including Talend Open Studio or Talend Integration Suite are optimized for enterprise-grade data migration. The following features are especially critical to the design, development and execution of open source data migration processes:

• Business-oriented process modeling that involves business stakeholders and ensures proper coordination during the open source migration of business data and processes.
• Fully graphical development environment that improves productivity, makes it easy to perform dry-runs, and allows reusability of data mappings and transformations for synchronization processes (if applicable).
• Highly scalable and fast execution platform with a grid approach that enables the processing of data close to its source and target, using both the open source ETL and ELT approaches for shorter downtimes.
• Broadest connectivity to support all source and target systems.