Data needs to be preserved when upgrading to a new version of a database, or when switching to a new version of an application. Oftentimes after a merger or acquisition, redundant applications are retired, but the data they contain needs to be preserved in the surviving system. The purpose of data migration is to transfer existing data to the new environment. Data needs to be transformed to a format suitable for the new system, while preserving the information present in the old system.
Obstacles: Managing Diverse Data
Data migration is often overlooked as a minute aspect of application or system migration; however, there are numerous challenges for successful data migration.
- Database migrations often involve high volumes of data - in some cases, the migration contains all the transaction history of the organization.
- By nature, migrations happen across heterogeneous environments with very different source and target data structures. Many times legacy systems are involved, data structures are poorly documented and complex mappings and transformations are required.
- Data consistency needs to be maintained between the old and the new systems. For example, when migrating multiple applications over different times that use the same database, or when a new system is gradually phased in with users. In these cases, there may be a need to perform complex bi-directional synchronization between the old and the new systems.
Solution: Talend for Data Migration
Talend's data integration solutions are optimized for enterprise-grade data migration. The following features are especially critical to the design, development and execution of data migration processes:
- A highly scalable and fast execution platform with a grid approach that enables the processing of data close to its source and target, using both the ETL and ELT approaches for shorter downtimes.
- Broadest connectivity to support over 800 source and target systems.
- Business-oriented process modeling that involves business stakeholders and ensures proper coordination during the migration of business data and processes.
- Fully graphical development environment that improves productivity, makes it easy to perform ‘dry-runs’, and allows reusability of data mappings and transformations for synchronization processes (if applicable).
- Task scheduling allows DevOps to manage the process of migration across a wide set of systems, including error detection, error handling and notification, depending on the result of the migration for a given source system.
Talend provides the following products to streamline database migration tasks.
Talend Data Integration
Talend provides an extensible and highly-scalable set of data integration tools to access, transform and migrate data from any business system. With support for over 800 types of data sources, Talend simplifies your database migration needs.
Talend Data Quality
Talend provides a powerful open source-based data quality solution that delivers end-to-end profiling, cleansing, matching and monitoring capabilities with the ability to identify anomalies, standardize data, resolve duplicates and monitor data quality over time. Data consistency is improved as you migrate to new systems.
Unlike other solutions where you need to integrate products to make a solution, Talend’s products improve your productivity through a unified platform - a common code repository and tooling for scheduling, metadata management, data processing and service enablement.
Talend Data Management
Talend Data Management turns disparate, duplicate sources of data into trusted stores of consolidated information, so your business can be more responsive and confident in daily decisions.