As Talend has released its newest version of Talend Data Fabric 6.2.1 for GA, I thought this would be a good time to talk about software upgrades within the Enterprise.
Software maintenance is a big part of IT budgets. As more companies offer software for all kinds of needs from instant messaging to analytics, maintaining the software can easily become a big cost factor. This is because software upgrades takes time and resources. I have been part of projects where it took years for the upgrades to be completed enterprise wide. In addition, multiple groups have their own timelines around releases which adds more complexity to the upgrade process. The activities also takes away valuable time from the developers. But it is essential for firms to keep their software systems up to date as the new releases of software can perform better, include new features which provide more value to the firm.
Below I provide my recommendations to upgrade your enterprise software. These practical methods reduce the risk of something going wrong when you are installing a completely new version of the software, upgrading the metadata and then migrating the actual data.
1. Never Upgrade in Place
My recommendation is to have two environments, one is your current system and other, a new installation, running in parallel. This may be obvious but not surprisingly too many organizations do not make the time or have the infrastructure to support two environments running at the same time. Having a separate installation for the new software version provides a fail-safe method in case something goes wrong during the upgrade process. Users would not be impacted and would be able to confidently test all the features required without any impact to the current environment. Having the flexibility to prove the new environment before switching over is a big plus and can save time and money.
For Talend Data Fabric, it is highly recommended that the server software elements are installed in a separate environment. It is best to install on completely new severs or VM’s and take advantage of any upgraded infrastructure that may be required or recommended. While it is also possible to install the new version of Talend on the existing infrastructure it can be tricky and system administrators need to ensure that different ports are used for web servers and that servers are not installed as services.
2. Always Take Backups
If the software comes with a native backup utility, use it back up all critical systems. These can include metadata databases, application backups, storage backups, code backups, etc. This provides a way to restore the systems if the upgrade did not work. So do we need to backup even if we plan to have parallel environments running? Absolutely!
If the upgrade spans a long period of time you need to take frequent backups so that no work is lost. This doesn’t need to be just a System administrator task. Developers can help by taking their own backups frequently while the upgrade is being done.
For Talend Data Fabric, the TAC database, the SVN repo or the GIT repo, and also the archive folder which stores the job binaries would need to be backed up. If MDM is being used, all the MDM systems database and the master databases would need to be backed up.
3. Check Compatibility
Software compatibility is a critical component of the upgrade process. Therefore, before upgrading, the new version needs to be checked to ensure that it can operate with other peer and dependent software within the enterprise. If any software is not compatible or supported, a decision needs to be made if that software needs an upgrade of its own. It is not uncommon for IT to be managing several upgrades simultaneously to ensure compatibility between differing vendor’s software.
For Talend Data Fabric, we would need to make sure the Java version (the big data distribution version) source and target database versions and the infrastructure OS version are compatible to the Talend version.
4. Always leverage vendor specific migration tools
Wherever possible, tools supplied by the software vendor will need to be used for the installation and upgrade processes.
Talend’s universal installer can install individual components on multiple servers. Moreover, The Talend Administration Center (TAC) makes it easy to upgrade Talend’s metadata repository from one release to the next. For MDM, Talend provides the data migration tool to migrate the models and the data from the old master database to the new database. These tools come out of the box and should be used as they automate some of the upgrade processes.
5. Plan, Test and Execute
Upgrading enterprise software can be complex and requires much process and planning; it needs to be treated and managed as a project.A project manager would be required to get business and technical buy in, plan the different activities from DEV installation, testing and cutover and execute the activities by having regular status checks. Resources would need to be assigned to do regression testing particularly those programs which touch other systems, which are critical to the business operation and which are complex in terms of logic and application.
Performance testing needs to be done as the new features of the new version might actually degrade the performance if the configuration is not done correctly. This would need configuration updates and these changes would need to be tracked by the project management team. The cutover needs to be carefully planned and executed as well. The software needs to be installed prior to the cutover so that during cutover it would only be a switch between the two versions of the software.
Upgrading enterprise software is often quite complex. There are many ways in which problems can arise making the process extremely challenging. In this blog post I introduced five key facets of a software upgrade process. These facets define a framework that is applicable to software upgrades of all shapes and sizes.
This framework, when leveraged for your next upgrade will help ensure that the process goes smoothly, is reproducible, and is a success.