One of my favorite new TV shows is APB, a story about a billionaire tech entrepreneur who goes into one of Chicago’s most crime-infested districts and changes things around. He roots out inefficient processes and legacy equipment by bringing in state-of-the-art technology such as drones, enhanced body armor, supercharged police cars, and the ability for citizens to call for help using a mobile app.
Something similar is going on today within corporate IT departments (albeit without the Hollywood sensationalism of a TV show) – a change from the ‘old world’ of on-premises IT to the new world of IT in the cloud. The cloud is being used to scale applications like never before and DevOps personnel are extremely important in developing the processes needed to scale these distributed systems.
Using the cloud to develop agile applications and an equally agile analytics backend means moving toward a model of continuous deployment for integration projects. Further, this transition means increasing collaboration between ETL developers and DevOps. DevOps is a portmanteau of “Developer” and “Operations” which implicitly states that no longer can developers simply check in code, and leave the testing and deployment to a separate operations team – they must be responsible for all their code from development to production. It is this discipline that ETL developers also need to adapt for integration projects, especially those involving cloud data sources. Managing source code, continuous deployment, and testing procedures become more important as one scales to hundreds of nodes across several applications and services that all need to be secured, and updated constantly.
In the race to acquire new customers, many companies are trying to build specialized analytics stacks to gain insights about every facet of their interaction with customers. From the very first touch, to the moment revenue comes in, and beyond, these analytics stacks are built into every cloud-native, customer facing application. To build these specialized analytics stacks requires a combination of real-time ingestion technologies, a myriad of open-source big data capabilities, and lots of readily available cloud infrastructure to run intensive workloads such as indexing, structured queries, machine learning, and more. The proliferation of these types of analytics projects stems from the need of line-of-business (LOB) users seeking direct access to accurate pools of data so they can perform detailed analyses, which requires flexible data structures.
However, most companies do not have a hundred percent of their operations running in the cloud, and must bridge legacy IT assets with the new world of cloud-enabled applications and this is where integration comes in, specifically cloud integration. Many LOB users are already attuned with using out-of-the-box cloud applications such as Salesforce that generate a ton of data that needs to be analyzed.
As William Fellows and Carl Brooks of the 451 research state, “Companies that want to do cloud must embrace DevOps. This is the process; it is here that applications and infrastructure are abstracted in such a way that it becomes possible to deliver continuous improvement – with automation and self-service – at speed. Removing barriers to scaling applications can allow product and marketing teams to take calculated risks and reach new markets before opportunities are lost.”
Talend has been actively working to ensure its products allow DevOps practices to be adopted by ETL developers when it comes to integration projects. Last September, we introduced the Big Data Sandbox on Docker which consisted of preconfigured containers of Talend components for Spark, Hadoop, and Kafka. Then in late January we added new advanced SDLC capabilities to Talend Integration Cloud that allow different environments for development, testing, and production, separate security privileges for each environment, and the ability to promote artifacts from one environment to another. These separation of environments allow ETL developers to push through integration projects in an agile manner, similar to the practices adopted by DevOps personnel.
Like the show APB, it’s not just about throwing a bunch of the latest and greatest cloud and big data technologies at a problem and hoping it solves the issue. It’s really about combining these technologies with innovative processes and the right skillsets to handle each process that results in success. In a world where using data in the right way can provide a competitive advantage, DevOps and cloud integration are the ingredients that will be key to this success.