As the CMO of a data integration software company, I’ve invested a lot of time convincing IT managers that they should switch from hand coding to a tool-based approach. Hence the reason I was excited to see and impressed by the practical advice outlined in Gartner’s recently published report, “Does Custom-Coded Data Integration Stack Up to Tools?”
If you are a customer working with new cloud and big data technologies, Gartner’s report offers several great points to contemplate, but there are also other questions you should take into consideration before initiating any custom coded projects. In this blog, I’ve summarized my key takeaways from Gartner’s research and translated them into a checklist of questions every IT manager should ask themselves as they evaluate the trade-offs between hand coding vs. a tool-based approach to data integration.
My key takeaways from “Does Custom-Coded Data Integration Stack Up to Tools?”
· Be sure to look at both short and long term costs: While your deployment costs may be reduced by 20 percent with a custom coded approach, the maintenance costs will increase by 200 percent.
· There is a time and place for hand coding, but only in very specific situations: Custom coding can make sense for very targeted, simple projects that will not require a lot of maintenance. It could also be necessary for situations where there are no tools capable of doing the work required. Additionally, only 11 percent of the organizations surveyed by Gartner were pursuing custom development.
· Data integration projects requiring multiple developers will benefit from the visual design environments provided by tools. This will make it easier to re-use prior development elements and can lead to more efficient data integration flows. This is because a single job can feed multiple targets instead of having a series of data integration flows all doing roughly the same thing over and over.
· It’s important to understand the maintenance and support costs that will go along with any projects. If different people maintain and support the code once it’s in production, their learning curve with a hand-coded approach will be high, and if the code is in production for years, turnover will lead to far higher costs in the long run.
Given the above points, I’ve come up with the checklist of questions below to help IT managers make a more informed decision about which approach to take.
The Hand Coding Checklist:
1. Does my development team have the expertise to do this using hand coding?
If you’re using a new technology like Hadoop or cloud platform, who is going to do the work and how much ramp time will they need?
2. Is this what I want my hand coding experts spending time on?
Hand coding experts are typically about a quarter of the full development team, making them a scarce resource. If a non-expert could do the same work using a tool—and save hours of time doing so—wouldn’t you rather have the experts doing something where their unique skills are required?
3. Can I do this same work with a tool cheaper and faster than my team can hand code it?
Most IT teams are constantly being asked to do more with less. A tool-based approach often allows a lower cost per developer to do the work, and accomplish it quickly.
4. Is this a one-off, stand-alone project or is this an area where I plan to continue doing more and more development over time?
If you are embarking on an initiative using a big data or cloud platform, chances are you are going to want to do more and more on that platform over time. If so, relying on expert hand coders will be a very hard approach to scale given the scarcity of these resources.
5. How portable will this code be if I want to repurpose it on a new technology platform like Spark or Flink?
Portability is an easily overlooked point. The Apache Hadoop ecosystem is moving incredibly quickly. For example, in 2014 and 2015, MapReduce was the standard, but by the end of 2016 Spark emerged as a new standard. If you’ve taken a hand-coding approach, it was impossible to port that code from MapReduce to Spark. Leading Data Integration tools allow you to do this, eliminating legacy code situations.
6. Will multiple developers be collaborating on this project?
As covered in the summary above, there are multiple benefits that come from a tools-based approach when you have multiple developers working together including easy reuse and code sharing, visual design environments, even wizards and experts to advise the developer.
7. How long will this code be in production?
When embarking on a new project, it’s tempting to focus on the time needed to develop and forget how long something will be in production. Often something that takes six months to develop will be in production for five years or longer. If that is the case, the support and maintenance costs of that code will continue for 10X longer than the initial development work making it critical that you understand your support and maintenance costs.
8. Who will own the maintenance of this code?
If you only have a handful of Spark developers, then they will be the ones forced to maintain and support their code. Eventually, support and maintenance will consume all of their capacity, making it impossible for them to take on new projects that could potentially be tasks that help your organization gain a competitive edge.
9. How often will the code need to be updated to accommodate new business needs or changes in the data sources or targets?
Data sources, targets, and business needs are constantly evolving. If it’s reasonable to expect this constant stream of changes, then the cost of maintenance and support will be significantly higher.