Big Data, Soon Shooting 100x Faster

This week our partner Hortonworks announced a whole set of initiatives and projects to make Hadoop more enterprise-ready. As I have written in the past, the Hadoop distribution vendors (Hortonworks, Cloudera, etc.) are playing a key role in the deployment of Hadoop, making it “enterprise-acceptable”.

Among the initiatives announced this week, Talend is very interested in The Stinger Initiative.  I have no idea whether this name is derived from the namesake portable surface-to-air missile (which, according to Google, can also be mounted on the Apache helicopter - or used to shoot it down), or from the appendix that insects use to deliver venom.  But anyway, it’s a neat name.

There is an area where Stinger will be very helpful for Talend. Our big data profiling capabilities, introduced with v5.2, rely on Hive for in-Hadoop profiling. The clear value is that you don’t need to extract (or even to re-map) the data before profiling it – you rather use the massively parallel processing power of Hadoop. The drawback of this approach so far was the latency of Hive, originally built for large-scale operational batch processing.

With Stinger, Hive will also be optimized for super-fast interactive queries, which will provide sub-second response times for profiling jobs. In am sure the community will appreciate.

The beauty of Stinger is that it is enhancing an already-existing platform, instead of creating a new one. Other interactive query projects have been launched for Hadoop, but they require that data be re-mapped, they may or may not be integrated with HCatalog… why reinvent the wheel when you can improve the existing wheel?

This is another proof point of the power of an open source community. Because our big data profiling features rely on Hive, our product will benefit from these enhancements without having to re-architect the solution.  And because several vendors and users are working together on this project, innovation goes faster than in the proprietary world.  And here comes the Stinger missile – mounted on the Apache helicopter, not used to shoot it down.  Pun intended.

Yves