Why should we care about CI/CD and Containers? Continuous integration, delivery and deployment, known as CI/CD, has become such a critical piece in every successful software project that we cannot deny the benefits it can bring to your project. At the same time, containers are everywhere right now and are very popular among developers. In pr...

Save the date! Talend Connect will be back in London and Paris in October Talend will welcome customers, partners, and influencers to its annual company conference, Talend Connect, taking place in two cities, London and Paris, in October. A must-attend event for business decision makers, CIOs, data scientists, chief architects, and developers, Talend Connect will share innovative approaches to modern cloud and big data challenges,...

Recently I had the chance to talk to an SVP of IT at one of the largest banks in North America about their digital transformation strategy. As we spoke, their approach to big data and digital transformation struck me as they described it as ever evolving. New technologies would come to market which required new pivots and approaches to leverage these capabilities for the business. It is more important than ever to have an agile architecture that can sustain and scale with your data and analyt...

In our last blog, we talked about developing data processing jobs using Apache Beam. This time we are going to talk about one of the most demanded things in modern Big Data world nowadays – processing of Streaming data. The principal difference between Batch ...

After reviewing the history of the OAI and API specifications last year, I wanted to take a look at the future of the OpenAPI Initiative (OAI) and outline my calculated, plausible evolution paths for the project moving forward. State of the OAI Over the past year, we’ve hit several milestones. We have released the OpenAPI Specification (OAS) version 3.0, a major...

Data Quality is often perceived as the sole task of a data engineer. As a matter of fact, nothing could be further from the truth. People close to the business are eager to work and resolve data related issues as they’re the first to be impacted by bad data. But they are often reluctant to update data as Data Quality apps are not really made for them or just because they are not allowed to use them. That’s one of the reasons bad data keeps increasing. According to Gartner, poor data quali...

Log management solutions play a crucial role in an enterprise's layered security framework— without them, firms have little visibility into the actions and events occurring inside their infrastructures that could either lead to data breaches or signify a security compromise in progress. Splunk is the “Google for log files” heavyset enterprise tool that was the first log analysis software and has been the market leader ever since. So lots of customers will be interested in seeing...

It’s an exciting time to be part of the data market.  Never before have we seen so much innovation and change in a market, especially in the areas of cloud, big data, machine learning and real-time data streaming.  With all of this market innovation, we are especially proud that Talend was recognized by Gartner as a leader for the third time in a row in their 2018 Gartner Magic Quadrant for Data Integration Tools and...

So far, our journey on using Apache Spark with Talend has been a fun and exciting one. The first three posts on my series provided an overview of how Talend works with Apache Spark, some similarities between Talend and Spark Submit, the configuration options available for Spark jobs in Talend and how to tune Spark jobs for performance. If you haven’t already read them you should do so before getting started here. Start with “...

This is the second part of my blog series on data preparation and data wrangling best practices. For those of you who are new to this blog, please refer to Part 1 of the same series ‘Data Preparation and Wrangling Best Practices – Part 1’ and for those who are following my blog series, a big thank you for all the support and feed...