For me, the most exciting thing about Pipeline Designer is the way that it makes working with streaming data easy. Traditionally this has required a completely different way of thinking if you have come from a "batch" world. So when Pipeline Designer was released, the first thing I wanted to do was to find a good streaming data source and do something fun and interesting with the data. Twitter was my first choice of streaming data. The data is easy to acquire, constantly being produce...

I have had the privilege of playing with and following the progress of Pipeline Designer for a while now. I am really excited about this new tool. If you haven’t seen it yet, then don’t delay and get your free trial now…..actually, maybe read this blog first ;-)  Pipeline Designer is an incredibly intuitive, web-based, batch and stream processing integration tool. The real geniu...

When working with Talend Pipeline Designer, you may come across situations where you need to write an Avro schema. Go to the Apache AVRO site to read more about this serialization technology. This blog has come about thanks to some experiences I’ve had when working with Avro schemas and getting them to work when serializing using Python and Java. Spoiler alert!....they don’t necessarily work the same w...

I am very excited to introduce Pipeline Designer, a next-generation cloud data integration design environment that enables developers to develop and deploy data pipelines in minutes, design seamlessly across batch and streaming use cases, and scale natively with the latest hybrid and multi-cloud technologies  <<...

Talend Cloud is an Integration platform as a service (iPaaS) offering by Talend. It is a fully managed cloud option which has the capabilities of data integration, data stewardship, data preparation, API designer and Tester and Pipeline designer. These tools can be used for lightweight ETL and detecting the schema on the fly. One of the unique features of Talend Cloud is it provides both on premise and cloud execution environments. Remote engines can be installed behind the firewall, which al...

Introduction Thanks for continuing to read all of our streaming data use cases during my exploration of Pipeline Designer. For the last article of this series, I wanted to walk you through a complete IoT integration scenario using a low consumption device and leveraging only cloud services. In my previous posts, I’ve used a Raspberry Pi and some sensors as my main devices. This single board computer is pretty powerful and you can install a light version o...

Introduction Thanks for continuing to read all of our streaming data use cases during my exploration of Talend Data Streams. For the last article of this series, I wanted to walk you through a complete IoT integration scenario using a low consumption device and leveraging only cloud services. In my previous posts, I’ve used a Raspberry Pi and some sensors as my main devices. This sing...

In our last blog, we talked about developing data processing jobs using Apache Beam. This time we are going to talk about one of the most demanded things in modern Big Data world nowadays — processing of Streaming data. The principal difference between Batch and Streaming i...

In our last blog, we talked about developing data processing jobs using Apache Beam. This time we are going to talk about one of the most demanded things in modern Big Data world nowadays – processing of Streaming data. The principal difference between Batch ...

In the last few years, Python has become the go-to programming language for Data Scientists. In a way, this is kind of surprising. Python was not originally developed to do analytical tasks or data science, but it has evolved to become the ‘Swiss-army’ tool in the Data Scientist toolbox. The reason for this comes from a large number of third-party packages avai...