Between 2 Bits: Processing your largest JSON files with ease

By Talend Team

We know - this one's a little different than our other Jobs of the Week (someone please give us a new title). But consider how much data the average company collects! Enterprise companies use an average of 150 software applications. That's a LOT of data to manage, move, and use. In order to do so, you need to be able to move your ever-growing volumes of data efficiently. Richard to the rescue!

Divide and conquer your parsing problems

Dealing with massive JSON files can prove tricky when you're talking several hundred of gigabytes of data. Loading complete JSON files can use too much memory, leading to slowing, crashing, or worse - the dreaded java.lang.outofmemoryerror. The JSON needs to be processed completely into memory before it can be processed using out-of-the-box tools. Thankfully, you don't have worry about these restrictions - we found a workaround to divide and conquer your parsing problems.

With Talend we're free to create our own mechanisms to workaround this hurdle


As always, download the job here (you're welcome!) and watch the Job of the Week below for a step-by-step:

Please enable cookies to access this video content.

This week, Jason Cruz Falzon, Joseph El Khoury, Justyn Davidson, and Richard Hall explain what exactly makes them qualified to talk about data and technology. It’s been a long and winding journey for all of them, from secrets, crimes, and assassinations to lifeguards and singing chefs. You won’t want to miss this one