I first want to start by thanking all the readers of my previous 2 blogs on the topic of Talend and Apache Spark. If you are new to this blog series and haven’t read my previous posts, you can start reading them here “Talend & Apache Spark: A Technical Primer” and part two here “...

[style-full-width-content type="video" url="https://youtube.com/embed/4BNhOFqxw58"]  For this episode of Craft Beer & Data, Mark and I hung out with Eric Matelski, tap-room manager at the Epic Brewing Company, and talked about how big data is transforming the agricultural industry. But first, Eric told us about one of his favorite beers (lately), Falling Monk: “Thi...

Apache Spark’s Resilient Distributed Datasets (RDD) are a collection of various data that are so big in size, that they cannot fit into a single node and should be partitioned across various nodes. Apache Spark automatically partitions RDDs and distributes the partitions across different nodes. They are evaluated lazily (i.e, the execution will not start until an action is triggered which increases manageability, saves computation and thus increase...

In early 2016, the International Consortium of Investigative Journalists (ICIJ) published the Panama Papers –one of the biggest tax-related data lea...

Recently, I've seen a lot of articles and conversation around DevOps. DevOps is a set of practices that automates the processes between software development and IT teams, in order to enable be shorter development cycles, increased deployment frequency, and more dependable releases all in conjunction with business needs. Companies are increasingly adopting DevOps practices utilizing agile methodologies along with automat...

A few years ago, Starbucks’ director of analytics and business intelligence, Joe LaCugna, said the Seattle coffee giant once struggled to make sense of the data pouring in from its loyalty card holders, which at the time was over 13 million and comprise 36 percent of all Starbucks’ transactions. The same was true of the coffee conglomerate’s social media data—they have mountains of it, but still can’t quite figure out what to do with it, according to Mr. LaCug...

When it comes to solutions for the big data sector, there is a clear split between the legacy and next-generation approaches to software development.  Legacy vendors in this space generally have their own large internal development organizations, dedicated to building proprietary, bespoke software. It’s an approach that has worked well over the years. However, the big data market has always moved at lightning speed, a...

When I graduated from college in the late 1990s, it was just in time to enjoy the Y2K crisis. If you remember those fun times, then you are old enough to enjoy this blog.  I graduated with a Management Information Systems (MIS) degree, which is a cross between Computer Science (CS) and Business Management, and although I was stronger in CS than Business Management, I survived.  There was a class spanning both disciplines that I partially excelled in called Database Theory, wh...

These days, it seems like every city is trying to implement “smart city” initiatives. Take Singapore for example, now known for the most extensive effort to collect data on citizens’ daily living h...

More and more companies around the globe are realizing that big data and deeper analytics can help improve their revenue and profitability. As such, they are building data lakes using new big data technologies and tools, so they can answer questions such as: How do we increase production while maintaining costs? How do we improve customer intimacy and share of wallet? What new business opportunities should we pursue? Big data is playing a major role in digital transformation pr...