Bi3 Solutions empowers SaaS cloud-based Social Media Intelligence Applications with Talend Integration Suite

Talend solves critical data integration challenges for real-time, high-volume social media streams

Based in San Mateo, California, Bi3 Solutions focuses on delivering Software-as-a-Service (SaaS) cloud-based Decision Intelligence applications that help customers manage and dynamically respond to specific business problems using relevant contextual data. The company’s data analytics applications help improve customer service by quantifying customer feedback and making it easier to understand customer sentiment. In addition, Bi3’s tools help customers simplify contract processes and mitigate risk by identifying standards deviations and possible areas of exposure.

Recently, Bi3 has also begun deploying applications that extract content from social media. These applications filter through real-time media streams from sites like Twitter and Facebook, locating and tagging specific topics of interest. That data is then processed continuously to create visual analytics that, much like an EKG, essentially depict the pulse of public opinion. The goal is to help customers gain a real-time understanding of how their brands are faring in the social media realm at any given moment—so they can make smarter, more proactive business decisions.

“But in social media, the volumes being churned out are much larger—almost infinite.”

Transitioning its focus to social media, however, brought new infrastructure challenges. To begin with, there was a drastic increase in the quantity of data to manage. “In the past, we’d always dealt with a limited amount of data flowing through a Customer Relationship Management or other in-house operational system,” says Thomas Groh, co-founder of Bi3 Solutions. “But in social media, the volumes being churned out are much larger—almost infinite.” Developers needed to ensure that their applications had the ability to process and make sense of thousands of tweets and posts per second, and for more than one customer at a time.

In addition, given the real-time flow of social media streams, Bi3 needed an infrastructure that could handle unpredictability dynamically. “We needed a platform that could provide leniency to accommodate the diverse structures of incoming data,” Groh explains. “If a certain type of message comes in and isn’t recognized, for example, we can’t just offload it into an exception record and deal with it later. We’d never catch up. In social media, if you stop listening to the stream, it just keeps building up. So we had to be able to deal with data on the fly using nonstop processes.”

Also critical to the success of its applications was the ability to support a distributed architecture. Because it runs in the cloud and extracts data from multiple sources, Bi3’s system features many individual nodes—all of which have a constant flow of data arriving. If a node were to fail for some reason, the application needed a way to hold on to incoming data during downtime and then continue processing as soon as the node was functional again.

Bi3 selects Talend as leading, mature provider of open source data integration platform

With real-time content analytics as its primary selling point, Bi3 knew that its application infrastructure hinged on a solid data integration platform. “We knew we wanted open source,” Groh recalls. “The space had been maturing for some time, and the solutions we saw were competitive with the big-ticket players.”

“Talend was the most mature product in the space, and provided a lot of room for us to grow our solutions over time.”

From the outset, Bi3 developers were drawn to the Talend Integration Suite over alternatives from other providers such as DataStage and Informatica. “Talend offered the leading data integration capabilities for our SaaS cloud-based Bi3 Solutions application, was the most mature product in the space, and provided a lot of room for us to grow our solutions over time,” says Eric Miles, CEO of Bi3. The company opted to partner with Talend through an OEM license of the Talent Integration Suite Professional Edition, applying the technology at the core of its newly revamped Integrated Decision Optimization Center (IDOC) platform.

Talend Integration Suite meets demands for performance, real-time execution, and reliability

Thanks to its real-time integration and execution capabilities, Talend ensures that Bi3’s Decision Intelligence applications have the performance capacity to handle vast, ongoing data streams at high speeds. “We’re able to perform a lot of parallel processing to cover many, many records across many different nodes,” Groh says.

For example, the Twitter Sentiment Analysis application can comb through every message as it’s generated on Twitter, filter them all based a customer’s desired search terms, and then tag them. “At that point,” he explains, “messages come through the Talend platform with tags attached, and the system looks at the tags and figures out where to deliver each message, which node it goes to, which information it needs to process, how to process it—all in real time. There’s really quite a sophisticated set of logic in there.”

The Talend Integration Suite also helps Bi3 cover its need for fault tolerance. To protect against unforeseen downtime, applications feature input queues on each node where the Talend system delivers data. “If one delivery mechanism goes down for some reason,” Groh says, “then we have a queue that’s building up on the other side. When we’re back up, we simply work through that queue, and we don’t lose a single message. That’s all built-in logic in the Talend tool.”

Scalable Talend platform confidently awaits future growth of Bi3 applications

While Bi3 only recently began rolling out its social media analysis applications, the company is already in high gear, with several customers currently engaged and another 40 on the horizon over the next 12 months.

In addition, the company plans to extend its applications’ social media coverage to include analysis of YouTube content, which—according to Wikipedia—is added at the rate of around 48 hours of video every minute. This further growth in data volume doesn’t give Bi3 executives any pause, however. Says Eric Miles, “We feel very comfortable that as we continue to scale up and add new capabilities, we’ve chosen a platform from Talend that covers us, no matter what’s coming.”