SupplyFrame Automates Catalog Building and Updates

Content and advertising network for the industrial electronics industry leverages Talend Data Integration to automate product updates from multiple sources.
The time savings are very apparent. What used to take us two weeks to develop, now takes less than a day - that's a 90% savings, not only in development costs but also in making new sources of information available.
Andrew Brown, Vice President of Content

Leading online component information service - electronics industry

Supporting both product manufacturers and industry advertisers, SupplyFrame provides a powerful, trusted search engine enabling designers and procurement professionals to find technical information on millions of electronic components from thousands of manufacturers. SupplyFrame marketing tools allow advertisers to engage design engineers and procurement professionals in a value-added format by delivering relevant, contextual advertising at the moment components are being researched. Its unique ability to support new product introductions, upselling, campaign promotions, lead generation, and e-commerce have made SupplyFrame the fastest growing pay-for-performance marketing system in the industry.

Technical content, including component datasheets, specs, sources, and cross-references, is provided directly from manufacturers and their franchised distributors. All use of the data has been authorized, and since it is free to provide data to SupplyFrame, all manufacturers maintain equal representation in searches.

Diverse and complex data collection

In order to load and maintain its database of electronic components, SupplyFrame has to deal with collecting and aggregating product information from a large number of manufacturers and distributors. The SupplyFrame catalog scales to information on several million distinct components, provided by thousands of different sources.

Some of the components manufacturers provide data via pro-active data feeds - files dropped on a server, daily or on a predetermined frequency. Others make data available on their own FTP site, so SupplyFrame has to go and retrieve it. Some send Excel spreadsheets via e-mail. And still others lack the bandwidth or the capability to provide data; SupplyFrame has to collect the relevant information from their corporate web sites with screen-scraping technologies.

"€œOriginally, the company built these collection processes based on custom code,"€ said Andrew Brown, Vice President of Content for SupplyFrame. "€œHowever, as the number and complexity of data providers grew, these manually-coded processes were becoming onerous to create and maintain. For the next generation of our automated data collection mechanism, we looked at the various data integration alternatives available on the market."

Openness at one-third the cost

SupplyFrame considered several alternatives, including proprietary solutions and open source solutions. "€œThe debate was not really so much about proprietary versus open source, but more about the technical capabilities of the tool and its cost,"€ explains Andrew Brown. "€œNot only did Talend Data Integration fit our technical requirements, but its total cost of ownership over a five-year period was approximately one-third the cost of the least expensive proprietary competitor."

€Other determining factors in the choice of Talend Data Integration were the accessibility of the solution, a very quick learning curve, and the breadth of its connectivity. "€œWith several hundred connectors included in the release, and more available from the community via the Talend Ecosystem, we've got all our bases covered,"€ says Andrew Brown.

Finally, the Java code generation and the ability to "€œlook under the hood"€ (due to the code openness), also contributed to the decision. "€œWe do not expect to have to modify the source code of Talend Data Integration, but knowing that we have this ability if needed makes us more comfortable,"€ explains Andrew Brown.

Faster time to market

The implementation of Talend Data Integration helped SupplyFrame in two primary domains - time savings and flexibility in meeting customer requirements quickly. "€œPreviously, when a data collection need was identified, we would create a ticket to develop a process depending on what the flow was,"€ says Andrew Brown. "€œFor example, we set up an FTP drop and told manufacturers and distributors that if they could get the information in our format to the FTP drop, we could set up a process. Now, we're able to handle a lot more formats, and we don't have to put the onus on the supplier of the data to meets as many requirements. That makes it much easier for us to get data."

Because it is so much easier to develop data flows and since its metadata repository promotes reusability, Talend Data Integration has also dramatically changed the "€œtime to market"€ of the information. "€œThe time savings are very apparent,"€ says Andrew Brown. "€œWhat used to take us two weeks to develop, now takes less than a day - that's a 90% savings, not only in development costs but also in making new sources of information available. When implementing a brand new process, we're able to re-use some baseline processes we've already developed. It's very easy to duplicate a process and then tweak it according to whatever custom needs the new process requires."

Talend Data Integration also helps SupplyFrame handle error conditions and guarantees the correct execution of the collection processes. "€œIf a process fails, it can automatically send an email to the supplier of data and let them know there was something wrong with it,"€ says Andrew Brown. "€œBasically it extends our reach and gives us a lot more flexibility in being able to handle a lot of different cases."