A leading integrated financial services provider
Allianz Global Investors Distributors LLC is the distributor for the Allianz Funds, PIMCO Funds, and Allianz Funds Multi-Strategy Trust. In addition, AGID performs distribution services, and serves as program manager, for the CollegeAccess Section 529 Plan sponsored by the State of South Dakota.
Meeting corporate objectives
Early in 2009, AGID realized that they needed a standard ETL solution that would become the tool of choice for the entire organization.
AGID began by examining its needs and realized that a large ETL platform would be too difficult to manage in terms of cost, personnel, and expertise. “Once we had sized our projects, we started to look at smaller, or mid-size, ETL solutions,” said Inderjit Chhatwal, Vice President and Business Systems Development Manager at AGID. “We considered several options before selecting Talend as the most appropriate for our size, needs, and the kind of things we wanted to do, as well as for its ease-of-use. This was out of the norm for us, because we're primarily a Microsoft shop; Talend is one of the first open source software that we've incorporated at AGID.”
AGID quickly saw the benefit of using Talend's technology with Gamma Systems' Data Warehouse Studio to facilitate implementation of their data warehouse projects. The software serves as an integrated platform bridging project participants and technologies, enabling architects, data analysts, and data modelers to contribute specifications within their domains of expertise, with the end-result of generating ETL code that fully adheres to all the input domains and best practices.
A powerful combination
Prior to implementing Talend and Data Warehouse Studio, Allianz only used SQL Server Integration Services (SSIS) for integration, along with some small pieces in scripts and stored procedures. “Talend offered an excellent means for developing ETLs at a very rapid pace,” explained Simon Eligulashvili, Principal, Gamma Systems, “and we felt that we could manage all projects in Talend more efficiently than developing an individual ETL in SSIS. Data Warehouse Studio eliminated the need to know Talend inside and out. The combined solution allows users to enter business rules without knowing the underlying ETL tool in depth. Data Warehouse Studio generates ETL processes-in our case, Talend jobs-according to the business logic entered by the business users or by the analysts, while the architecture is specified by the warehouse architect. Basically, the solution can specify a set of generic templates. What changes from job to job is purely business logic; the overall framework stays the same.”
Data Warehouse Studio works with Talend to provide a very efficient data warehouse build. “We immediately saw the benefit of using Talend as our ETL tool,” said Inderjit Chhatwal. “Data Warehouse Studio in conjunction with Talend is basically a fast-track solution and saved us both time and money. It's proving to be 3 times faster than developing the same ETLs by hand and the ability to reuse jobs, instead of rewriting them each time, is extremely valuable.”
Developing a data mart; processing incoming feeds
The company had two projects that immediately required an ETL tool. The first populates a data mart from the sales accounting system on a daily basis and will soon be in use for production reporting. “Reporting will be more efficient and more readily available for the front office,” said Inderjit Chhatwal. “Because this is such a core set of data elements for our organization - we distribute mutual funds and these are the mutual fund transactions - reporting will come from the warehouse in the future, we expect that about 200 employees will access the system.”
“We're aiming high in terms of getting the data warehouses up and running in a really short amount of time,” added Simon Eligulashvili. “We're on plan to have all the data warehouses loaded in less than an hour.”
The second project involved processing incoming feeds. “These feeds aren't particularly complex, said Inderjit Chhatwal, “but in our world it's not that straightforward. We receive unified files from our Transfer Agent - the 3rd party that provides mutual fund transactional data to Allianz GID. Today we have a number of data sources that we are consolidating into two primary feeds, with a few additional smaller ones. We need to put together between 30 and 40 individual loading processes to get the right data from those sources into one operational data store. We are using Talend to move data from these standardized sources to the operational data source in addition to the loads of the data warehouse.”
Architecture flexibility is also important for AGID. “I've found that the convenience of using Talend both in a team environment on large projects, and as a stand-alone tool for smaller one-off tasks, is very appealing,” added Inderjit Chhatwal. “I don't have to worry about setting up an ETL repository to accomplish small tasks-such as generating Excel files for analysts or doing one-off data loads from files provided by users."
“Talend is now our standard ETL tool,” said Inderjit Chhatwal. “We are re-architecting our application portfolio to create specialized applications instead of the monolithic applications we have today. As we do that there will, of course, be more interaction between these applications and proportionately more use of Talend.”