Once we had factored in all the elements, including license costs, availability and cost of expertise, cost of hardware, and ramp up time, the equation favored Talend's solution. We needed a product that would allow us to get started immediately and without training, that would scale to support our high data volumes, and offer connectivity to all our data sources. Talend was the obvious choice.Andrew Ross, Senior Development and GIS Specialist
DAI delivers development
DAI is an employee-owned international development firm. With some 2,500 people worldwide, the company is at the vanguard of international development, uniting technical excellence, consummate project management, and uncompromising customer service to solve its clients' most complex problems. Since 1970, DAI has worked in 150 developing and transition countries, providing comprehensive development solutions in areas including crisis mitigation and recovery, democratic governance and public sector management, agriculture and agribusiness, private sector development and financial services, economics and trade, HIV/AIDS, avian influenza control, and water and natural resources management. Clients include international development agencies, international lending institutions, private corporations and philanthropies, and host-country governments. Headquartered in Bethesda, MD, DAI maintains regional headquarters in Europe, the Middle East, and Africa and, at any time, has teams operating in many countries over the world.
Needed data consolidation
"Our projects tend to produce a huge amount of data that is not always optimally leveraged," explains Andrew Ross, Senior Development and GIS Specialist at DAI. "For example, when we send teams into the field to map the road infrastructure of Afghanistan, or when a crew in Haiti is collecting markets and fairs locations and periodicities, the data is used for the current project. But what we needed was a data consolidation mechanism, so that we can get a global view of our projects and reuse some of this data from one project to another."
In addition, each project run by a DAI team is managed like an independent business unit, which selects the IT systems most appropriate for the task at hand. "We have Lotus Notes, Oracle or PostgreSQL databases, many legacy applications, Excel spreadsheets, geographical coordinates systems, etc. In addition, we often need to tap into IT systems of government agencies in developing countries," says Andrew Ross. "That's a lot of heterogeneous data sources, and our challenge was to find a data integration solution that could be used across the board and would allow data sharing, cleansing and consolidation."
DAI identified Talend Open Studio for Data Integration as a great solution, not only because the open source model implied no license cost, but more importantly because the total cost of ownership was right. "Once we had factored in all the elements, including license costs, availability and cost of expertise, cost of hardware, and ramp up time, the equation favored Talend's solution," recalls Andrew Ross.
"We needed a product that would allow us to get started immediately and without training, that would scale to support our high data volumes, and offer connectivity to all our data sources. Talend was the obvious choice."
The absence of run-time also played an important role, explains Andrew Ross. "We have distributed teams around the globe, and the ability to deploy data collection and integration processes directly on their computers was critical. We needed a solution with no run-time license costs and with no engine to deploy - the fact that Talend Open Studio for Data Integration generates a stand - alone executable program was a key factor in our decision."
Geographically dispersed data
With teams operating on numerous projects in many different countries, one of the challenges faced by DAI headquarters is to keep track of the ongoing projects and locations around the globe. Andrew Ross explains: "Our teams often operate in remote locations, where communication meansare scarce. They are working on projects that can be as diverse as helping local communities manage their natural resources, monitoring avian influenza outbreaks, training elected officials, or mitigating conflicts. It is critical that, at any time, our senior management and the agencies that are contracting projects to DAI are able to get a global view of ongoing activities and of where our staff is deployed."
To provide this information, the Geographical Information Systems team at DAI has developed a real-time dashboard that leverages systems like Google Earth and Microsoft Virtual Earth, and shows DAI activities the world over. But the main challenge of this project was to collect disparate data and aggregate and cleanse it, in order to ensure the accuracy and completeness of the information presented by the dashboard.
"We were able to use Talend Open Studio for Data Integration to consolidate the data provided by teams in the field, in a variety of formats and at very different time intervals, into the PostgreSQL database that is used as the back-end of this dashboard," explains Andrew Ross. "Prior to building this system, this process required a lot of manual reconciliation work, performed on demand, and we had trouble getting the big picture."
Tracking bird flu in Indonesia
Avian influenza, often called "bird flu," threatens the viability of global poultry industries, the economic well-being of small farmers, and the health and nutrition of communities worldwide - especially the poor and more vulnerable. In addition, if the avian influenza virus were to establish itself in the human population, an influenza pandemic would have profound and drastic impact on every sector of society. The CBAIC (Community-Based Avian Influenza Control) project in Indonesia, financed by the US Agency for International Development, prepares the country to deal with avian influenza by training local people to work with farmers and to respond to contain outbreaks.
"One of the key challenges of outbreak monitoring and containment is to get a view of the surveillance network and of the cases as they occur," clarifies Andrew Ross. "The DAI teams in the field are collecting data on CBAIC program coordinators, trained farmers, and avian influenza cases. All this data needs to be aggregated and enriched with geographical coordinates, in order to produce maps showing how the country is covered and letting us track the progression of outbreaks."
For this task, DAI staff has implemented a lookup database that includes geographical coordinates for all the DESA - the administrative units in Indonesia, slightly larger than a village. This database is combined with data coming from the field, as Andrew Ross explains. "We have successfully implemented the data aggregation, merge, deduplication and enrichment routines using Talend Open Studio for Data Integration. All this data is then consolidated in a database and presented graphically using cartography software."
Monitoring financial development in Haiti
Access to credit and credible financial services is fundamental to private sector development. In developing countries, where communication and transportation are often lacking, access means proximity.
In Haiti, DAI ran a project to map all locations of banking facilities and the geographical sectors they serve. "The mapping project actually involved people driving around with GPS units and collecting coordinates," remembers Andrew Ross. "We then needed to correlate this information with data we had about the centers of economic activity - where markets and fairs are taking place is where people need access to banks, to deposit or withdraw money, contract credit or micro-loans, etc. Once again, data integration allowed us to reconcile this data, analyze the coverage of the financial institutions, and make recommendations for developing the population's access to financial services."
"Beyond the Geographical Information Systems data, there are many other applications for data integration at DAI," concludes Andrew Ross. "We are embarking on a very important data consolidation and optimization project, and I expect Talend Open Studio for Data Integration to play a key role in it."