Compared to manual coding, Talend Open Studio for Data Integration saved us 57% of the development time-for the first project alone. Using Talend Open Studio for Data Integration also lets us reuse data integration processes from one project to another. This means that after an initial investment, we can leverage the work done for subsequent projects (with different structures and technologies) much faster than if we had to start over every time.Christian Roszak, CEO
Specialized road traffic consulting
Based in Aachen, Germany, Theis Consult GmbH was founded in 1991 by Theo Theis, a specialist in civil engineering. The company has since expanded its operations to include IT, and today Theis Consult provides services from consultancy to software development in this sector. Theis Consult recommends solutions and helps its clients develop a variety of projects, including urban management, road traffic security, research and development, etc.
Theis Consult has become one of the leaders in road traffic consulting, planning, and management, developing software and IT solutions to support the traffic network. Theis Consult provides digital documentation and host systems to control and manage traffic.
Improving traffic conditions on Germany's extensive Autobahn system
Germany prides itself on having one of the best motorway (Autobahn) systems in the world - 12,500 km (or 7,767 miles, the third largest after the US and China) of roads in excellent condition linking all the major cities in the country and providing travelers optimal driving conditions and real-time information on traffic situations with a dense network of electronic signs.
However, like any country, Germany is subject to fluctuating traffic conditions: school vacations and major holidays, poor weather, rush hours, road works - all these factors can contribute to increased circulation and traffic jams.
Germany is a federal republic comprising 16 states, each responsible for managing its own territory. Even though the road system is planned and organized centrally at the federal level by the German Federal Ministry of Transport, each state has a transport administration responsible for traffic planning and optimization, for road works, etc.
"Understanding the traffic patterns requires analyzing tremendous amounts of data in advance," explains Christian Roszak, CEO of Theis Consult. "In addition, each transportation administration uses its own system for traffic monitoring and for counting vehicles, adding another layer of complexity - all this data needs to be processed and formatted to fit our analytics databases."
Data integration to disperse traffic jams
To develop a traffic monitoring and analysis system that could be deployed easily in all roadway management entities nationwide, Theis Consult needed a powerful and versatile data integration solution. "The data gathered by road sensors (car counting, speed, etc.) exists in a variety of formats - Oracle, Informix, PostgreSQL databases, as well as XML files, ASCII flat files, etc. In addition, every configuration is different, with as many database schemas as there are source systems," explains Christian Roszak. "We also need to aggregate RDS-TMC data used to broadcast traffic alerts messages over regular radio programs, weather and road construction-related data that is stored in a specialized format, plus other miscellaneous information (population density, vacation dates, construction sites, etc.)." Even though there are several defined standards for interfaces and data exchange (like TLS, DATEX II, TPEG, OKSTRA, etc), these are often not useful or accessible for the daily business of analysis, strategic and conceptual planning.
Theis Consult deployed a similar project for a roadway agency, using custom scripting to aggregate and transform all the source data. "The data processing is very comlex," recalls Christian Roszak. "The manual coding of data integration processes ended up being very time consuming, difficult to maintain, and the resulting processes could not be reused for other similar projects because of all the specifics that had to be hand-coded."
Theis Consult then decided to look at data integration solutions. "Having used proprietary ETL products in the past, I knew about their power, but also about their limitations," explains Christian Roszak. "Because of the high data volumes we are processing, a lightning fast run-time execution is required. We also need the ability to customize the solution with additional connectors and data processing components. After a short evaluation, the choice of Talend's solutions became obvious."
Talend Open Studio for Data Integration generates native Java and Perl code with embedded SQL (for database interactions), and provides the highest performance run-time execution without having to rely on a cumbersome and slow integration engine. Its openness makes it easy for users and integrators to build their own components and connectors and add them to the solution, should they need additional features. However, because Talend Open Studio for Data Integration comes out of the box with over 250 connectors, most of the bases are already covered.
"Compared to manual coding, Talend Open Studio for Data Integration saved us 57% of the development time for the first project alone," concludes Christian Roszak. "But beside the added flexibility and the initial time savings, using Talend Open Studio for Data Integration also lets us reuse data integration processes from one project to another. This means that after an initial investment, we can leverage the work done for subsequent projects (with different structures and technologies) much faster than if we had to start over every time."