Talend was definitely the right solution for us, especially in terms of a fast and simple conversion, the variety of connectors, and its performance when it comes to large data volumes.Cyril Zenger, OSS/-BSS Engineer
In 2008, a new era arrived at the Swiss telecommunication industry: with the approval of Zurich's voters, the electric power company of the City of Zurich (ewz) was provided with a global credit for the construction and operation of a broadband network, which is based on the ewz glass fiber network, to modernize the communication infrastructure. Within just a few years, ewz's business unit responsible for this new telecommunication technology had done some groundbreaking work in the "Fiber to the Home" area and is currently providing 16,000 households with ewz.zürinet.
Digital television, Internet, and phone (key word Triple-Play) are now available at sometimes breathtaking bandwidths and have even become more affordable. ewz.zürinet is also very popular with service providers: Orange was the first service provider to offer its services through ewz.zürinet. There are currently twelve providers offering a variety of individual services and service packages through ewz.zürinet.
In addition, ewz Telecom (or short, just ewz) also offers professional services for business customers and telecommunication providers, in particular high-performance optical connections for local area networks (LAN) and computer centers/storage area networks (SAN) within Zurich. All these services require a solid high-performance IT architecture and efficient data management. For the development of the required data hub and all data integration processes, ewz is counting on the open source based Talend Data Integration.
"Telecommunication services can only remain competitive, if the underlying IT infrastructure supports the business process at an optimum level. This is particularly true, when not only internal applications and users need to access master data, but also external clients such as Orange," explains Cyril Zenger, who is the OSS/BSS Engineer responsible for operations and IT support at ewz. "It was clear from that start that this is where we have to build a high-performance data management infrastructure."
When the ewz.zürinet network was first started, the main focus was on the development and operation of the glass fiber network. The IT support for the business processes, for example for billing or support services, was to become the next focus. The same applied to data management: different data formats, different storage locations, and transfer protocols as well as the sometimes not so optimal data access were not really a problem in the beginning. As the system became more successful and grew, the data volumes and the requirements of the various applications and services providers increased.
"Our ewz.zürinet is very successful: we have clearly exceeded our planned growth rates and have to provide new connections faster than originally planned. Also the master data volume, which includes not only customer-related data, but also all network data, is growing dramatically," explains Cyril Zenger. "As a result, the processes and applications have only had limited access to some needed master data until now." In order to close this gap, a central master data hub was developed in 2009, which is to be populated with a central data integration solution.
Until mid-2010, ewz had been using a point-to-point architecture; the data was located in various systems and computer systems, and every system and/or application accessed it as needed. In this star-shaped architecture, data exchange between various nodes is possible, but it is not very efficient as a certain complexity level is reached. The lack of common protocols, proprietary interfaces, a low level of automation, insufficient data quality and the number of interfaces made this system very difficult to administer and maintain. The objective was therefore to develop an EII platform (Enterprise Information Integration) with a central data hub, into which the ewz business data model is implemented and to which all applications have access. The advantages of this central data management are transparent data quality, defined data content, higher data availability, and especially less interfaces between the systems, which, moreover, can be handled automatically.
The Talend solution
During the conceptual phase, which took six months, the "Zachman Framework" was used to develop information systems. The search for a software provider, who was able to implement the data integration accordingly, did not start until the conceptual data model had been developed with all its entities. First, the Internet was searched for possible solutions, and subsequently, all potential providers were subjected to a thorough evaluation. The provider did not only have to be able to fulfill various, predefined functions, but also had to offer an attractive price-performance ratio, and provide references from other users or system houses.
Due to the high licensing fees, the standard solutions of established providers such as IBM or Informatica could, in the end, not be considered; the added value offered in comparison with the Talend Data Integration, which was ultimately chosen, was simply not convincing enough for the decision makers. Aside from its range of functions, which left almost nothing to be desired, Talend also convinced with its many existing connectors and especially the integration of Filemaker, which no other provider was able to solve in an equally elegant manner. In addition, we liked the open user community that was able to provide quick answers to questions, the positive feedback from the surveyed users and consultants, as well as the solution's transparency. As a code generator, Talend produces program codes in Java or Perl, which can be viewed and modified by experienced users at any time.
The development of the master data hub started with the transfer of the conceptual data model to a logical data model. It was first determined which data from which sources and under application of which transformation rules would be moved and where it would be loaded. Subsequently, Talend jobs were created with the help of the integrated graphic modeling tool that is also able to develop complex ETL processes via "drag & drop". During the design and development of the ETL processes, ewz Telecom cooperated closely with rubcom GmbH, a Talend Platinum Partner. In spite of the complexity of the project, the implementation turned out to be very easy and was completed in just a few weeks. In total, ewz is currently using two licenses of the Talend Data Integration with six users.
In summer 2010, Cyril Zenger's team started developing the architecture of the master data hub. Prior to the transformation, the data had first been transferred to an offline data store for performance reasons. Aside from the databases Oracle, MSSQL, MySQL, and Filemaker, file types such as Excel, CSV, and XML had to be taken into consideration. The data mainly consists of customer and network data with information about the connections and the overall glass fiber network. All this data is first reviewed for errors and duplicates to improve data quality. It is then converted and made available internally and externally to applications such as billing systems, picking systems, customer administration, network documentation systems, or SAP. Also the scheduling, i.e. the time - and event - related planning and management of the ETL processes, can be through the "Admin Center" integrated in Talend.
In total, the data sources contain approximately 1.5 million master data sets that are pulled to the central hub every night. In the former point-to-point environment, the transfers had been semi-automatic and relatively slow, because it used various, in-house developed middleware applications. Tests have shown that Talend is able to move the entire data volume within 45 minutes. Going forward, internal and external data sources from partners, customers, and especially other public utility companies who want to use the ewz system will be added. For this reason, multi-client capability was very important from the start of this development project. With Talend, ewz is poised for the future; there are already connectors for almost all potential data sources, and additional connectors or modification can be easily done by ewz itself or together with Talend.
Cyril Zenger is especially impressed by the support of the community and the manufacturer Talend. When it comes to easy questions, there is almost always someone who has already solved this problem and who can provide some suggestions. More complex problems are quickly and efficiently solved by Talend's support team. The installation will continue to grow in the next few years; medium-term, the master data hub is to be used as the basis for a data mart and data warehouse in order to implement business intelligence aspects such as analyses and reports. In this regard, the Talend Data Integration will play an important part. In addition, Swisscom and ewz decided at the end of 2009 to work on the expansion of the glass fiber network in Zurich together. By the end of 2017, they will have invested approximately 430 million Franks to be able to reach more than 200,000 households and companies with glass fiber connections.
"Talend was definitely the right solution for us, especially in terms of a fast and simple conversion, the variety of connectors, and its performance when it comes to large data volumes," Cyril Zenger concludes. "We can now manage the data volumes that such a big network brings by using the central data hub and the Talend Data Integration. We also really like Talend's licencing policy: our company and data volume growth does not impact the licensing fees. That helps our budgets and provides us with planning reliability."