We are in the era of the information economy. Now, more than ever, companies have the capabilities to optimize their process through the use of data and analytics. While there are endless possibilities to data analysis, there are still challenges with maintaining, integrating, cleaning it to ensure that it will empower the people to take decisions.
Bottom up, Top down? What is the best?
As IT teams begin to tackle the data deluge, a question often asked is, if this problem should be approached from the bottom up or top down. There is no “one-size-fits-all” answer here, but all data teams need a high-level view to help you get a quick view of your data subject areas. Think of this high-level view as a map you create to define priorities and identify problem areas. This map will allow you to set up a phased approach to optimize your most value contributing data assets.
The high-level view, unfortunately is not enough, to turn your data into valuable assets. You also need to know the details of your data.
Getting the details from your data is where a data profile comes into play. This profile tells you what your data is from the technical perspective. The high-level view (the enterprise information model), gives you the view from the business perspective. Real business value comes from the combination of both views. A transversal, holistic view on your data assets, allowing to zoom in or zoom out. The high-level view with technical details (even without the profiling), allows to start with the most important phase in the digital transformation: Discovery of your data assets.
Not Only Data Integration, But Data Integrity
With all the data travelling around in different types and sizes, integrating the data streams across various partners, apps and sources has become critical, but it’s more complex than ever.
Due to sizes and variety of data being generated, not to mention the ever-increasing speed in go to market scenarios, companies are looking for technology partners that can help them achieve this integration and integrity, either on premise or in the cloud.
Talend is one of the companies determined to be this partner. Starting as an open source ETL tool, Talend has evolved into an enterprise grade cloud data integration and data integrity platform. This vision becomes clear in the unified suite of applications they offer and focus to get the foundation of your data initiatives right.
Talend strategically moves data management to the cloud to provide scalability, security and agility. The recent acquisition of the Stitch Data platform and full support for the only ‘made for the cloud’ data warehouse platform Snowflake, makes the offering even more complete
Your 3 Step Plan to Trusted Data
Step 1: Discover and cleanse your data
A recent IDC study found that only 19% of data professional’s time is spent analyzing information and delivering valuable business outcomes. They spend 37% of their time preparing data and 24% of their time goes to protecting data. The challenge is to overcome these obstacles by bringing clarity, transparency, and accessibility to your data assets.
Building this discovery platform, which at the same time allows you to profile your data, to understand the quality of your data and build a confidence score to build trust with the business using the data assets, comes under the form of an auto-profiling Data Catalog.
Thanks to the application of Artificial Intelligence and Machine Learning in the Data Catalogs, data profiling can be provided as self-service towards power users.
Bringing transparency, understanding and trust to the business, brings out the value of the data assets.
Step 2: Organize Data You Can Trust and Empower People
According to the Gartner Magic Quadrant for Business Intelligence and Analytics Platforms, 2017: “By 2020, organizations that offer users access to a curated catalog of internal and external data will realize twice the business value from analytics investments than those that do not.”
An important phase in a successful data governance framework is establishing a single point of trust. From the technical perspective this translates to collecting all the data sets together in a single point of control. The governance aspect is the capability to assign roles and responsibilities directly in the central point of control, which allows to instantly operationalize your governance from the place the data originates.
The organization of your data assets goes along with the business understanding of the data, transparency and provenance. The end to end view of your data lineage ensures compliance and risk mitigation.
With the central compass in place and the roles and responsibilities assigned, it’s time to empower the people for data curation and remediation, in which an ongoing communication is from vital importance for adoption of a data driven strategy.
Step 3: Automate Your Data Pipelines & Enable Data Access
Different layers and technologies don’t make our lives easier to keep our data flows and streams aligned and adopt to swift and quick changes in business needs.
The needed transitions, data quality profiling and reporting can extensively be automated.
Start small and scale big. A part of this intelligence these days can be achieved by applying machine learning and artificial intelligence. These algorithms can take the cumbersome work out of the hands of analysts and can also be better and easier scaled. This automation gives the analysts faster understanding of the data and build better faster and more insights in a given time.
Putting data at the center of everything, implementing automation and provisioning it through one single platform is one of the key success factors in your digital transformation and become a real data-driven organization.
This article was made in collaboration with Talend, and represents my view on data management, and how these align with Talend’s vision and platform.
About the author Yves Mulkers:
Yves is an industry thought leader, analyst and practicing BI and analytics consultant, with a focus on data management and architecture. He runs a digital publication platform 7wData, where he shares stories on what you can do with data and how you should do it.
7wData works together with major brands worldwide, on their B2B marketing strategy, online visibility and go to market strategy.
Yves is also an explorer of new technologies, and keeps his finger on what’s happening with Bigdata, Blockchain, Cloud solutions, Artificial Intelligence, IoT, Augmented Reality / Virtual Reality, future of work and smart cities, from an architecture point of view, helping businesses build value from their data.