In the past few years, there has been a shift in the data industry, leading to the emergence of a new category of data citizens: the ‘ad hoc’ or ‘citizen’ integrators. With these new personas adding to the (already long) list of data workers having access to corporate information, companies are needing to re-think the way they approach their data security and data governance strategies. Unlike data engineers, this new class of ‘citizens’ or personas, don’t necessarily utilize data integration as part of their day-to-day job, but it does still come into play every so often.
So who exactly are these integrators? According to Gartner, ad hoc integrators can include line-of-business developers, such as application developers, who may need to integrate data for a portion of their development, but data integration may not necessarily be an ongoing daily task. Citizen integrators include data scientists, data analysts, and other data experts within the business who may need to integrate data for their main job: analytics.
Innovation Brings the Opportunity for Insight
With an increasing number of data sources, there is a corresponding increase in the number of use cases that allow for every part of the business to become data-driven. Large-scale adoption of streaming data processing technologies like Kafka have increased the velocity at which data is absorbed by businesses, making IoT and clickstream data analysis feasible.
At the same time, open source big data technologies like Apache Spark have provided a framework for processing and analyzing growing volumes of data. Even Cloud service providers such as Amazon Web Services and Microsoft Azure have played a part in encouraging company-wide data-driven practices by further enabling businesses of every size to store, process, and explore more data without the monetary and resource investment that was previously required with on-premises technologies.
All of these innovations have created an environment that nurtures and encourages nearly every segment of the business—from supply chain to marketing—to utilize their data to inform choices that range from daily tasks to major strategic initiatives. Furthermore, deriving accurate insights and acting on those insights has become a competitive advantage in every market. In other words, having inaccurate analytics or even delayed insights can put you in a vulnerable position—a position where your competitors can overtake your market share by acting on industry and customer needs that you haven’t yet identified.
In order to remain competitive, companies have hired armies of data scientists and data analysts who are tasked with identifying trends and articulating actionable insights to each segment of the business.
The Non-Traditional Integrator is Born
IT is often not able to manage both the increasing number of data sources and the exponential number of ad hoc and citizen integrators’ requests to prepare data sets for analytics. Unfortunately, IT typically becomes a bottleneck where ad hoc and citizen integrators can be left waiting days or even weeks to access the data they need for analytics, which is not acceptable in a world where faster time-to-insight separates an industry’s leaders from its stragglers.
As a result, line of business developers, data scientists, and data analysts are left to integrate and prepare their data if they want to work with their data instantly instead of waiting for a few days or a few weeks to start their analytics.
The Future of Integration: Enabling New Integrators Without Losing Oversight
Now that we understand ad hoc and citizen integrators are assuming an increasingly important role in the world of data integration, where do we go from here?
Since ad hoc and citizen integrators have often been neglected or flat out ignored by IT due to lack of resources, many have resorted to finding their own tools for the collection, cleansing, and preparation of their data for analysis. And often, they use tools that are outside IT’s purview and thus outside of IT’s oversight and governance. While this gives the ad hoc and citizen integrators the ability to prepare their data quickly, it also opens up the risk that comes with unmonitored, ungoverned data.
In a time where data security and data governance are often the subject of front page news, it is paramount that every company restrict who is accessing what data, what each person is doing with that data, and how they are storing the data. In order to achieve this level of governance, companies need to focus their attention on a few things: people, process, and products.
First, people working with data within your organization need to understand that data management has become a team sport, and just like in any team, every person needs to understand his role. Just as important as understanding your own role, you need to understand how you can interact and contribute to your team to get the most accurate data (and subsequently, insights) possible. Because data management often spans across different teams, it is important for a process to be agreed upon by the different teams for some of these interactions. Last, it is important to find a product that will help you enable all of your data team members and operationalize the processes set between data team members all while being governed by IT.
With the changing data integration and analytics landscapes, the people who are interacting with the data, the processes they use to interact with each other, and the products that can support them are all changing. Is your organization prepared to enable and empower your new integrators?