Trends for 2022: How is data management evolving?
By Jamie Fiorda
Data management trends for 2022 and beyond
We all know the world is changing in profound ways. In the last few years, we’ve seen businesses, teams, and people all adapting — showing incredible resilience to keep moving forward despite the headwinds.
To shed some light on what to expect in 2022 and beyond, let’s look at five major trends with regard to data. We’ve been watching these particular data trends since before the pandemic and seen them gain steam across sectors in the post-pandemic world.
Trend 1: Accelerated move to the cloud(s)
We’ve seen a rush of movement to the cloud in recent years. Organizations are no longer evaluating whether or not cloud data management will help them; they’re evaluating how to do it. They are charting their way to the cloud via cloud data warehouses, cloud data lakes, and cloud data ecosystems.
What’s driving the move to the cloud(s)?
On-prem hardware comes with steep infrastructure costs: database administrators, data engineering costs, flow sweeps, and management of on-prem infrastructure itself. In a post-pandemic world, that’s all unnecessarily cumbersome. Organizations that move their databases and applications to the cloud reap significant benefits in cost optimization and productivity.
What to know about moving to the cloud(s) in 2022:
Note that I’m saying “cloud(s)” for a reason: the vast majority of organizations opt for multi-cloud and hybrid cloud solutions. Why? To avoid putting all their data eggs in one cloud basket.
While cloud data management services make it easy to move the data to their cloud, they also make it easiest to stay in their cloud — and sometimes downright hard to move data from it. Remember, a cloud vendor is typically aiming to achieve a closed system where you’ll use their products for all your cloud needs. But if you rely on a single provider in that way, a service change or price increase could catch you off-guard.
To stay flexible, many organizations are using best-fit capabilities of multiple cloud providers; for example, one cloud service for data science and another for applications. Integrating data across a multi-cloud or hybrid ecosystem like this helps organizations maintain the flexibility to manage their data independently.
Trend 2: Augmented or automated data management
Every organization relies on data — even those without an army of data engineers or data scientists. It’s very important for organizations of any size to be able to implement data management capabilities.
According to Gartner, “data integration (49%) and data preparation (37%) are among the top three technologies that organizations would like to automate by the end of 2022."
What’s driving the shift to augmented or automated data management?
Data management has traditionally taken a lot of manual effort. Data pipelines, especially hand-coded ones, can be brittle. They may break for all kinds of reasons: schema drifts when there are changes between source and target schema; applications that get turned off; databases that go out of sync; or network connectivity problems. Those failures can bring a business to a halt — not to mention that they are time-consuming and expensive to track down and fix.
Automating data management also frees up engineering resources. Gartner also says that by 2023, AI-enabled automation in data management and integration will reduce the need for IT specialists by 20%. Click to read the Gartner report.
What to know about data management in 2022:
By tapping into data services, even small and under-resourced data teams can implement data management and integration — by automating pipelines, quality, and governance on demand. Automation supports flexible pipeline creation, management, and retirement, granting organizations of any size or stage of growth the data observability they need in a continuous integration, continuous deployment (CICD) environment.
Trend 3: Metadata management
Since metadata is the glue that holds necessary data management pieces together, it’s no wonder that organizations are aiming to improve their handle on it.
As different lines of business develop their own shadow IT, the ecosystem grows in complexity: many companies end up buying multiple solutions and tools and then often need to pay consultants to make them work together.
What’s driving interest in metadata management?
Business agility is a requirement in today’s chaotic business landscape, which creates enormous demand for analytics. Healthy data is now a must-have for users with varied levels of technical skill. It’s impossible to expect them to become data analysts and engineers overnight in order to find, share, clean, and use the data they need.
What to know about metadata management in 2022:
Many companies have multiple data integration tools, quality tools, databases, governance tools, and so on. As data ecosystems become increasingly complex, it’s more important than ever that all those tools can speak to each other. Applications must support bi-directional data exchange. According to Gartner, data fabric architecture is key to modernizing data management. It’s the secret sauce that allows people with different skill sets — like data experts in the business and highly skilled developers in IT — to work together to create data solutions.
Trend 4: Real-time data access
Real-time data is no longer a nice-to-have; it is vital to operations ranging from manufacturing to utilities to retail customer experience. In addition, every company needs operational intelligence.
Any time an event is created, you should be able to provide that event in real time to support real-time analytics.
What’s driving interest in real-time data access?
We haven’t just seen the arrival of the Internet of Things (IoT) and Industrial Internet of Things (IIoT) — businesses are now reliant on them. In a world fueled by real-time data, batch integration and bulk integration are no longer enough to keep up.
What to know about real-time data access in 2022:
Extract Transfer Load (ETL) has to be supported by other integration styles including streaming data integration to capture event streams from the logs, sensors, and events that power your business. Make sure you’re building an architecture that supports both batch streaming in real time, and also virtual data access such as data replication and change data capture. That way you won’t have to move the data when you don’t want to.
Trend 5: Line of business ownership of data
Data is no longer tightly controlled in the back end by a central IT or data organization. In more and more businesses, the organization reporting to a CDO or CIO focuses on governance and compliance while business users process data within their own lines of business.
What’s driving line of business ownership of data?
As data becomes the language of business, we’re seeing the proliferation of citizen data scientists, citizen data integrators, citizen engineers, citizen analysts, and more.
What to know about line of business ownership of data in 2022:
Low-code and no-code data preparation and self-service data integration tools equip data users on the front end to ingest, prepare, and model the data for their business needs. These new “citizen” data workers are business experts who don’t have a PhD in statistics or engineering. They don’t know R, Python, Scala, Java, C Sharp, or Spark — and they shouldn’t have to. On the other hand, decentralizing data management can create data governance, compliance, and security headaches.
As more and more data software sits with the line of business, organizations should look for a data fabric that will enable central data engineering teams to monitor what the data preparation teams prepare. That way, data experts can improve data governance and compliance while lines of business maintain ownership of the data itself.