The Next Revolution: Data Reimagined for the Business User
Summary: What if everything we thought we knew about business intelligence was just the beginning? Across history, true breakthroughs rarely come from incremental improvements—they come from reimagining the problem entirely. The same could be about to happen with data.
“If I built what people asked for, I would have built a faster horse”
... is a quote attributed to Henry Ford, who pioneered building affordable automobiles for everyday people. Even though Ford never actually said that, it’s a great illustration of where iteration based on customer feedback can make a product better, but game changing products often come from starting from scratch by focusing on the users needs and solving the problem in an entirely new way.
Technology is constantly having moments like this, from the personal computers of the 80s, broadband internet in the 90s, and more recently, smart phones and AI chatbots. Data, which is the raw resource that so much of that technology makes use of, has changed surprisingly little in that time. When will data have its big game changing idea? It might be sooner than you think.
Incremental improvements in BI data
The world of business intelligence is one that has gone through a series of incremental product improvements largely driven by user feedback. Products have been expanded to include more and more sophisticated features so business users can solve more complex data problems. Data modelling has also gone through many iterations of improvements over the years from Relational modelling to OLAP cubes and everything in between, in order to unlock those new capabilities in BI tools.
One of the latest such iterations of the data layer is the semantic layer, which builds on existing data layers to add metadata and query abstractions effectively producing a BI data model over an existing data stack. This improves the user experience in BI by simplifying queries and describing data to make it more consumable for BI products and business users. However a semantic layer is still effectively another metadata model over an existing data layer which does not differ in principle from the BI models of yore. The data is still modelled and managed by the data team. As a result, business users often need to rely on the expertise of the data team to understand and consume the modelled data effectively.
Rethinking BI Data
For much of the past few decades, data models used in BI have been primarily focused on making it easy to model the data and improving data team efficiency, rather than the needs of the business users. In other words, they have been primarily designed around principles that are natural when organizing data, in much the same way as carriages were designed to make the best use of a horse.
Just like the automobile was a rethink of personal transportation with a focus around people, BI data is in need of a rethink around the needs of the consumers of that data, the business users. In fact, there has been just such an approach available in the market for a few years now. This approach turns the problem of organizing data on its head by creating a set of data artifacts (metrics) that are each built to expose data in a way that is directly consumable by a business user. Rather than starting with existing data and building a metadata layer on top of it, a metric is defined in business terms, based on business needs, and then connected to data to expose that information in those business terms.
The result is that the metrics used by a business are created to perfectly meet the needs of the business by ensuring the consumers of the data are an integral part of defining the data they use everyday. The data is still managed by the data team, but the metric is the contract for ensuring the data is set up to meet the needs of the business. Meeting the needs of the business user is the primary purpose of a metric.
Unique attributes of metrics that change the game for business users
1. Enforced meaning: A metric has its meaning enforced, by only exposing query options that alter the context of the data, but not its meaning, a user always knows what they are getting when they query a metric. If you have a "Revenue" metric, it can never be queried to have any other meaning. However you can still query it under different contexts, such as different periods of time, or for different product lines. This helps to build trust in the data, and promotes confidence for the business user that they are using the right data to make the right decisions.
2. Built in time series: Meaningful business data relates to points in time. When making meaningful business decisions, it’s not just enough to know what your numbers are, you need to know when those numbers were for. Furthermore, many business decisions are made not just on the data, but on how it has changed over time. For example, Is the revenue growing or shrinking of late? Metrics enforce that every value has an associated point in time, allowing that context to always be returned with the data in the metric. This also makes it very easy to generate comparisons between points in time and to visualize and analyze overall trends in the business data.
3. Composability: Metrics can be intuitively combined by business users in a couple of different ways to create more sophisticated analyses:
- Metrics can be queried together to get a joined result set. This allows for visualizations to display aligned data from multiple metrics at the same time surfacing new patterns in the data based on the combination of multiple metrics. The query results of any metrics can be combined based on shared context, even if the data they surface is from different systems. Since all metrics share a time context, at minimum this can be used to join the results from the metric queries. If the metrics have other shared dimensions, those can also be used to join the results creating the more granular breakdowns in the shared data.
- Metrics can be combined using a formula to create a new metric. These are often called calculated metrics. Making a new metric in this way can be done directly by business users by using existing metrics as operands in a mathematical formula. This allows business users to build a richer set of metrics based on their existing metrics without having to have access to the modelled data directly.
4. Metric catalog: Metrics can be individually managed and configured. Each metric can come from a different source, as needed to meet the business definition of the metric. Access to each metric can be individually controlled, giving a business fine grained control over who has access to what data. The shared metric catalog further ensures that all business users are using the same data by making it easy to find the metrics they need. Having a central metric catalog can help the data teams to identify and eliminate duplication, ensuring there is a single version of the data used by all users in the business.
5. Intuitive self serve: Since metrics are data in business terms, BI products that use metrics are designed to be intuitive for business users. Consuming data in meaningful ways by building reports, dashboards or analyses rely on an understanding of the data being used. Since metrics data is presented in business terms, it can be intuitively understood by the business user directly, making it a breeze to use it directly when creating and managing various BI artifacts. Gone are the days of having to contact the data team each time you need to build or edit a dashboard.
Furthermore, the structure of the data returned by all metrics is consistent, making it easy for BI tools to consume and make use of the data. This allows BI tools to present the data in ways that make sense without requiring much configuration by the user.
Because the metric has a single, non-negotiable business definition, AI systems can consume the data with the same confidence as a human, drastically reducing the chance of hallucination or misinterpreting the core business value.
6. Rich business metadata: Metrics incorporate metadata that relates directly to the business usage of the data. A good metric doesn’t just ensure the data has meaning, but also that the meaning is described exposed by the metric to the consumer so they understand how to use that metric. Metrics can also be tagged, certified and have an associated owner. This type of metadata goes beyond information about the data structure and enforced query operations typically found in BI models, by also giving the user all the information they need to understand the metric and how to use it.
Metrics are a game changer
These fundamental advantages are because metrics are built first and foremost around the needs of the business user. Metrics are not simply an iterative improvement over existing data models. Because of this new focus on building a data artifact specifically for the needs of the consumers of the data, Metrics are proving to be the big game changing moment for data that the BI industry has been waiting for.
This is the beginning. It's time to get started.