TimeXtender

At its core, Discovery Hub is a data repository. It ingests data from a variety of data sources before refining, processing and finally exposing it to users for consumption. However, one of the differentiators for the product is that this data is not exposed at a single point at the end of this process, but instead at multiple stages throughout it, via the ODX (Operational Data eXchange), MDW (Modern Data Warehouse) and finally the Semantic layers, as seen in Figure 1. This allows users to access the data at a level of refinement that is most appropriate for their ability and use case.

Data is initially ingested into the ODX from a number of data sources and held there as raw data. Much of the ingestion process can be automated, including data discovery. The ODX itself could be either a database or a data lake. Users are able to access the raw data held in the ODX, but as the data has yet to be processed, this will most likely be left to power users.

The refinement process begins by moving raw data into a data staging area (DSA), via a drag and drop interface. Once there, you can refine it by adding relationships, fields, tables, transformations and so on, before moving it into the MDW. You also have the option of using multiple DSAs, or even none at all. The MDW itself is a full-fledged data warehouse, intended to store and expose governed data. Accordingly, it is targeted at business users. It keeps full historical records and, moreover, data in the MDW can depend on data that is retained only in the ODX.

Figure 2 – A data lineage view in Discovery Hub

Data kept in the MDW can be used to develop semantic data models. To do this, you must select data to include in your model. Configuration options are available, including data security, scripting, and transformations. Discovery Hub will then take over, inheriting relationships from the MDW automatically and building the model accordingly. If part of the model is changed at a later date, only those parts need to be redeployed, and not the entire model. Data models are stored and exposed in the Semantic layer, the highest level of the Discovery Hub and the most suitable for casual users. You can export your data models into Qlik, Tableau and Power BI. Moreover, data stored in the ODX and MDW can be extracted by any product – most notably analytics platforms – that will accept data from a Microsoft environment.

Throughout this process, Discovery Hub automatically creates documentation, data lineage and impact analysis information. These last two can be displayed visually, as can the relationships within your data. An example of this is shown in Figure 2. In order to optimise performance, the product utilises an intelligent execution engine, incremental load, and parallel threading during heavy processing. The latter is optimised via the intelligent execution engine, which leverages machine learning.

Data growth, the shift towards digital transformation, competitive pressure and increasingly stringent compliance requirements all mean there is a need for users to be able to quickly and effectively access governed, analytics-ready data in a self-service fashion. Discovery Hub is a clean solution to this issue: a centralised data store which provides self-service at every level, from the raw data stored in the ODX to the curated data and data models held in the MDW and Semantic layer. This grants you fast access to the data that you need, whether your use case is to build advanced AI and predictive analytics models, support self-service data discovery or secure enterprise-wide reporting. Moreover, much of the creation of this store and the curation of the data within it is automated, removing tedious legwork and making it much easier to provide refined, governed data and data models. Finally, the framework that Discovery Hub provides, enables collaboration between your IT and business users, allowing the former to work with raw data within the ODX, before processing it and delivering it to the MDW for use by the latter.

The Bottom Line

Discovery Hub is a highly automated solution that provides strong support for self-service data access. Moreover, it provides this capability via a single, unified platform rather than a patchwork of unrelated tools. If you’re in the market for a centralised data store, and you’re running a Microsoft environment, you should seriously consider Discovery Hub.