WhereScape

Figure 1 – The data warehouse design process in WhereScape 3D

As mentioned above, WhereScape offers three data warehouse automation products. WhereScape 3D allows you automate the design phase of the data warehouse lifecycle, as shown in Figure 1. This includes both designing and modelling your data warehouse, supported by, among other things, data discovery and data profiling. This is then integrated with WhereScape RED, which takes over after the design stage is finished in order to create and implement your new data warehouse. An overview of WhereScape RED can be seen in Figure 2. Finally, WhereScape Data Vault Express combines WhereScape 3D and WhereScape RED in service of automating the creation of a data vault “2.0”, an evolution of the original data vault concept.

WhereScape’s offerings are metadata-driven. This means that they leverage your metadata to generate the infrastructure of your data warehouse. It also means that as your metadata changes, your infrastructure can change accordingly and automatically. This goes a long way towards future proofing your data warehouse, thereby mitigating any risk that could result from it falling out of date.

Figure 2 – The WhereScape RED architecture

WhereScape also offers an optional Streaming add-on for WhereScape RED and WhereScape Data Vault Express (with support for Apache Kafka) that allows you to leverage your metadata – the same metadata used to create your warehouse – to create a streaming platform, which can then be used to analyse data as it flows into your system (and therefore your warehouse). In other words, it enables streaming analytics. This is usually, but by no means exclusively, used to analyse sensor data arriving from the edge. This data can then be used, among other things, to enrich the data stored in your warehouse. In addition to streaming platforms, WhereScape can use your metadata to generate data pipelines, although implementation and execution of these pipelines is left to third party products, such as StreamSets (a partner of WhereScape) and Apache Spark. WhereScape also offers Change Data Capture for a select number of source databases, including SQL Server, Oracle and MySQL. This provides event-by-event recording of source system data record changes, meaning that you can automatically ingest and process only changed data rather than whole data sets. This reduces data-processing time and eases the burden on both source systems and data warehouse infrastructure.

Automation, as a rule, improves productivity and efficiency. Moreover, the more automation you have, the greater the benefits. In the case of data warehousing, the greatest returns come from automating the entire data warehouse lifecycle, from design through to implementation and beyond. This is exactly what WhereScape does. Moreover, the process of creating a data warehouse normally consists of so many different tools, many of which are designed and used exclusively for their own associated subprocess, that creating and enhancing a data warehouse is a laborious process that is therefore ripe for automation. This is even more true if you intend to create a data vault: normally, creating a data vault would have a steep learning curve. Via automation – “data vault automation”, if you will – WhereScape Data Vault Express dramatically reduces this learning curve and makes creating a data vault a much more reasonable endeavor.

The Bottom Line

WhereScape provides automation for the whole data warehouse lifecycle, allowing you to spend your time modelling and designing your warehouse instead of manually creating it. It also provides a variety of additional features to go along with this capability. In particular, its support for streaming analytics via the Streaming add-on is very useful if your goal is to ingest sensor data into a data warehouse in real-time. Moreover, its ability to support the creation of a data vault is significant given the growing popularity of data vaults, particularly – but by no means exclusively – in Europe. Ultimately, if you are in the market for data warehouse automation, then WhereScape should be on your shortlist. This is especially true if you also have an interest in either data vaults or streaming analytics.