Hitachi Vantara

Fig 02 - The functionality of the Lumada DataOps Suite

The Lumada DataOps Suite aims to provide the functionality illustrated in Figure 2. That is, it is intended to support everything from the ingestion of data through to its analysis and presentation in dashboards. That said, while it includes some capabilities for data profiling within its data cataloging product (see below) it does not offer data quality or governance capabilities, for which it relies upon partners such as Collibra, as it does with data preparation   and data masking. In addition, Lumada Data Optimizer for Hadoop offers orthogonal capability by providing intelligent data tiering for HDFS environments. The company is a partner of Cloudera’s.

Fig 03 - Lumada Data Integration capabilities

As far as data integration is concerned, Figure 3 provides a snapshot of the capabilities offered though what it doesn’t tell you is that there is a distinction between Pentaho and Lumada Data Integration, where the latter consists of Pentaho Data Integration combined with Dataflow Studio, which is a web and microservices-based  application that enables scalable execution of dataflow jobs for business users rather than only being targeted at IT folk. It also doesn’t tell you that Lumada Edge Intelligence (IoT Edge) provides a lightweight environment that supports the hosting of AI and machine learning in edge devices, along with real-time KPI-based dashboarding, alerting, and support for streaming data. More generally, streaming is supported in all its varieties: micro-batches or real-time, continuous or event-driven, as you require. Also not mentioned is the fact that the company offers more than 200 connectors; support for Jupyter Notebooks along with R, Python and TensorFlow; and an extensible framework that includes SDKs, APIs, and plug-ins. And that change data capture (CDC) is supported.

Finally, we should discuss the Lumada (previously Waterline) Data Catalog. This is targeted at both enterprise data lakes and traditional data environments. It provides a complete solution for data discovery, cataloguing and compliance on these platforms, and is particularly notable for its discovery process, which is based around using machine-learning driven “data fingerprinting” to tag data consistently and intelligently using   a bespoke, machine learning driven solution for data discovery and tagging.

The discovery and tagging solution works by creating a “fingerprint” for each of your data fields. Each fingerprint is a collection of metadata – in some cases, more than 100 pieces of metadata – which includes information about both the content of your data and the context it exists in. Lumada Data Catalog can then intelligently and automatically classify and tag each of your fields based on its fingerprint. Once your data has been classified, it is exposed to your users as part of the data catalogue and is open to curation and crowdsourcing, the former of which feeds the machine learning algorithm, thus improving its accuracy over time. Different confidence levels can be attached to tags based on the likelihood that a particular tag applies.

The biggest differentiator for Hitachi Vantara’s Lumada DataOps Suite is its end-to-end capabilities from federated edge functions through to multi-cloud centralised operations. Other data integration vendors do not typically support sensors and other devices to the extent that Hitachi Vantara does. Conversely, the suppliers that focus on IoT and similar environments do not generally offer strong support for data integration in the traditional sense of that term. We should add that the acquisition of Waterline Data in March 2020 adds significantly to the capabilities that Hitachi Vantara can offer and we would not be surprised – indeed we would recommend – to see further relevant acquisitions in the future.

The Bottom Line

With Lumada being modular, and the fact that Hitachi Vantara can leverage partners when it needs to (depending on the use case), we regard the DataOps Suite as a platform play. We particularly like it within the context of distributed deployments where it can leverage its major differentiators.

Hitachi Vantara correctly recognises that the majority of enterprises will have far too much data for searching for sensitive data manually to be a viable prospect. On the other hand, automated discovery can sometimes be too simplistic to determine whether data is truly sensitive, especially when that data is only sensitive indirectly. The company’s solution to this problem is data fingerprinting, its bespoke, machine learning driven AI for data discovery and tagging.

Fig 01 - Data fingerprinting using the Lumada Data Catalog

Data fingerprinting works by creating a “fingerprint” for each of your data fields. Each fingerprint is a collection of metadata – in some cases, more than 100 pieces of metadata – which includes information about both the content of your data as well as the context it exists in. Data fingerprinting can then intelligently and automatically classify and tag each of your fields based on its fingerprint. Once your data has been classified, it is exposed to your users as part of the data catalogue and is open to curation and crowdsourcing, the former of which drives data fingerprinting machine learning, thus improving its accuracy over time. Figure 1 illustrates how fields in a data set are tagged by data fingerprinting, with different confidence levels based on the likelihood that a particular tag applies.

For the purposes of identifying sensitive data, a number of possible classifications are of interest, and in fact over 300 pre-defined and pre-trained tags are provided to identify and classify data that is sensitive under GDPR alone. What’s more, Lumada Data Catalog allows you to leverage tag-based rules to automatically tag data sets based on the fields they contain. For example, you could automatically tag your data sets as sensitive under GDPR if they contain fields tagged as first and last name, and where a field tagged as country contains data points corresponding to a European country. As you can tell from this example, these rules can mix data and metadata checks. What’s more, they are applied irrespective of data source, data format, and field names.

Fig 02 - Visually exploring data tables containing sensitive data

However, what if first and last names are in one table and country information is in another? With Lumada, you can visually explore any data tables that are related to the sensitive data you have already identified, as shown in Figure 2. This can be used to discover yet more sensitive data, and moreover, the software will automatically discover potential join conditions between these data sets and virtually join them using those conditions. This allows you to view them as a single data set. In turn, this can reveal additional sensitivity within your data that is only evident when it is seen in this way.

Fig 03 - Tracking data lineage with the Lumada Data Catalog

In addition, Lumada allows you to add custom properties to your data sets, which you can subsequently search on or filter by. For sensitive data, the utility here is that you can use these custom properties to store compliance metadata, such as the business purpose of your sensitive data. The product also provides visual, traversable tracking of data lineage, see Figure 3, which can either be imported from an existing source of lineage or inferred from your fingerprints. This capability may be useful for, say, locating data movement in and out of the EU. Finally, all of the searches and metadata available in the Lumada Data Catalog – including custom properties such as compliance metadata – are exposed via REST APIs, enabling integration with other compliance products.

Automated data discovery is essential for classifying your data consistently and comprehensively at an enterprise level, and this is just as true for sensitive data as it is for any other kind. Therefore, as a data catalogue, it should come as no surprise that Hitachi Vantara positions these features prominently. In this regard, Lumada’s adoption of machine learning in the form of data fingerprinting and the fingerprint system is notable.

The fingerprint system also has a number of features that are of particular benefit to sensitive data discovery. Fingerprints themselves contain a wealth of metadata, much of which concerns the context in which your data exists, and the rules engine that the product uses is able to operate in response to a combination of both metadata and data. Since whether or not any particular piece of data is sensitive can be highly complex and contextual, these are useful features for identifying the sensitive data in your environment.

The Bottom Line

The Lumada Data Catalog provides formidable sensitive data discovery capabilities as part of a data catalogue. If you are in the market for the latter, Lumada is a strong choice for the former.