SAP

Fig 2 - Dashboards

SAP Data Intelligence Cloud has the features that you would expect of a modern data quality solution. It supports data profiling, so examining, analyzing, reviewing and summarizing data prior to further data quality work. Data profiling is not just limited to customer data. Data can be validated against business rules, and potential duplicate records can be detected. Duplicate records can then be merged, and incomplete or inaccurate data cleansed. Location data can be validated by the use of the prebuilt integration with SAP’s Data Quality Management microservices. This deals with issues such as misspelled street names, missing postal codes etc, as well as allowing geocoding and data enrichment. In one case study, a construction company used this ability when hiring workers, with the address validation feature used for onboarding new workers.

The Data Intelligence product has a unified data catalog that curates data with centralised authorisation and security, applying data quality business rules. Metadata can be interrogated to detect the origins of data sources, and there is a business glossary so that company-specific terminology and definitions e.g. of terms like “net sales” can be applied. There is also monitoring of data quality and the scheduling of tasks, for example prompting data stewards to review possible duplicate records that require human intervention. The product has connectivity to a wide-range of data sources, both SAP and non-SAP (including relational and many non-relational databases, numerous file types and even old COBOL copybooks), and is extensible so that partner companies can build specific complementary solutions on top of the core functionality. One nice feature is that customers can examine profiled datasets and supply comments and ratings which are then visible to other authorized colleagues. The product has yet to provide the wide range of matching algorithms of its predecessor, but we expect to see that situation improve in the course of future product development.

Fig 3 - Functionality overview

SAP Data Intelligence is a modern, cloud-based data quality solution that has reasonably complete functionality, especially when used in combination with other SAP services such as its location validation microservice. It is well integrated with other SAP services, allowing connection to both SAP and non-SAP data, and its heritage as an independent product that was acquired by SAP means that it is by no means tied to core SAP ERP data. In practice, one customer (Evonik Industries AG) achieved significant tangible benefits by tracing errors in material classification and reducing out-of-stock situations by improving the accuracy of packaging specifications. This company halved the time for systems maintenance tasks, and speeded up the time to process complex packaging information from their suppliers by a factor of seven. Obviously, benefits will vary depending on specific use cases and industries, but SAP Data Intelligence has gained a “Top Rated” award from independent firm TrustRadius based on customer feedback, demonstrating that many companies are deriving value from the product.

The Bottom Line? 

The SAP Data Intelligence product is a natural solution for SAP customers that want to improve their data quality, which is an increasingly important issue, especially in industries that have significant regulatory requirements, but also available on-premise with good links to such as financial services and pharmaceuticals. Its cloud-native architecture but also good links to existing on-premise data and software means that customer can deploy their data quality solution in support of a migration of their core systems from on-premise to cloud at their own pace. Customers of prior on-premise SAP offerings in this area can be reassured by the bi-directional metadata exchange capabilities between this product and the older tools, easing migration at a pace that the customer can dictate. SAP’s strong market position will and indeed already has attracted plenty of partner companies to build extensions and complementary products on top of the core capabilities to round out any functionality gaps and allow application to specific industry niches.

SAP Information Steward contains a breadth of capability. Among the most integral is its enterprise data catalogue, which provides a central location in which to govern your various data assets. These assets can be collected from a variety of sources using “metadata integrators”. Native integrators are provided for several SAP products, relational databases, and modelling tools, and a variety of other integrators are also available. Once your assets have been ingested into the product, SAP’s Metadata Explorer provides browser-based access to view and edit them. Some detail is provided within each asset view, including data quality and profiling results, any associated rules, business terms, or informational policies, and links to the underlying, physical data. You can also add annotations to your assets, which effectively act as comments. Search access to your assets is provided, as are data profiling capabilities, which are separated into point-and-click “column analysis”, which provides field-level statistical information, and semantic content type profiling, which categorises your assets into content types, effectively acting as data discovery.

Another important capability is the Metapedia. This acts as a central location for defining, managing and implementing your business terms and policies. They can also be associated with your other assets (including other terms and policies). As with the data catalogue, search access is provided. Policies can also be time limited, and may be grouped into policy sets. Your organisation is scored against each policy (and each policy set) based on compliance with the data quality and validation rules associated with it. There is also an acceptability threshold for your policies that you can use to trigger actions if any policy score falls too low. You can import or export your terms using a spreadsheet, and you can access both your policies and your terms from outside SAP Information Steward via an API.

Fig 01 - Data quality scorecard in SAP Information Steward

Your validation rules themselves can be generated automatically from your data profiling results and can then be refined – or created from scratch – in the provided rules editor. Rule checking and alerting are both automated, and you can also view a selection of recommended rules via the Data Quality Advisor, which will assess the state of your system and suggest rules that might improve it. You can then fine tune these suggested rules and either accept or reject them. You can also view your existing and proposed rules in a somewhat more technical context using the Data Validation Advisor. The results of these rules are presented visually within your ‘data quality scorecard’, as seen in Figure 1. An additional, more data-centric results view is also provided. You can also share your rules with SAP Data Services, thereby integrating it with SAP Information Steward and providing the latter with advanced data quality capabilities, as well as enabling collaboration between your business and technical users.

Fig 02 - Data lineage in SAP Information Steward

Your data quality scorecard also includes data quality monitoring and financial impact analysis, the latter in the form of the Business Value Monitor, which puts a numerical, monetary value on how much your data governance and data quality practices are costing you. Moreover, as shown in Figure 2, the product provides visualised data lineage as well as more traditional impact analysis. It also includes the Cleansing Package Builder, which helps you to develop custom data cleansing solutions for any of your data domains – including domains that contain unstructured or semi-structured data – which can then be used to parse and standardise your data within SAP Data Services. Finally, SAP Information Steward offers centralised task management, including approval processes and workflows.

SAP Information Steward provides a centralised, enterprise-wide location for data access, understanding, and governance. By itself, this is not particularly notable for a data governance platform. What allows SAP to stand out from the crowd is the sheer quantity of data governance capabilities that it brings to the table. Its data cleansing functionality, in particular, is not something we’re used to seeing in data governance products.

The Business Value Monitor is a particularly impressive feature. By attaching a monetary cost to your current data governance practices, and therefore a concrete monetary gain to any improvements you might make to those practices, you ought to find it much easier to sell your organisation on data quality and governance initiatives. Given that these initiatives are often a negative sell, this is helpful indeed.

The product’s integration with SAP Data Services is instrumental, allowing you to put the data quality rules and cleansing packages you’ve built using SAP Information Steward into action. It is therefore unfortunate that its integration with other SAP products is somewhat lacking. For example, SAP Data Intelligence (which combines SAP Data Hub with machine learning capabilities) is designed to bring together AI and information management. It would doubtless prove useful to integrate it with SAP Information Steward, but such an integration does not exist (although we are told it is on the roadmap). The same is true of SAP Data Mapping and Protection by BigID, which provides data discovery. SAP Master Data Governance, however, does integrate with SAP Information Steward, and is therefore able to access its rules and scorecards as well as submit change requests.

The Bottom Line

SAP Information Steward is a very capable data governance platform that offers an extensive range of functionality.

Figure 1 – Mapping relational data to a graph model in Teleporter

The developers claim that OrientDB was the first multi-model graph database to be launched (in 2009). Originally, it was designed as a hybrid document/graph database, but since then the core (native) engine has been extended to include objects, spatial and key-value elements. Moreover, the product now offers a multi-model API that allows you to work with all types of model (including documents, graphs, objects, and so on) seamlessly through a single interface. The product is ACID compliant and supports strong consistency, though eventual consistency is an option.

It supports Apache TinkerPop (either as an API, for backwards compatibility with TinkerPop 2.6, or as a plugin for later versions) and, especially, Gremlin. More importantly, it uses an extended form of SQL for query processing that leverages MapReduce under the covers. OrientDB’s version of SQL notably includes live querying, the ability to create queries that are updated dynamically as your data changes over time. The product can be used in full schema, schema-free or hybrid schema environments and uses sharding for distributing data across a cluster. Scheduled full and incremental backups are supported.

In addition to the database itself, there is a native ETL engine that can be used to import and export JSON documents. Teleporter, as seen in Figure 1, is an additional tool included with Enterprise Edition that provides transformation capabilities for mapping from a relational to a graph model. OrientDB also includes a connector for Apache Spark and the Neo4j Importer. The latter allows you to readily and easily import an existing Neo4j database (or, to be more specific, the nodes, relationships, indices, and unique constraints contained therein) into OrientDB.

Figure 2 – The server management dashboard in Studio

There is also a Studio tool that allows you to view and edit your OrientDB environment via a web browser. This provides a variety of ways to view and manage your database, including a server management dashboard (see Figure 2), which displays various metrics relating to your server cluster, as well information about any single server in real time; a SQL browser; and a graph editor, allowing you to view, explore and modify your graph through a visual medium. There is also a JDBC connector to support integration with various (partner) visualisation tools.

OrientDB’s number one selling point is that it is a multi-model database. The chief advantage this offers is that it allows you to store and manage all of your data in a single place in whichever form best suits it. This is particularly relevant with the rise of big data and the quantity and variety of data that comes with it. Moreover, although other multi-model databases exist, all the data types supported by OrientDB are addressable vis a single language. This is not typically the case with other multi-model offerings.

The Community Edition is open source, with all the benefits and caveats that entails. Combined with the ability for multi-model databases such as OrientDB to obsolete several of your existing database products and licenses at once – that is to say, to take their place – the Community Edition’s total cost of ownership will often be quite low. This lends the product a measure of financial efficiency, which is retained (although obviously to a lesser degree) with the Enterprise Edition.

The company has also taken great pains to ensure that OrientDB is easy to use. The time it takes to start using it is minimal – configuration and installation are strictly optional – and several tools are made available to make your life easier, particularly when it comes to converting traditional data formats to graph (via Teleporter) and migrating data from an existing Neo4j installation (via the Neo4j Importer). In addition, Studio makes it easy to view and explore your system, and querying is intuitive thanks to the product’s support for SQL, which is rare within the graph space.

The Bottom Line

As a multi-model database, OrientDB is ahead of the pack, owing to its relatively long lifespan and a thriving open source community. As a graph database, it is exceptionally easy to implement and to use, thanks in large part to the great work done by Studio, Teleporter, and the Neo4j Importer. Moreover, the ease of use benefits of being able to write graph queries using SQL cannot be overstated.

Given the support of SAP, we expect a bright future for OrientDB. If you are in the market for a multi-model graph database, it is certainly worth a look.