Redis Labs

From the perspective of time-series data the key issue is how Redis modules work. Note that these are embedded into the database engine and not just layered on top. The relevant options are illustrated in Figure 2 and, of these, it is the time-series module, along with RedisGears and possibly RedisAI, that are of interest here, though Redis Streams may be useful in supporting in high ingestion rates.

RedisTimeSeries provides built-in aggregation functions such as calculating minima, maxima, sums, averages and so on. More significantly, you can label data – based on timestamps – on either an individual or global basis and then you can use those labels to support analytics. This will be particularly useful in Internet of Things (IoT) environments where you are doing initial analytics at the edge. Additional capabilities include the ability to compress data across different time series functions, specific time-series indexing, support for time buckets, programmable retention policies, and downsampling (reducing the sampling rate so that you can determine the granularity – typically of sensor data – you require). There are also  built-in Prometheus and Telegraf interfaces and visualisation support through Grafana.

All Redis modules are designed to interoperate with one another but RedisGears takes this one step further by facilitating inter-module transformations. It provides a serverless environment and allows you to aggregate data across multiple Redis database instances and react to activity based on pre-defined triggers. In other words, you get event-driven data transformations from one model to another, in real-time, in memory.

Then there is RedisAI. This enables the deployment of machine learning models and model serving, with the database supporting relevant languages such as Python, R and Scala plus deep learning support with the ability to embed TensorFlow, PyTorch and TorchScript models into your analytic workflows.

Finally, Redis is well-known for its performance, not least because it has historically run everything in memory. However, as the company moves away from focusing on caching use cases into wider environments, it can no longer assume that all of its clients can afford the amount of memory that may be required. For this reason, warm (as opposed to hot) data may be stored on SSDs and the company has been working with closely with Intel on its Optane Persistent Memory technology.

Apart from its performance and scalability – which are obviously major factors – the most outstanding thing about Redis Enterprise is the flexibility that its support for different data structures and modules provides. From a time-series perspective the relevant module has some significant features. However, we would like to see more geo-spatial support as the product is limited to supporting latitude and longitude, which may be enough for some IoT applications, but it doesn’t offer the breadth of capability that some other databases in this space do. Conversely, Redis offers many other functions that competitive time-series offerings do not do.

The Bottom Line

It is interesting to observe how Redis has managed to leverage its initial success as a caching technology, into something more general-purpose. It is now a major contender across a range of functionality and we expect that to be also true with respect to time-series.

Fig 02 Redis shared nothing symmetric architecture with node and cluster watchdogs

The architecture of Redis Enterprise is illustrated in Figure 2 where the Cluster Watchdog does what its name suggests; the Node Watchdog supports a secure (multi-tenant) user interface, a call level interface and a REST API; and the proxies handle the complexities of, for example, shared memory.

Fig 03 Combining analytic processing with operational and transactional processing

Perhaps more interesting from the point of view of combining analytic processing with operational and transactional processing is the way that Redis leverages different data structures. This is illustrated in Figure 3, along with Redis’ own suggestions about where these particular different data structures might be useful. Alongside these data structures, Redis provides a variety of inline analytics through its modules, which range from specific functions such as Topk, which tracks the most frequent elements in a data set, to RedisAI, serving deep learning models with built-in integrations into popular AI frameworks, supporting machine learning models and model serving. Further, there are also operational analytics capabilities such as RediSearch, RedisGraph, RedisJSON and RedisTimeSeries, amongst others.

More generally, Redis supports a wide range of programming languages, including Python, R and Scala. RedisAI supports TensorFlow and PyTorch and in the future, ONNX.

Finally, Redis is well-known for its performance, not least because it has historically run everything in memory. However, as the company moves away from focusing on caching use cases into wider environments, it can no longer assume that all of its clients can afford the amount of memory that may be required. For this reason, warm (as opposed to hot) data may be stored on SSDs and the company has worked closely with Intel on its Optane DC Persistent Memory technology, with general availability being announced in April 2019.

Apart from its performance and scalability – which are obviously major factors – the most outstanding thing about Redis Enterprise is the flexibility that its support for different data structures and modules provides. Moreover, a significant number of these features explicitly support analytic as well as transactional capabilities. For example, prior to the introduction of modules, Redis did not have any of the core data structures to enable fast and efficient manipulation of JSON documents or the ability to use Redis as a Graph database. Many competitive solutions fall short as they impose restrictions on the data to fit their native data model. Modules like RedisJSON are designed to be  intuitive regardless of whether someone is familiar with Redis or JSON. We should also say that we are especially impressed with the architecture behind RedisGraph, which uses sparse adjacency matrices. More broadly, we can say that Redis supports a broader range of pre-built analytic capabilities than most, if not all, other NoSQL databases.

The Bottom Line

It is interesting to observe how Redis has managed to leverage its initial success as a caching technology, into something more general-purpose. It is now a major contender in the provision of hybrid analytic and operational/transactional processing.

Figure 1 – A graph and corresponding adjacency matrix

Generally speaking, there are two ways to store and represent graphs. The first of these is the adjacency list, which consists of a list of all the nodes in the graph it represents, each paired with the set of nodes with which it shares an edge (or, in other words, has a relationship with). This is effectively the industry standard. The second is the adjacency matrix, which represents its graph as a single, square matrix with one row and column for each node within the graph. Within this matrix, nonzero values indicate the presence of an edge between the nodes represented by the corresponding row and column. An example of an adjacency matrix, alongside the graph it represents, is shown in Figure 1.

This method has several advantages in terms of performance. For example, determining whether two nodes share an edge is significantly faster when using matrix representation. Queries can be performed using direct mathematical operations such as matrix multiplication, which is often much faster than the traditional approach using adjacency lists. Performing a self-join, for instance, is simply a matter of multiplying a matrix by itself. Similarly, ingestion rates can be improved using adjacency matrices.

Historically, matrix representation has had two disadvantages. Firstly, it scales extremely poorly in terms of storage. An adjacency matrix for even a moderately large graph can take up far more space in memory than is commercially viable. However, the vast majority of matrices representing real world graphs are ‘sparse’ – meaning that almost every value is zero – and RedisGraph stores adjacency matrices in Compressed Sparse Column (or CSC) format, meaning that they are, effectively, only storing nonzero values. This almost always results in a very large saving in terms of memory. Notably, storing matrices in CSC format does not impact RedisGraph’s ability to use them in mathematical operations.

Figure 2 – Breadth-First-Search via adjacency lists and linear algebra

The second hurdle is that, until recently, there was no easy or standardised way to implement queries based on linear algebra (which refers to mathematical operations on matrices, such as matrix multiplication). This has been solved by 
the release of the GraphBLAS engine (see 
http://graphblas.org), which underpins RedisGraph. GraphBLAS is an open effort that provides standardised building blocks for graph algorithms based on linear algebra and using algebraic constructs known as semirings (rings with an additive inverse). ‘BLAS’ stands for Basic Linear Algebra Subprograms. The combination of matrix representation and linear algebra optimises and simplifies many different graph queries and algorithms. For example, a comparison between implementations of Breadth-First-Search using linear algebra and the standard approach using adjacency lists is shown in Figure 2. It should be clear that the algebraic approach is easier to write and to understand. It is also computationally simpler.

There’s one more advantage of the matrix representation that is worth noting. Matrix operations (such as multiplication) are extremely and easily parallelisable, and this property carries over to queries and graph algorithms based on linear algebra. This means that RedisGraph benefits very significantly from parallelisation. It will, in the future, be able to take full advantage of the massive parallelisation offered by GPU based processing.

RedisGraph is an extremely performant graph database. Thanks to the matrix representation it uses and the linear algebra algorithmics it implements, it is able to create over one million nodes in under half a second, and form three million relationships in the same timeframe. What’s more, early benchmarks performed by Redis Labs suggest that their graph is order(s) of magnitude faster than its competitors.

Moreover, RedisGraph’s CSC storage format mitigates the problems with storage and memory usage that the matrix representation of graphs has had in the past, providing a sixty to seventy percent reduction in memory usage. This makes adjacency matrices a practical proposition, allowing RedisGraph to benefit from all of their advantages with few – if any – of the accompanying downsides.

The Bottom Line

Although RedisGraph has yet to be officially released, it has already seen use by dozens of non-production users running real-time graph use cases. While it is early days in terms of features the theoretical advantages of using adjacency matrices are considerable. Even if you are not already using Redis, it is certainly worth looking into.