MemSQL

Last Updated: 28th February 2020
Analyst Coverage: Philip Howard

MemSQL was established in 2011 by founders from Facebook and Microsoft. Its eponymous product was first released in 2013. The database is available to run both on premises and on any public cloud.

The company has raised $110m in venture capital and it is based in California. As yet there are no overseas offices, though it does have staff working internationally. It primarily uses a direct sales model and, while it has an existing partner programme, it is in the process of extending its channel support. The company particularly focuses on legacy database ((Oracle, SAP, Microsoft and so on) replacements – at which it has had significant success – though migration support is currently a service engagement: however, we understand that relevant tools will be introduced in the near future.

MemSQL is primarily targeted at Fortune 1000 companies (major clients include Kellogg’s, Disney, Comcast, Uber and Cisco, amongst others) and the product is available both for on-premises and cloud-based deployments. A managed service offering is available. A fully-featured version of the product is available for download for free, the only restriction being the amount of in-memory capacity that is supported.

Company Info

Headquarters: 534 4th Street, San Francisco, USA
Telephone: +1 855 463 6775

MemSQL (January 2020)

Last Updated: 16th January 2020

What is it?

Fig 1 - MemSQL Ecosystem overview

MemSQL is a scale-out distributed database, with a lock-free architecture that supports both row and column storage. Unlike traditional relational offerings, and as can be seen in Figure 1, row-based data is processed in-memory for transaction and operational processing, while columnar data (which is compressed) is persisted to disk to support analytics. Relational behemoths such as Oracle, IBM, SAP and Microsoft do the opposite of this because, historically, they originated as row-based databases. In our view, this re-thinking of how and where you store and process columns and rows makes a lot of sense: if you need to process millions of transactions per second (which are naturally row-based) then doing this in-memory seems an obvious choice if you want to minimise your hardware requirement while still ensuring minimal latency.

As its name suggests, MemSQL uses SQL(which is compiled into machine code) and, in particular, its SQL support is wire-compatible with MySQL. The product is ACID compliant but immediate consistency is supported through synchronous replication that can cause a performance overhead, so this is not the default setting. The company will remove this barrier to adoption in a future release.

As can be seen, containers are supported and the company is planning to introduce support for Kubernetes.

Customer Quotes

“MemSQL as a core technology in our IT infrastructure allows us to ingest granular data more rapidly with higher scalability and efficiency.”
Akamai

“MemSQL allows us to do what we could never do before with Oracle.”
CIO at major Oil and Gas producer

How does it work?

Fig 02 - MemSQL logical architecture

The logical architecture of MemSQL is illustrated in Figure 2. The bottom layer reflects the processing and storage architecture already discussed. However, in addition to tabular format MemSQL also supports the ability to store data in JSON format, and it also supports geospatial datatypes and operations as well as full text search and time series. In the case of the former the product has datatypes for points, curves and polygons along with geographic indexing. Geographic features understand the curvature of the Earth but planar capabilities, for example mapping a factory floor, are not provided. On the time-series front the product’s architecture is well-suited to the efficient storage of time-series data but it currently lacks some specialised functionality. The company is planning extensions to be introduced towards the end of 2019 or into 2020. Features are likely to include the introduction of time buckets, first and last aggregate functions, the ability to join based on the latest time stamp, and interpolation.

Returning to Figure 2, the distributed cost-based query optimiser is aware of how data is stored so that it can direct query operations to in-memory or disk-based storage, as required. In this context, note that MemSQL uses skip-lists as its primary indexing technique, as well as supporting secondary indexes.

In the third tier are the Workload Manager and Distributed Ingest, where the former allows both memory and CPU resources to be allocated by user group, and the latter supports both batch and stream-based ingestion. Note that above this layer is multi-cluster management, which is provided through MemSQL Studio; along with facilities such as encryption, security, role-based access control and auditing; and a SQL API to support third-party environments such as Tableau and Qlik.

Why should you care?

As far as developers and users are concerned MemSQL is a traditional relational database and brings all the advantages that you would expect from that familiarity. And the wire compatibility with MySQL is a major advantage for anybody looking to migrate from an incumbent relational supplier. More significantly, MemSQL reverses the use of memory and disk storage with respect to rows and columns. This means that you can get better performance for transaction processing while using fewer hardware resources and maintaining low latency for analytics and query processing. The product has strong geo-spatial capabilities and the company has plans to significantly extend its time-series capabilities.

The Bottom Line

In our view, MemSQL starts from the right place without being encumbered by historical design decisions. The company has already proved that it is a serious competitor to the established relational database vendors and we expect this competitiveness to grow as the company introduces new features and capabilities in the years ahead.

MemSQL (June 2019)

Last Updated: 28th June 2019

What is it?

Fig 1 - MemSQL Ecosystem overview

MemSQL is a scale-out distributed, relational database, with a lock-free architecture that supports both row and column storage. Unlike traditional relational offerings, and as can be seen in Figure 1, row-based data is processed in-memory for transaction and operational processing, while columnar data (which is compressed) is persisted to disk to support analytics. Relational behemoths such as Oracle, IBM, SAP and Microsoft do the opposite of this because, historically, they originated as row-based databases. In our view, this re-thinking of how and where you store and process columns and rows makes a lot of sense: if you need to process millions of transactions per second (which are naturally row-based) then doing this in-memory seems an obvious choice if you want to minimise your hardware requirement while still ensuring minimal latency.

As its name suggests, MemSQL uses SQL(which is compiled into machine code) and, in particular, its SQL support is wire-compatible with MySQL. Historically, it did not support either stored procedures or triggers but the former was rectified in the 6.5 release (the current version is 6.7) while a combination of stored procedures and database polling can be used in place of triggers. The product is ACID compliant but immediate consistency is supported through synchronous replication that can cause a performance overhead, so this is not the default setting. The company will remove this barrier to adoption in a future release.

As can be seen, containers are supported and the company is planning to introduce support for Kubernetes.

Customer Quotes

“MemSQL as a core technology in our IT infrastructure allows us to ingest granular data more rapidly with higher scalability and efficiency.”
Akamai

“MemSQL allows us to do what we could never do before with Oracle.”
CIO at major Oil and Gas producer

How does it work?

Fig 02 Workload Manager and Distributed Ingest

The logical architecture of MemSQL is illustrated in Figure 2. The bottom layer reflects the processing and storage architecture already discussed. However, in addition to tabular format MemSQL also supports the ability to store data in JSON format, and it also supports geospatial datatypes and operations as well as full text search and time series. Above that is the distributed cost-based query optimiser, which is aware of how data is stored so that it can direct query operations to in-memory or disk-based storage, as required. In this context, note that MemSQL uses skip-lists as its primary indexing technique, as well as supporting secondary indexes. While there are both advantages and disadvantages when using skip-lists versus traditional Btrees, the former will tend to outweigh the latter for memory-optimised environments such as MemSQL.

In the third tier of Figure 2 are the Workload Manager and Distributed Ingest, where the former allows both memory and CPU resources to be allocated by user group, and the latter supports both batch and stream-based ingestion. Note that Figure 1 is not complete when it comes to supported technologies. Thus, for example, the company has customers using Flink as a streaming platform rather than Kafka. Above this layer is multi-cluster management, which is provided through MemSQL Studio; along with facilities such as encryption, security, role-based access control and auditing; and a SQL API to support third-party environments such as Tableau and Qlik.

Why should you care?

More significantly, MemSQL reverses the use of memory and disk storage with respect to rows and columns. This means that you can get better performance for transaction processing while using fewer hardware resources, while maintaining low latency for analytics and query processing. However, the performance overhead related to immediate consistency will be a worry for transaction processing environments so we are pleased to hear that the company will be rectifying this.

The Bottom Line