Analyst Coverage: Philip Howard
Kx Systems was founded in Palo Alto, California in 1993. The company is a subsidiary of First Derivatives Inc. plc, which acquired a majority shareholding in Kx in October 2014. FD employs over 2,000 people worldwide and has operations in cities around the world including London, New York, Stockholm, Singapore, Hong Kong, Tokyo, Sydney, Toronto, Philadelphia, Dublin, Belfast, Zurich, Seoul and San Francisco.
In capital markets, where the company originally focused, Kx used both direct marketing and channels. It has partners around the world that provide product sales and consulting. In addition to these companies, the company also has OEM partners. However, for Internet of Things (IoT) applications Kx primarily goes to market directly. Kx started with the smart energy and utility sectors, and since has added customers in several industries including precision manufacturing and IoT more generally.
Last Updated: 3rd March 2016
Kx technology consists of the kdb+ database, the q language, a CEP (complex event processing) engine associated with the database, and various tooling products designed for non-specialist users. Kdb+ is a column-based, hybrid in-memory database with stream processing capabilities, primarily designed for analytic workloads. In so far as in-memory capability is concerned, we refer to it as "hybrid" because it uses in-memory processing as much as it can but recognises that in some cases it may be impracticable to load all relevant (typically historic) data into memory and that you therefore need to employ techniques that will not only leverage memory-based processing but optimise performance when not all the data can fit into memory.
The product's stream processing capabilities (which means that you can analyse very large quantities of information in-flight, in real-time) arise from the fact that kdb+ is tightly integrated with the product's development language q. This is a vector (array) processing language that is much more efficient than SQL and which can be used to develop analytic applications as well as for query purposes.
The company's historical approach to marketing has been both through direct marketing and via channels. It has a number of partners around the world that provide product sales, training and installation as well as first line support. Many of these partners have extended Kx's capabilities by offering specialised financial capabilities with things like graphical user interfaces for business intelligence and extended complex event processing. In particular, First Derivatives has built a suite of business intelligence products, including a visualisation tool, on top of kdb+.
In addition to these partners the company also has OEM partners. For example, 1010Data, a cloud-based data warehousing solution for financial service companies, is based on Kx technology. Datawatch is another Kx OEM client.
However, while this has been Kx's approach to financial markets it has been directly addressing big data opportunities outside the financial sector. As this becomes more mature we expect the company to follow a similar channel model.
Kx Systems is active in encouraging the use of its technology in the academic world and a number of universities in North America, Germany and the UK are using it.
Kx has an impressive customer list. In the financial sector users include Goldman Sachs, Morgan Stanley, Merrill Lynch, JP Morgan, Deutsche Bank and many others. Nine of the world's top ten financial institutions are clients and more than a hundred financial institutions and hedge funds leverage Kx's technology.
The company's move into the big data space and the Internet of Things is in its early days but it has already attracted a number of customers, include Purdue Pharma and Total Gas and Power UK.
Kdb+ stores data in a columnar database, which may be compressed, if required. The software implements a hybrid in-memory/disk architecture that is optimised to use whatever resources are available. Data [in] particular, stream processing often requires that historic data is needed for contextual purposes and, given that a year's of worth of tick data (for example) comprises something like 6 or 7TB then you really do need hybrid capability. As far as availability is concerned kdb+ stores memory images on disk to facilitate rapid recovery.
A further feature of kdb+ that is worth mentioning is the fact that it supports user-defined datatypes as well as built-in datatypes. In the latter case, time series support (which can be accurate down to nanoseconds) is implemented by means of specific datatypes for elements such as dates. Similarly, a relational table is a base datatype.
On the language side q is a declarative language with similarities to SQL. For example, it has functions such as SELECT statements and WHERE clauses as well as updates, deletes and so forth. Where it goes beyond other environments is in its mathematical functions, such as variances, and in the fact that developed applications are tightly integrated with the data and database. More technically q is a vector programming language (that is, it addresses vectors [arrays] rather than tables) that uses memory mapped files for numeric processing. This has important implications, not only in its own right (because you get better performance) but also because Intel is increasingly adding vector processing capabilities to its processors. As Kx Systems is a partner of Intel, the former aims to leverage each new piece of relevant functionality that Intel introduces, so that performance will continue to improve.
A further consequence of the fact that most programming is written in q rather than SQL is that there is not a native fit with third party BI and visualisation tools. However, First Derivatives have built these capabilities on top of kdb+ and there is also an ODBC 3 driver that allows kdb+ to be used in conjunction with Tableau and Excel.
A number of IDEs (integrated development environments) can be used in conjunction with kdb+. Because of the conciseness of the code, programs tend to consist of relatively few lines, which makes debugging relatively simple. Code is interpreted but the interpreter takes up not much more than 100K and this makes it easy for the environment to support many simultaneous processes. This small footprint not only means that installation is very fast, it also reduces the risks and costs associated with upgrades and maintenance.
Kdb+ runs on Solaris, Windows, Linux and OSX platforms and is currently in version 3.3 with version 3.4 to be available shortly. There is also a free-to-download 32-bit version that runs on practically everything.
In addition to the usual sorts of services one might expect (training, support and so forth), which are provided by both Kx Systems and its partners, the company runs a web community (www.kxcommunity.com). This provides details of events, jobs and meet-up groups as well as a discussion board, conversations and blogs. It also details free tools that are made available within the Kx community (whether by Kx or by a member of the community) and which you can download.
Last Updated: 10th December 2018
Kx technology supports the rapid analysis of time-series data – something that has historically been a rare capability – and, more generally, the analysis of streaming, in-memory and historical data, and combinations thereof, with very low latency.
Kx technology is built on the kdb+ database, which the company has recently taken to calling as a time-series database. While this is valid as a description it does not reflect the architecture of Kdb+, which is columnar. On top of Kdb+ the company provides a built-in array processing language called q. The array-based nature of this processing means that it can work across both columns and rows simultaneously, making it much more efficient that traditional approaches. It is also worth noting the terseness of q, which is actually SQL-like. The combination of Kdb+ and with array processing facilitates concurrent execution and parallelism. The environment supports deployment across multiple machines with a distributed capability for scale-out in clustered environments. Resilience and automatic recovery are provided, together with load balancing and replication.
In practice, the combination of stream processing, together with in-memory processing and persistent data storage, means that Kx provides a so-called Kappa architecture (a simplified version of a Lambda architectures), as shown in Figure 1, even though this is hidden under the covers. Where it differs from many other potential providers of Kappa, or indeed Lambda, architectures – particularly where these are to be built on a variety of open source offerings – is that Kx offers a consistent, unified approach to supporting the combination of real-time and batch analytics that Lambda and Kappa architectures are aimed at, with a single code base.
"Through its ability to rapidly process vast amounts of time-series data, provide analytics in real-time, and integrate with our machine learning pipelines, Kx is ideally placed to power our neuroscience platform."
"There are many valuable applications of satellite imagery across a range of industries, many of which are time sensitive and require powerful analytic processing. By combining our data and Kx technology we expect to be able to provide new and existing customers with unique and valuable insights."
Airbus Defence and Space
As can be seen in Figure 1, Kx is not limited to providing an engine but also offers Control for Kx, Monitor for Kx, Analyst for Kx, Dashboard for Kx and various industry-specific solutions that reflect the company’s target markets. Some relevant points include the fact that Kx has a small deployment footprint (600KB) so is suitable for embedding in edge devices; is available in the cloud as well as on premises; and includes an ODBC driver for connection to popular third-party dashboard platforms such as Tableau, as an alternative to the tools Kx offers.
Of these tools, Analyst for Kx provides data preparation capabilities, native statistical libraries and integration with machine learning environments such as TensorFlow, Theano and others. You can also call python libraries from within q code. Kx for Monitoring, on other hand, is specifically targeted at network, IT and similar environments where you need to monitor the state of your infrastructure. This is illustrated in Figure 2. It supports the real-time analysis of the network to display critical information and raise alerts, and it can combine this with historic information to support such things as trend analysis.
Kx has proved itself in what is unarguably the most demanding big data market: financial trading and risk management. The company has been targeting IoT and related environments over the last few years and has had some notable success across a range of applications, where the collection and analysis of time-series based data is important.
We would have to say that Kx has been fortunate. We do not believe that it anticipated the Internet of Things. What it did was to focus on offering the best possible performance for its chosen (financial) market. It just happens that the (big data) issues faced by that market are exactly analogous to many Internet of Things environments. Not only is Kdb+ suitable for implementation for many of these use cases but, more importantly, the product has significant technical advantages in these areas, not least that it provides both real-time and batch-based capabilities from a single platform.
The Bottom Line
Kx offers a small deployment footprint, a single platform for streaming analytics and a fast (time-series) database, and the optimisation of in-memory and persisted data processing. It therefore fulfills the requirements for embedding in edge and gateway devices as well as, more generally, as a streaming analytics platform for live sensor data.