Ververica

Apache Flink is an open-source stream processing framework. Its core is a distributed streaming dataflow engine that executes dataflow programs in a parallelised and pipelined manner. The pipelined runtime system enables the execution of both bulk/batch and stream processing programs. There is support for event-time processing and state management, with fault tolerance provided in the event of failure. It can be deployed standalone or on top of Kubernetes or Apache YARN, and can ingest and write data to and from other open-source offerings such as Kafka and Mesos, as well as public cloud services like AWS Kinesis Streams.

Fig 1 – The Ververica Platform architecture

Ververica, then, both provides commercial support for Flink and extends it via the Ververica Platform. Two versions of the platform are available, one fully-managed and in the cloud, and one self-managed and on-prem. Both are built on the same technology stack. At present, the former is available exclusively on Alibaba cloud and within the Chinese market (although we are told this will change in due course) while the latter is available everywhere. Either way, the platform is designed to allow you to run Flink easily and effectively at enterprise scale. This includes significant ease of use improvements, extensive commercial support, and more.

Customer Quotes

“Ververica Platform’s simple deployment abstraction enabled us to turn Apache Flink into a Platform-as-a-Service offering for our internal teams and prevent fraudulent transactions as we rapidly grow our services.”
Quan Phuong, Head of System Platform at One Mount Group

Apache Flink is a stream processing platform designed for high throughput and low latency, complete with automated up- and down- scaling. It can work with both bounded data sets (batch) and unbounded streams of data through a single API, and in turn a single user experience. It supports development in Java, Scala and SQL, again through the use of APIs, and sports an optimised SQL runtime. There is also FlinkCEP, a complex event processing library built on top of Flink to support the detection of event patterns. A notable feature is support for dynamic tables that are continuously updated and can be queried just like any other table (either via the Table API used by Java and Scala developers, or the SQL API). Queries on a dynamic table run continuously, and streams can be converted into dynamic tables or vice versa.

Ververica adds to this via enterprise support and enhanced ease of use. For instance, one of the product’s chief areas of concern is DevOps application lifecycle management. This includes support for upgrades, migrations, re-sizing, rollbacks and replays as well as features such as identity management, logging metrics and support for continuous integration and delivery. It is also worth noting that the Ververica Platform keeps track of all Flink jobs triggered by the platform. This includes the configuration parameters effective for each job, allowing users to reconstruct the settings of a Flink job at a specific point in time. There are also direct links to the metrics and log events that are generated by the job.

There are two elements to this: why you should use Flink instead of its immediate competitors, and why you should use Ververica’s solution instead of a different implementation of Flink (including your own). The second of these is the simpler to answer: Ververica is the foremost expert on Flink – its bona fides in this regard are substantial – and is an obvious choice if you want commercial support. At the same time, you will want that support, especially if you plan to implement Flink at an enterprise scale. Experience has shown that building a bespoke streaming solution out of one or more open-source projects is extremely complex, time-consuming, and expensive. In contrast, adopting a solution like Ververica is both far simpler, leaving you free to concern yourself with higher-level matters, and most likely more effective. Needless to say, we recommend the latter over the former.

The case for Flink itself is more subjective. It certainly has its benefits: performance and scalability, for instance, largely due to its highly distributed architecture. It also provides a series of APIs (and, consequently, connectors) that offer both flexibility and a wide range of connectivity. All this makes Flink very well suited for stream processing at a massive scale, as evidenced in its adoption by Alibaba (which liked it so much it bought the greatest contributor to it – Ververica), to name only one example among many. Moreover, in the coming months, Ververica plans to continue to embrace the cloud, and develop Flink into what it describes as the first truly cloud-native stream processor, with all the inherent advantages of the cloud (such as at-will, up- and down- scaling with high guarantees) baked into it at a fundamental level.

The Bottom Line

Ververica provides substantial enterprise capability and commercial support for a very capable stream processing engine (namely, Apache Flink). At the end of the day, there are two questions worth asking in regards to its offering: do you want to use Flink? And, if so, do you want to use Ververica? We think that the answer to the former may well be yes; the answer to the latter almost certainly is.