The developing data integration market

You would think that the data integration market was pretty much mature. After all, conventional ETL (extract, transform and load) tools have been around for the best part of 15 years. Sure, they are developing and vendors are implementing new procedures and capabilities: providing change data capture and real-time capability; offering enterprise information integration (EII) and/or enterprise application integration (EAI); implementing grid architectures and so on – but you could be forgiven for thinking that this should be a consolidating market.

Consolidating markets typically occur when technology is mature, but all the signs are that the market is expanding rather than consolidating. I am currently researching a new report into this area and all sorts of interesting companies and approaches are starting to come out of the woodwork. In the past, I have written about Sunopsis and the growing impression that it is making in the market, and of course both Oracle and Microsoft are substantially expanding the facilities that they provide in this area, but there are also many interesting new (or relatively new) companies entering the market.

I have already written about ETL Solutions, the Welsh company that has a product that generates Java; there is also at least one open source ETL product (which also generates Java); there is Solonde (an acronym for Solutions On Demand, where the company came up with the On Demand tag before IBM, much to the latter’s chagrin) which is a German/US company that provides a solution that has been designed from the ground up to run on a grid architecture; and there is even a shareware ETL product. There are also a bunch of other companies in the market that I haven’t got around to talking to yet.

So, why does the data integration market have all the signs of an expanding market? The answer was suggested to me by Solonde but I might reasonably have deduced it for myself. The data integration market as a whole is estimated, by analysts, to be worth many billions of dollars per annum, but sales of data integration products amount to less that bn. In other words there is a large proportion of the market that is not being addressed by the traditional vendors. Indeed, talk to Informatica or Ascential (IBM), for example, and they will consistently tell you that their major competitor is not each other (or anyone else) – but hand coding.

Now, you can go on forever about the downsides of hand coding (no data lineage, continuously reinventing the wheel and so on) but the bottom line is that most of the leading products are too expensive, both in terms of initial cost and implementation, given that many data movement tasks are small, rush jobs. This is what leaves an opening for new suppliers: not just new technology but also easier-to-use interfaces and, in particular, a much lower price point.

Of course, most of these new vendors are targeting project or departmental solutions and not the enterprise capabilities offered by Informatica, Ab Initio and IBM – but that is not entirely true. Sunopsis, Solonde (about which I will write more in a subsequent article) and others are all aiming to move up to the enterprise level and certainly they have major accounts using their products. The question is whether one or more of them can genuinely move into the enterprise space and challenge the big boys. While marketing obviously has a lot to do with it I am beginning to think that, at least from a technical (and price) perspective, the answer is yes.