Big data summary

Philip Howard

Written By:
Published: 6th September, 2013
Content Copyright © 2013 Bloor. All Rights Reserved.

I wasn't originally intending to write a sixth article in this series but I think it is worth a brief, formal wrap-up. I have discussed context, trust, security, ethics and integration but the overarching message is that big data environments in many ways mirror conventional data environments: they are not different in substance only in kind. However, users have to have confidence both in the data itself and the conclusions that are drawn from analysing that data: otherwise actions that should be taken will not be taken.

Having confidence means trusting that the quality of the data is sound and that the context within which the data has been analysed is also valid. In other words you need data governance just as much for big data as for conventional data. And, while some aspects of that governance will be simpler than it is for transactional data, the environment as a whole will be more complex (different strokes for different data folks) and will require a more agile approach. And the same applies to the integration environment, where multiple different approaches will be required depending on what is to be integrated.

Now, I don't mean to put you off. There are clearly lots of benefits to be derived from big data. However, it does need to be thought through: it is not simply a question of setting up a Hadoop cluster, throwing some data at it and waiting for the data scientists to tell you what to do—who, of course, will be infallibly correct. If this was the right approach they would be better called data magicians. They aren't and the actual truth is that "scientist" in this context is a misnomer: what they do is much of an art as it is a science and it needs to be treated in that context, with governance, management and understanding.     

Post a comment?

We welcome constructive criticism on all of our published content. Your name will be published against this comment after it has been moderated. We reserve the right to contact you by email if needed.

If you don't want to see the security question, please register and login.