Is Big Data too big for any one CIO to manage?

This was a question recently posted on a LinkedIn group on Business Intelligence. My response was as is posted below.

--- * ---

In terms of Big Data as it seems to be understood today, possibly, but it probably depends more on the size of the organisation, the organisation's priorities, and the experience, competence and guts of the CIO to ably push the frontiers.

My first completed and successful Big Data project constituted data (volume) in the terabyte scale within a multinational organisation, both structured and unstructured (variety), growing in some areas at almost 1000 rows per second (velocity). Within this, we generated an incremental $100 million in value in five months, after about eighteen months of planning and prep, with a team of only 25, and dare I say it, with 'normal' skills and people, a team who became supernormal once our hard work began paying off.

Interestingly, this was all done before the amazing variety of Big Data tools at your disposal today, which meant that pretty much everything we did, especially when going beyond the ETL processes of multiple transaction systems, such as with government legislative changes, regional economic development reports and other geographical information, frequent and dynamic customer surveys etc. still had to be done manually.

As a result of this Big Data success, from a business point of view, I advocate that the secret to making a Big Data project (or indeed any data project) work for you is not on making what is already complex more complex, but instead looking for simplicity. As an example, I advocate starting your project with a business goal, and using this goal to drive your technology goals and planning. I believe this will help simplify your initial approaches tremendously. Given the business goals, the project then becomes an easy sell to the board, to the various executive committees, and to various other stakeholders such as the floor staff and key account managers and most importantly, your customers. And it's the latter that determines whether or not your project is a success, not merely a successful but otherwise useless 'implementation'.

In my case, the business problem to solve was to improve our customer satisfaction rankings and to grow our business, and we worked from there. Trying to work from the point of view of "let's do Big Data", and because we're now doing Big Data, we need lots of money and new technology, and new techniques, and new terminology, and then trying to build business value (and ROI) thereafter is the wrong way round in my opinion, as it becomes a tool looking for a problem to solve, instead of a challenge requiring a tool to solve it. After all, change management 101 demands you start with a burning platform...

Did this all require a super power CIO? No. It required a super team of fellow adventurers with a penchant for exploration and on keeping an eye on the ball, instead of whether the ball was the right colour or round enough. Of course, all of this should be done within your existing information policies in terms of governance, security, permanence and so on. And if your policies and procedures are unsuitable in the new data milieu, build a case for them to be enhanced!