Success with Big Data

My success with big data started when I led a world class team in a large international bank in 2005, way before the subject even had a name!

Gartner now describes big data as follows: "Big data are high volume, high velocity, and/or high variety information assets that require new forms of processing to enable enhanced decision making, insight discovery and process optimization." TBDI defines big data similarly, as voluminous structured, unstructured or a semi-structured data, including sources internal or external to an organization, generated at a high degree of velocity with an uncertainty pattern.

Did our project qualify as big data? Oh yes it did!
VOLUMINOUS?  Terabyte scale (exabyte in the operational systems)
HIGH VELOCITY? Banking transactions were being generated at over 700 transactions per second, requiring a re-analysis of trends and patterns at daily intervals in some cases to maintain their validity. Analytics on large data sets is a considerable challenge.
HIGH VARIETY: We integrated structured and unstructured data, including bank customer transactional data, carefully remodelled internal and external market research data, government publications, carefully remodelled macro economic and regional economic data, remodelled tax data, remodelled GIS data and even securities exchange index data.

Figure 1: Really big data in action - analysing what our banking customers were spending their money on by integrating economics with customer data, and incidentally noticing the impact of spend on the share price of the providers of those goods and services 

With this we created a successful, truly world class customer experience strategy that generated considerable returns for the bank (over $100 million in incremental value was generated for the bank in five months), survived international peer review and was published and presented to an international audience in Canada in 2008, as well as being presented to a Teradata user group conference in the USA in 2008! Big data still didn't even have a name by then!

Figure 2: Coordinating the execution of a big data initiative is complex beyond just because you're managing interfaces outside your organisation, but because there are so many more stakeholders to manage expectations with. The key thing is to keep your eye on the goal, which in our case, was to increase customer engagement and our sales hit rates

I should start leveraging this knowledge and experience! A key lesson is not only to see big data as an IT challenge and as a component of an IT strategy alone, but to see it in parallel with marketing strategy. Integrating the thinking from the start will ensure the best results with minimal wasted effort and expense.