Rock Stars, Big Data and Graph Databases

The Innovatenow team recently returned from Rock Stars Of Big Data 2013, a conference organised by the IEEE in Mountain View, CA. It was a great conference with a good line up of speakers, but we arrived there expecting it to be long on the real-life examples of “big data” uses and short on defining the discipline; we were wrong! The enthusiasm for big data was palpable, but there was general undertones of frustration about what big data is and what it means to “be doing big data”. Reflecting on a full day of discussion and debate, it is clear that more companies are actively seeking to harness their data assets than ever before; but for me as a concept, big data is like a Cappuccino sold in the conference venue – initially frothy but when you finally get to the expresso part it indeed packs a punch. In many ways finding a definition for big data is a semantic and academic exercise, but clearly the group are pining for a generally-accepted definition.

Similarly, last week an ex-colleague, a senior IT executive in industry, approached us for help in understanding “big data” opportunities in his corporation. Herein I was reminded about this pattern we found in California. My colleague had become acquainted in the technical aspects of big data – such as noSQL databases, Hadoop and Graph, but had not yet re-tuned his mind to the business of transformation and how to get practical benefit out of his work. My advice was to postpone the technology-focus and go back to basics. By seeking out areas where his company was under-stress, finding where market share is being lost and where operations are inefficient seems to be a better strategy. It is only then where data can support hypotheses and mathematical models can be built to model the business. From both recent experiences in California and closer to home, it seems there is an unhealthy focus on the means rather than ends.

Our first foray into the big data world was with our flagship product innovatenow ( The business problem here is how to give an organisation the right data to make their products better – meaning more profitable and attractive to their market. Turning this into a technology response, the problem became how to navigate a huge amount of complex data in a visually appealing way while always showing the path to greater insight. In this process, we realised that all data is intrinsically linked – for example, a Product has features; features come from business requirements and they in turn come from stakeholders. If we are smart about building the relationships, we can show them simply and effectively with the opportunity to progressively navigate the data set.

The perfect solution to our problems came with a well established technique using Graph Theory to build a series of “node-relationship” pairs which can be easily visualised. Once the data analyst has worked out how to extract the relationships from the databases in use, the visualisation layer takes over. Graph can be used starting with single node – say a customer and explore to the products he buys, the features he likes and make the link to new products the customer might want to try. You can see an example of the relationships we have built in the visual below.pngbase643637662a21c15468

As the Graph approach works effectively for data sets of any volume, it is debatable whether this will be seen as a big data approach, but is extremely effective at navigating complex data sets. After demonstrating our technology to users, they are amazed at how accessible their data realm is and in a single session, it provided ideas on how to integrate with other products (for example CRM) to get an unprecedented view of a client’s organisation. Interestingly, the unintentional side-effect of Graph is to drive some amazingly intuitive searches: by reading the relationships and node names, we have built an algorithm to navigate this complex graph and continually navigate around the graph to get the right search results incredibly quickly. For example, the see the Graph Search possible in a Product Management context; note the deepening of search results as the user navigates around the graph structure. This is possible with no custom coding required. Note the name of products have been subtly changed for the purposes of illustration (Mc into Mac)!

Let’s see everything with “Mac”

Type-ahead progressively searches for any data item with content consisting of “Mac”

Let’s go for “Mac Chicken Sandwich”…what is around this in the graph?

The user has selected the Mac Chicken Sandwich. It is now likely the user will want to explore data around the sandwich. Easy…just click the search bar to navigate around this point in the graph.

User is interested in “Projects relating to the Mac Chicken Sandwich”. Graph returns the most relevant pieces of data around this point

This can continue all day until the user finds the right data or insight


In summary, a lot of people get hung-up on the definition of big data and even more people think knowing the technology is enough to “do big data”. In reality, definitions are not that important – based on our experience, what is important is to find your own value proposition for data analysis (whether big or not), find your entry point to Big Data and never lose sight of the business. For us, Graph was the best first step for us; It is almost certain that this is not the same for everyone, but the first data discovered exercise should be to expose overlooked insights and hidden relationships between data. We will be sharing more information about our work on Graph in the coming days and weeks.

Register for the newsletter to be notified when this is available.