When I started my career, I built a database of the open, high, low and closing prices for all commodity futures going back to 1920. At that time, it was considered one of the largest databases ever built. All of that data probably took up about 10 MB of data – yes megabytes. A few years later, I built a database of everything that the Government was buying, who they were buying from and all the technical characteristics of each item. This was a huge database – taking up an entire data center. However, the total size of that system was only about 12 GB. That being said, using current database software, we quickly realized that in order to make that data actionable, it required a different approach – and so began an exploration into creating technology that could provide sub-second response time to complicated queries.
Having learned how to make large databases actionable, I looked for an area where this technology could be better leveraged and discovered the world of database marketing. I joined a marketing services company and started building marketing systems clients which meant creating databases of all of their customers from multiple silos – multiple departments, brands and companies in order to create a 360 degree view of their customers. Mostly, the data consisted of millions of offline customer purchases for companies with tens of millions of customers. And while all of it had to be stored in a relational (SQL) database, access to even simple queries was measured in minutes, if not hours. In order to make it actionable, the data needed to be brought out of the database to achieve sub-second response time, thereby making the data truly actionable.
The data problem became much more intense as consumers started purchasing online and we now had to combine even more silos in an effort to better understand and predict customer behavior. Then the real data explosion began as web site behavior, email, SMS, Facebook, Tweets, and thousands of mobile transactions were added to databases which are now measured in not only terabytes but petabytes.
This new horde of marketing information now had an official name, Big Data. And not surprisingly companies realized that in order to make this data actionable, it had to come out of SQL databases and the world of NoSQL was born and new technologies with strange names (Hadoop, Cassandra, Mongo, HBase, MapReduce) were introduced on a seemingly daily basis. While these technologies work well for storing very large amounts of data and are great for discovering insight into customer behavior, in order to make it actionable, different techniques are necessary. Given the processing speed of computers and solid state drives today, extremely complex queries can be run in a few milliseconds using the right technology.
A few years from now we will look back on what was considered big data and laugh at how small it actually was – similar to how I laugh at the “huge” databases I created in the past and how they easily would fit on a flash drive the size of a fingernail. Nevertheless, it will be critical for marketers to focus on the actionable insight that Big Data can bring to the bottom line rather than focusing on just the insight. Data is undoubtedly the foundation for learning more about your customers, but transforming the information into actionable data is what will create value for your organization.