Why is Nobody Talking about Small Data?
A Reminder that Essential Value Exists in our Little Columns and Rows
Dr. Jonathan Reichental, Chief Information Officer, City of Palo Alto, (@reichental)
–December 28, 2014
It seems everyone is focused on big data. And why not? Today the world is producing an extraordinary volume of data.
Our prolific machines and interactions are now venting a massive scale of data exhaust unprecedented in our short digital history. Big data is spinning up stunning visuals that are providing completely new understandings. Suddenly we’re embedded in the petabyte era. But doesn’t all data matter? Might our obsessive focus on big data come at the cost of data at the edges? That’s the small data. It’s the frequently used but largely unglamorous data that exists in every organization.
It’s the monochromatic and fundamental storytelling that remains largely untapped. It seems big data has become the main act and all other data has been relegated to playing support.
Big data really is a big deal
No doubt, capturing and analyzing massive scale data is changing the world. That’s not hype. Public safety teams can anticipate crime through predictive policing; public sentiment can be derived with uncanny accuracy on almost any topic; marketers can promote products to potential buyers with pinpoint accuracy; and we’re tantalizingly close to medical breakthroughs only imagined just a few years ago. These are just the tip of the iceberg of the places big data will take us.
An impressive new industry is emerging in support of big data. Remarkable new software tools and an army of newly minted data scientists have come to bear on a flourishing industry. The results are impressive. Everyone recognizes this and there is much work ahead of us.
With big data, the datasets involved have records in the millions, if not billions and likely much more. Our new capabilities mean we can store, organize, analyze, and attempt to make sense of it all. We acknowledge that a large number of organizations have datasets of this scale and they desperately desire the potential competitive advantage of having data science applied against it. But isn’t it possible that an equal number or more of organizations have datasets that fall well below this threshold and thus may be failing to realize the value in their small data? Will the brigades of consultants and the investments in new big data software pass by the magnitude of little opportunities that exist in every organization and every team?
Small data still runs organizations
Each of us who have, for example, worked in a business, served in government, or had a leadership position in a club, has used or been exposed to small data. It’s the spreadsheet, the contact list, the survey results. Small data is sometimes the by-product of big data, reducing it to tiny chunks that humans can understand. It’s the experiment results, the financial information, the queries, and reports that are so meaningful and so frequently requested. A legion of office workers toil over this data daily often hindered by incomplete skills and poorly understood productivity software. This is the data value that is so often only superficially gleaned. But it’s equally essential data that informs decisions every moment of the day.
Sure it doesn’t make the big headlines like big data, but it’s no less important.
Is the value of big data realized at the cost of small data?
Yet, we’re all enamored by big data. We’re simply not talking right now about small data the same way. Of course we have to continue the pursuit of unleashing the full potential of the former. There’s no doubt it’s changing the world and in the months and years ahead it will be decisive in helping to deal with issues of climate change; for our understanding of the Universe; for enabling astounding medical breakthroughs; and to help power our future Smart Cities. But let’s not redirect all our energy such that we lose sight of the innovation necessary in managing and understanding data in the nooks and crannies of every organization. Let’s also recognize that some of our big questions will be answered in small data. Critical signals in the noise are not limited to the big datasets. There are beautiful patterns in little queries if we know how to look for them.
The bottom line? A pat on the back to us. We’ve recognized the enormous and increasing value of data, inherently acknowledging that all data is important. We can’t ignore this fact in our pursuit of the new shiny thing. We have to continue our relentless innovation and skill building around data and we’ve got to make sure we’re including small data as we do that.