Even bigger data

"As big data transforms our lives — optimizing, improving, making more efficient, and capturing benefits — what role is left for intuition, faith, uncertainty, and originality?

"If big data teaches us anything, it is that just acting better, making improvements — without deeper understanding — is often good enough. Continually doing so is virtuous. Even if you don't know why your efforts work as they do, you're generating better outcomes than you would by not making such efforts. [Mike] Flowers and his 'kids' in New York may not embody the enlightenment of the sages, but they do save lives.

"Big data is not an ice-cold world of algorithms and automatons. There is an essential role for people, with all our foibles, misperceptions and mistakes, since these traits walk hand in hand with human creativity, instinct, and genius. The same messy mental processes that lead to our occasional humiliation or wrongheadedness also give rise to successes and stumbling upon our greatness. This suggests that, just as we're learning to embrace messy data because it serves a larger purpose, we ought to welcome the inexactitude that is part of what it means to be human. After all, messiness is an essential property of both the world and our minds; in both cases, we only benefit by accepting it and applying it.

"In a world in which data informs decisions, what purpose remains for people, or for intuition and going against the facts? If everyone appeals to the data and harnesses big-data tools, perhaps what becomes the central point of differentiation is unpredictability: the human element of instinct, risk-taking, accident, and error.

"If so, then there will be a special need to carve out a place for the human: to reserve space for intuition, common sense, and serendipity to ensure that they are not crowded out by data and machine-made answers. What is greatest about human beings is precisely what the algorithms and silicon chips don't reveal, what they can't reveal because it can't be captured in data. It is not the 'what is,' but the 'what is not': the empty space, the cracks in the sidewalk, the unspoken and the not-yet -thought.

"This has important implications for the notion of progress in society. Big data enables us to experiment faster and explore more leads. These advantages should produce more innovation. But the spark of invention becomes what the data does not say. That is something that no amount of data can ever confirm or corroborate, since it has yet to exist. If Henry Ford had queried big-data algorithms for what his customers wanted, they would have replied 'a faster horse' (to rephrase his famous saying). In a world of big data, it is our most human traits that will need to be fostered — our creativity, intuition, and intellectual ambition — since our ingenuity is the source of our progress.

"Big data is a resource and a tool. It is meant to inform, rather than explain; it points us toward understanding, but it can still lead to misunderstanding, depending on how well or poorly it is wielded. And however dazzling we find the power of big data to be, we must never let its seductive glimmer blind us to its inherent imperfections."