Gutfeel advice: Make space for intuition in a data-dazzled world

As a statistician myself, I understand their reliance on data. Yet, I find myself wondering: Where do we, as data enthusiasts, get it wrong? Is it the quality of data, its insufficiency, or various data points collected failing to offer a meaningful picture?

In today’s world, data is produced at an unprecedented scale. Humanity generates around 2.5 quintillion bytes of data every day. A quintillion, to put it into perspective, is a one followed by 18 zeroes. 

Despite this overwhelming ocean of information, many decisions still go wrong. Why? Are we relying on the wrong data-sets, misinterpreting correlations, or erroneously assuming that more data equals better decisions?

Sometimes, I try to step into the shoes of my friends who have fully embraced data as the essential basis of every decision, from their work decisions and daily meals to calorie intake, travel plans, exercise routines and sleep cycles. 

They meticulously track every aspect of their lives with gadgets designed to measure things, count stuff and deliver data points. But then I wonder: What would they do if they took a data detox? No gadgets, no numbers. Would they lose their sense of direction?

How does one quantify or even qualify the smile of a child on the street, the joy of eating a wonderfully made pani puri, or the calm that comes from walking through a lush green garden? These experiences do not fit neatly into rows and columns. They elude classification. 

Yet, they are just as real, just as impactful. So, back to the fundamental question: How can someone hold a valid opinion or make a decision without sufficient data? Why do some experts insist that without numbers, it’s down to one’s instinct or gutfeel, which is less reliable?

Effective data gathering and analysis help decision-makers quantify, verify and understand complex issues that demand rational and insightful solutions. There is no denying the importance of data. 

It can dissect problems, identify patterns and provide clarity in a way that intuition cannot. But what happens when the issues at hand are abstract, or when the data is simply incomplete or irrelevant? 

That’s when intuition steps in. Our finely honed instincts, built on years of experience and understanding, provide a different form of insight that data cannot always capture.

Intuition is often treated as the opposite of reason. We tend to think in binaries, and in doing so, we define intuition as a vague and unreliable yardstick with no place in the age of science and data. But to rely solely on data as the final arbiter is to ignore the complexity of human experience.

The truth is, intuition is not the enemy of rationality; it is a loyal companion. It is not a sudden, baseless judgement, but the culmination of countless experiences that we have processed subconsciously over a span of time.

For data purists, anything outside the zone of data feels like a betrayal of logic. They view a decision made on gut instinct as a gamble, an irrational leap. Cynicism colours their view of those who, as they see it, make decisions blindly, without data to back them up. 

Yet, intuition is far from blind. It is simply a different way of knowing. When we make decisions that seem instinctual, we are often drawing from a deep well of past experiences, knowledge and emotions—factors that cannot always be distilled into numbers.

Marvin Minsky, a pioneer of artificial intelligence, argued that emotions are not separate from rational thinking. In his book The Emotion Machine, he makes the case that emotions are not irrational interruptions to thought, but one of the ways in which we think. 

They are essential to our decision-making processes, often guiding us when the data is insufficient, contradictory or simply unable to address the nuances of a situation.

There is elegance in recognizing the value of both data and intuition. The best decisions often emerge not from one or the other, but from an interplay between them. 

Data provides the scaffolding—the framework that supports our understanding of the world. But intuition adds the human element—the ability to sense what the numbers do not show and to interpret inputs from beyond a spreadsheet.

In a world awash with data, it is tempting to believe that every decision can be calculated and every outcome predicted. But data alone cannot capture the full richness of the human experience. Sometimes, it is the unquantifiable, the deeply felt, that provides the clearest insight. 

As we continue to generate vast quantities of data each day, we must remember that not every question has a numerical answer, and not every decision needs a data-set. Sometimes, the best choice is the one that feels right—not because it defies reason, but because it comes from a place that transcends numbers.

True wisdom lies not in the precision of data, but in knowing when to trust the silent truths that numbers cannot speak.