fear and loathing in the Hadoop Cluster

What I've noticed most on the business end of Big Data is the love/hate dichotomy.

I'm not talking about DW or BI folks who underestimate the promise of never having to throw data away, using commodity hardware, realizing the scaling efficiencies of the cloud, etc.  It will take time, but they are at least coming around.

I'm talking about folks who are actually scared of data.

Now, I'm a pretty open guy.  I actually like getting proven wrong, which happens an astonishing amount of the time.  It won't stop me from having an opinion, arguing for what I think is right, etc.  It is just the Socratic worldview that I employ.  My goal is to get better, learn more ... not be right.

As I've amassed a good amount of our working business data, I talk it up.  A lot.  To anyone who will listen.  What was shocking from my end were the reactions I often get:

1. "the data is wrong"
2. "our data is better" (even if only a small subset ), cousin of #1
3. "what you are looking at doesn't matter ... what really matters is X" 
4. "we can't possibly show that to end users without causing a thermonuclear meltdown"
5. <heads firmly implanted in sand>

People are pretty tied to their preconceived notions, and while I've had some very receptive and faithful allies across our business units there are a lot more people who'd rather I go away.  If my data doesn't prove their line of thinking, they'll do what they can to squash or ignore it.

I doubt I'm alone here.  

My guess is that while businesses want to be results driven, it sounds better in a powerpoint deck than in reality.  They need to be a bit more open to what they might find, even it it's not good.  I once got an "A" on a 20 page Math Paper in college for proving that I was an idiot.  

The future will not be won by the blissfully ignorant, but those willing to learn their way forwards. 

Business successfully leveraging BigData will follow an iterative, lean, and inquisitive mindset.  Twitter started out as a file sharing app, and Flickr as a game, Nokia making paper/galoshes, IBM actually made stuff.  Call it a pivot, iteration, rehash ... whatever you like.  

I call it winning.  

Big Data can help business known when and how to pivot, giving insight into black holes and unverifiable ideas.  It won't solve all our problems, but let's not dismiss what makes us uncomfortable.