Mike Loukides recently recapped a conversation we’d had about leading indicators for data science efforts in an organization. We also pondered where the role of data scientist is headed and realized we could treat software development as a prototype case.
It’s easy (if not eerie) to draw parallels between the Internet boom of the mid 1990s and the Big Data boom of the present day: in addition to the exuberance in the press and the new business models, a particular breed of technical skill became a competitive advantage and a household name. Back then, this was the software developer. Today, it’s the data scientist.
The time in the sun improved software development in some ways, but it also brought its share of problems. Some companies were short on the skill and discipline required to manage custom software projects, and they were equally ill-equipped to discern the true technical talent from the pretenders. That combination led to low-quality software projects that simply failed to deliver business value. (A number of these survive today as “repair-ware” that requires constant, expensive upkeep.)
How will the data science field avoid software development’s pitfalls? (As an aside, we shudder to think what would be the data science equivalent of “repair-ware.”) We started to explore some ideas but realized they were all rooted in education, business value, and openness:
Company leaders must educate themselves in order to understand how data analysis can improve their firm. That knowledge will guide them in building out a data science team and establishing its mission. Leaders otherwise risk trivializing the data scientist role or overindulging in analytics for the sake of analytics.
In turn, data scientists must understand how their work is meant to improve the business. That will serve as their compass when they explore new ideas so they can aim to deliver solid value. Without that guidance, it’s too easy to get stuck in rabbit-holes and yak-shaving.
Both parties must be vigilant of any needless barriers forming around the data science team, especially after the initial novelty fades. Open communication between the data science group and the rest of the business will ensure the former doesn’t land in a separate silo, marginalized out of the company mission.
These ideas may be a start, but are they enough? Probably not. What would you recommend to steer data science clear of pitfalls?