Ever since data science became an established business unit you hear about failures.

Classic software shops especially like to dip their toes into data science and machine learning projects. After a few, they deem data science as a failure and sometimes even abandon the entire team.

But how come?

Why does data science fail?

The name is the game. Science has an openness to results at its core.

In the worst case for a project, this can mean that a data set doesn't contain the information necessary to confirm or reject a business question. In data science management this would mean going back to the data acquisition stage, which would be ok. In software management returning to step 0 of the process would equal a catastrophic failure. Data Science at its core is iterative between stages, where later stages can inform changes in previous processing steps.

Additionally, due to its non-linear nature, it often happens that data science projects take longer than classically anticipated.

However, that's not all, often expectations are difficult to meet.

What if the answer is uncomfortable

Sometimes data science feels like it gets in the way.

Rumour has it that sometimes management does not like some answers a data science team extracts from the data. This happens particularly in offices with politics between higher-ups, where the data is supposed to confirm the hunch of a particular manager. If it doesn't, the project can be deemed a failure.

This may seem unfair, but can be disempowering to data scientists.

Unfamiliarity with the data science project

Putting a pure software project manager in front of a data project can be disastrous.

I may have sat in multiple meetings in different places, where the project manager was utterly surprised when I asked for the budget (time and money) for data labelling. Essential steps in the machine learning workflow that were unfamiliar to said manager.

This can of course happen with other essential steps that differ from software.

Managing data science projects can be difficult. Specialized knowledge is often necessary to succeed.

This atomic essay was part of the October 2021 #Ship30for30 cohort. A 30-day daily writing challenge by Dickie Bush and Nicolas Cole. Want to join the challenge?