Data Science and Agile: a love-hate relationship ?

I shall perhaps start this article by distinguishing (albeit in a broad manner) between a discipline and a philosophy. 

A discipline is a branch of knowledge, a field of study, or an area of expertise. Whereas philosophy is the pursuit of wisdom, and thus a philosophy is a set of philosophical concepts, beliefs, teachings, doctrines, ideologies…

I think you’re already getting the gist of where I am going with this, right? Agile is a philosophy, It is not a methodology.

Agile is a philosophy that guides decision-making. It is not a methodology, much less a workflow or process.

Agile was developed in the 1990s as a counter-movement to Waterfall’s restrictions in software development. This culminated in the Agile Manifesto in the early 2000s.

The Manifesto stresses principles such as collaboration, adaptability, and simplicity. And data science operates on scientific foundations that guide its approach to analyzing data and extracting insights from it such as empiricism, testability, and cumulative knowledge. Ergo, one can imagine many ways in which Agile's principles can be assimilated into data science practices.

But if you sift through online discussion forums or talk to data scientists in your entourage, you'll see and hear some data scientists sharing horror stories about Agile. So where does the rift come from ?

A sea of misinterpretations

I’ve come across many misinterpretations, and used to have some myself, from which, in my opinion, a lot of misunderstandings and faulty implementations stem. But in this article, i’ll just list some common ones:

• Doing things faster, hence avoiding proper problem framing and project specifications and requirements.
• Going agile will help deliver more data science results. Going agile does not solve data value chain bottlenecks, and promising that can lead to big expectation misalignment.
• Breaking down data science work to thinly granular tickets. Hence, a backlog overload and a lot of time wasted on synchronizing the team's work instead of focusing on value steams. Remember: Agile is neither a euphemism for micro-management nor for laisser-faire.
• Emphasis on continuous delivery. Focusing on continuous delivery without taking a holistic approach to bridging Agile principles with data science workflow specifications can lead to delivering faulty products, which in turn might lead to mistrusting the data science team and its deliverables.
• Leadership dictating scope alone because data scientists are “just too technical”. Leads to misalignment in vision and misunderstanding what’s feasible. This can also lead to a frustrated and disengaged data team.

When implementing Agile methodologies, perhaps the crucial first step is minimizing misconceptions around them. By educating leaders, organizations can ensure that those in steering positions understand what these methodologies entail and where they start and stop benefiting their goals, their leadership and their teams.

With that being said, and as I hinted above, some Agile principles can indeed be adopted in data science work. 

Can Agile be adopted by data scientists ?

I don't see why not ! The adaptive and iterative approach of Agile can certainly complement and enhance the data-driven and experimental approach of data science. So I do think there is a lot of common ground we can leverage as data scientists and/or as data science managers.

In each card below, I'll summarize an Agile principle or belief, contrast it with data science practices' and projects' specifics, to then suggest some approaches that can combine the two.

Where does Agile adoption in data science projects stop working ? 

As we've seen, one can envision and implement many possible ways to assimilate Agile principles into data science practices.

However, there are still some areas where I don't see data science work benefiting from Agile methodologies.

When Data science + Agile becomes a methodological nightmare

Just for fun, I asked ChatGPT to suggest titles similar to mine. Some of its suggestion, I think, will illustrate what I'll talk about in this section:

Prompt: Suggest more titles like this: "When Data science + Agile becomes a methodological nightmare"
Picked results:
"Agile vs. Data Science: Balancing Speed with Accuracy"
"Scaling Data Science in Agile: When Flexibility Turns to Frustration"
"Navigating the Agile-Data Science Paradox: When Innovation Slows Down"

Catchy titles aside, there are elements I see as forced fits or ill fits when it comes to assimilating Agile principles into data science projects cycles or workflows:

• Forcing timelines to fit into fixed-length sprints;
• Adopting a product approach throughout all phases of a data science project cycle;
• Exclusively assessing quality proactively;
• Prioritizing delivery over comprehensive and extensive documentation.

I'll elaborate on each element one by one.

Unpredictable timelines vs Fixed-length sprints

The data science value chain includes many phases that can be unpredictable.

Problem understanding, for example, can sometimes span across weeks because of the necessity to federate different stakeholders from different business units (BU) and the difficulty to align on a scope for the project especially when BUs have different priorities.

Another issue I typically observe is the difficulty to scope data science projects. From my experience, the modeling aspect of data science projects makes it difficult for stakeholders, especially those who didn’t work closely with data teams before, to rapidly grasp the potential nuances that go into the scoping phase.

Also, the stochastic nature of data analysis, data exploration, modeling, and model training does not fit with the short and fixed-length nature of sprints. Sticking with such time constraints can pressure the data team to deliver under-developed or faulty products and foster a culture of delivery over quality.

Exploration & Research vs a Product approach

Agile methodologies were developed to orient product development in software value chains. While data teams strive to deliver data products as well, research and exploration underline a lot of the work being done. This can heavily impact the project's roadmap.

The product approach can also foster a culture of risk aversion, because data scientists will avoid experimenting with techniques, approaches or methods that can lead to better results but require time. 

In addition to that, data scientists will often require more periods of uninterrupted focus to deep dive into a project's specifics and complexities. Rituals, such as stand-up and unscheduled face-to-face communication can heavily disrupt this process, resulting in the opposite of what Agile intends, i.e. higher efficiency and productivity.

Quality engineering vs Quality assurance

One of the practices of data science is to build quality into the project or product from the start. This why data science revolves around quality engineering instead of quality assurance.

The unpredictable nature of data projects requires a more active and/or reactive approach rather than the proactive one of quality assurance.

Documentation maintenance & knowledge management

Agile believes in prioritizing working products instead of complete and comprehensive documentation. This would just be an irresponsible choice, in my opinion, in data science projects where a lot of decision making is taking place in terms of data engineering, feature engineering, modeling assumptions, experiment design, feature selection, modeling choices, ..etc 

Conclusion

Agile principles and methodologies can offer benefits to data science teams and projects, but they can easily turn into a "be careful what you wish for" situation if applied rigidly and not made to align with the specifics of data science work.

Leaders and teams should choose and adapt frameworks and methodologies to their context and continuously educate themselves, learn and test what best works for them.

Frameworks and methodologies are means to an end, puristic approaches will only hinder innovation and progress.

@article{nnz2024agileds, 
 title   = "Data Science and Agile: a love-hate relationship ?",
 author  = "NNZ",
 journal = "nonneutralzero.com",
 year    = "2024",
 month   = "Oct",
 url     = "https://www.nonneutralzero.com/blog/engineering-7/data-science-and-agile-a-love-hate-relationship-19"
}

​