Planning Under Uncertainty: Agility in Machine Learning

“Even a poor plan is better than no plan at all.”

In business — especially in startups — we tend to want to be the first to move. But the above quote, from Russian chess player Mikhail Chigorin, is an important lesson that we should all keep in mind if we want to be around to finish the game.

That’s what this section is all about. We’re going to take the company strategy and product requirements you put together in the last section and turn them into an actionable development plan.

While a bad plan is better than no plan, let's aim a bit higher. A truly great plan must evolve over time, and I'm going to discuss the necessary tools to make that happen. This section will help you develop your initial plan, highlight the areas to watch, and demonstrate how to adapt when the time comes.

Case Study: Helixa

To introduce the planning techniques, I’m going to lean on some real examples that I know very well. Specifically, we will be studying the AI-powered market research platform developed here at Helixa, where I am the Chief Scientist.

The Helixa platform enables marketers and researchers to get global insights about the consumers that matter to their businesses, without running expensive research studies. A process that would usually take days or weeks is accomplished in minutes, thanks to an ensemble of machine learning and statistical models on the back-end.

Helixa’s technology can be summed up in three pillars:

1. The importance of privacy and responsible AI
2. The power of multiple data sources
3. The use of machine learning to generate human insights

The case for business cases

When we talk about translating the long-term strategy into actionable tactics, what we’re really trying to figure out is how to take the company’s current situation and move it in the direction of the desired strategy.

We should start analyzing the “here and now,” which includes current company assets and resources. These could include:

• • Human resources and expertise
  • Current products and infrastructure
  • Current customers
  • Data availability
  • Technology assets portfolio

We would then relate those with strategic objectives. Let’s suppose the top management defined the following goals:

• • Build advanced features based on text analytics
  • Add many different datasets into the platform
  • Beat competition in terms of variety of provided insights
  • Spread machine learning knowledge and data-driven culture throughout the organization

For the third step, we would then define business cases to fill the gap, as the missing link between the “here and now” and the “strategy”.

We could brainstorm some potential business cases, which might look like the following:

1. Implement a natural language processing (NLP) model for social conversations?
2. Add datasets about online behaviors?
3. Make operations more scalable?
4. Organize Machine Learning seminars?

Now we have a bunch of ideas to start with... and it’s time to make some decisions!

Conducting feasibility studies

Time-boxed feasibility studies are a common and valuable way to kick off the business case process.

A feasibility study should at least cover the following scenarios:

1. Legal feasibility: Would this new feature conform to terms of services and ethical requirements?
2. Economic feasibility: Does the cost-benefit analysis validate the feature?
3. Technical feasibility: Is the open problem solvable? How complex would the solution be if implemented?
4. Operational feasibility: Can the system continue to meet customer needs as it scales?
5. Scheduling feasibility: How long would the project need to run to return reasonable results?

Once a business case is validated, we can turn into a series of tactics that, together, make up a feasible actionable plan 

Laying out the roadmap

Generally, a company’s major deliverables and milestones are organized by quarter. So, you could start by using a timeline to map out the most relevant business case proposals alongside the outcome of their feasibility studies. 

This will provide the roadmap for each quarter and, ultimately, the full year. Let’s use the business cases from above to demonstrate. 

Mapping out the business cases

1. NLP model: The outcome of a feasibility study reveals that a stance-detection feature is more valuable than a sentiment model in the specific case of brand reputation analysis.
2. Additional datasets: The business requests a dataset on online behavior, and an e-commerce offerings fit the bill.
3. Scalable infrastructure: The engineering team is too busy. This is OK! If a feasibility study is not possible, leave a placeholder (see Q3 below).
4. Machine learning seminars: HR doesn’t have time for seminars, so we can then create an engineering blog — or a whitepaper.

All of these things need to happen, but they can’t happen all at once. Here’s what that timeline might look like.

It’s important to validate business cases in advance, whenever possible, because they have the potential to impact the roadmap and even the overall development plan.

Making it actionable

By this point, the initial roadmap is set and we know what we’re developing, as well as the priority for each item. Now, how do we break down the amount of work required in order to estimate ownership and effort for each team member?

A clear hierarchy could help us plan at different levels of detail and timelines. While business cases are generally planned across a few months, they can be broken down into epics, which are a few weeks each. Those epics can be further broken down into stories, which only last a few days. And even those can be broken down further into tasks of no more than a few hours.

Here’s what it looks like in practice.

The agile iterative plan

Everything we have discussed so far has moved somewhat linearly, like in the old waterfall approach: a single roadmap, validated business cases, and development work divided into epics and stories. 

Here’s where that changes.

Structured changes are good indicators that we are incorporating what we learn and proving our hypotheses like a Lean Startup. And you achieve them through organized sprints.

We want to organize the stories into sprints, lock the sprint scope, review the plan following each sprint, and iterate on all of the above as we move to the next one. The SCRUM project management methodology works well for these kinds of projects, where uncertainty is inevitable.

While your original strategy will certainly change over time, it’s crucial that you have a directional plan to reference throughout the project. It serves as a north star for both the dev teams and executives, in a way that makes communication simpler and more effective. It also allows teams to maintain consistency while designing on a bigger scale and implementing in smaller chunks.

The long-term plan represents the best guess of what will happen next at a given point in time. And it should be continuously revised and discussed, without fear of changing tactics or pivoting more radically.

The elephant(s) in the room

People tend to be overly optimistic during planning activities, and it doesn’t help that stakeholders are often pressing to accelerate the delivery of results.

As a consequence, technical debts tend to be naively underestimated or pushed aside completely.

"Technical debt is the implied cost of additional work caused by choosing an easy (quick) solution now instead of using a better approach that would take longer."  

SOURCE: Techopedia

Unfortunately, technical debts do not disappear over time. And, as in finance, pushing off the payment will incur compounding interest in the form of edits and reworks.

It is fundamental for engineers to avoid lazy solutions and strive for optimally designed, robust solutions. 

Similarly, it is fundamental for business stakeholders to properly estimate the impact of technical debts and allow enough capacity to repay them as soon as it is convenient.

When is the right time to pay debts? Just as the debt-ratio defines how much debt a company can leverage to finance their operations and growth, a similar financial analysis should be done to figure out how much a feature is worth today at the cost of making future development more expensive. 

In this article, the author suggests we measure technical debt as the ratio between the “remediation cost” and the “development cost.” 

SOURCE: Detroit Labs

Another common mistake of overly optimistic planning is to avoid accounting for failures. When your development process is carried out in a scientific way, you will require a considerable number of trials and errors. 

Neglecting those failures is a fast track to poor outcomes.

A realistic development plan

Now that I have briefly covered the “why” behind the different elements of your development plan, let’s get to the “how.”

When assigning stories to the timeline, I would suggest you start by dividing it into bi-weekly sprints. As you move further in time, leave placeholders for unpredictable activities, reserve enough capacity to fix technical debts, and account for failures that will need to be reworked.

A plan built for uncertainty

Now, you have an actionable plan for your first machine learning undertaking — or at least the tools and knowledge to build one.

We have identified business cases and tested them with feasibility studies, established and prioritized our road map, and discussed the ways we will adapt when the time comes.

That last part is the most important: You need to revise and adjust the plan continuously So, don't shy away from a detour or two, as long as they align with your strategy and move your team closer to the company's goals.

------------

Download Our Machine Learning Guide

The thought of getting started with machine learning can be paralyzing. There are a million different things to consider and just as many ways to get it wrong. 

To help you nail it the first time, our Chief Data Scientist captured all of his hard-won lessons from the past 10 years and poured them into our brand new, non-technical machine learning guide for managers.

Gianmario is the Chief Scientist and Head of AI at Helixa. His experience covers a diverse portfolio of machine learning algorithms and data products across different industries. He is also co-author of the book "Python Deep Learning", contributor to the "Professional Manifesto for Data Science", and founder of the DataScienceMilan.org community. Read more of his work at Vademecum of Practical Data Science.