I. AI Returns Based on ROI

First of all, before defining ROI, we need to understand why companies use this indicator. Among other things, it allows knowing if a project is economically viable. The interest is not necessary to have the highest value, everything depends on the strategy of the company and the goal associated with the project.

For example, it may serve as an investment in training teams to learn new technology. As a result, the primary goal is not economic profitability, but knowledge acquisition, so the associated ROI may not necessarily be very high.

ROI also makes it possible to prioritize projects and request funding. The great advantage of this indicator is that it is understandable by any decision-maker because its value removes all the technical aspect that is induced. However, this measure must be taken with caution, because the theoretical ROI determined at the beginning of the project is often quite far from the real one.

To more accurately estimate the final value, this indicator is defined at several stages of its development. Most companies use an iterative process that allows them at the end of each sprint to redefine this metric and take action based on its outcome.

However, it is sometimes complicated to determine this indicator. There are a lot of variables that can come into play, and many believe that the accuracy of the estimate is highly dependent on the technological maturity of the company. To solve this problem, most companies also determine business KPIs that allow them to have a quantifiable objective that is easier to track.

II. AI Performance Based on KPIs

Why Determining a KPI?

As explained earlier, ROI is not always the best indicator to track a project. It is therefore important to understand why individuals within companies use KPIs as an additional measure.

1- Understandable Indicator:

It allows to have a value that everyone can understand and that does not require an understanding of AI. This makes it easy to track the effectiveness of the project for both business leaders and business employees.

2- Realistic Indicator:

The ROI as explained earlier is sometimes very complex to calculate. The use of a simpler indicator helps to overcome this problem of definition. In addition, defining a KPI and its associated objective helps avoid tracking an ROI that is sometimes too far away from reality.

3- Defined Goal:

The objective linked to the KPI makes it possible to define the project’s temporality. Indeed, not having a measurable goal makes it impossible to know when the desired maturity is reached. It is therefore impossible to stop the project because there is no indicator of success, the cost of development is mainly impacted by the time it takes.

4- Knowing Required Efficiency:

With algorithm optimization, knowing how well the AI should perform is part of the ROI optimization equation. Quantifying your goal allows you to measure the effort required to reach it as efficiently as possible.

However, too many projects start without defining this objective. This is why some consider the KPI to be a key element in validating the launch of a development project because, without it, it is impossible to quantify its success. In addition, the definition of this measure has the advantage of allowing decision-makers to continuously monitor the progress of the project.

Who determines the KPI?

Now that we have determined why it is so important to track this metric, we need to understand who determines it. To do this, business teams need to address it for several reasons.

On the one hand, to ensure that the indicator is understandable and correlated with the need. This approach is relevant because those who really know the company’s strategic objectives are the ones who know which indicators are important to track. In addition, asking business teams to determine this metric allows them to be involved in the project and facilitates the dialogue with the team that develops the solution.

On the other hand, the IT teams are also involved in determining this objective, as they are there to validate the feasibility and viability of tracking the KPI. They determine what data is needed and verify its access for the study of this indicator. This IT approach is important because sometimes some data are not accessible. This knowledge of the data, access, and feasibility enables an exchange with the business teams to find other solutions if necessary.

How to measure the effect of AI on the KPI?

Having a KPI to track is great, but knowing how to measure the impact of AI on it is just as important. Indeed, the addition of this technology can sometimes be one factor among others in the evolution of a metric. Indeed, other factors can also have an influence such as seasonality, promotions or actions taken by the competition.

First, the business must determine when the use of the algorithm will have an impact on the indicator. To determine its effect, the method used by the majority of people is A/B testing. This method consists of comparing the evolution of the KPI on a target population that uses the AI and one that does not. This approach makes it possible to isolate the contribution of the algorithm in relation to the variations of the indicator. This technique quantifies the effect of the project allowing to validate the achievement of the objective and thus indirectly the ROI.

Depending on the company’s strategy, different KPIs are monitored. For example, for some companies, AI is used to improve the customer experience, to do this the Net Promoter Score (NPS) is tracked. For others, this measure corresponds to the evolution of their retention rate. Whether it is the conversion rate of an advertisement or the increase in sales, it is always possible to use A/B testing in one way or another.

III. ROI Timeline

Two approaches are possible to determine the ROI timeframe. Since ROI is strongly impacted by the development time, it is important to have an idea of the ROI timeframe.

On the one hand, if the approach used is to achieve a specific goal, time will be the variable to adjust. The majority of the people use a method consisting of iterations to deliver several more or less completed versions of the project. According to peoples, the iteration times differ, nevertheless, it is still possible to have average estimates:

• For the realization of a “Proof of Concept” (POC) to verify the usability of AI, it takes an average of 2 weeks to 1 month.
• To realize a “Minimum Viable Product” (MVP), for small projects, it can take between 2 and 3 months and for larger ones, it can take up to 6 months.
• On average, the time between iterations is 3 months. However, the shorter the iterations is, the more effectively it will allow to verify if the solution meets the needs. Moreover, it is at this time that the ROI is recalculated.

According to different sources, the final ROI deadline and the achievement of the KPIs are determined after 8 to 12 months. This means that on average the development time taken into account when calculating the ROI is 8 to 12 months.

On the other hand, the second approach is to define a period for the project. This allows you to indicate exactly how much development time will facilitate the quantification of the ROI. Then, the project teams define what is technically possible to do during this time frame. With this approach, it is easier to use an agile methodology with sprints by defining precisely the objectives for each sub-period. Moreover, it allows for easier correction of the objectives, because the different stages are clearly defined from the start. The last major advantage of this approach is that there is a real deadline for the project, regardless of its success. This avoids having infinite developments thinking that the team is always close to the objective without ever reaching it.

In the scientific literature, the issue of the ROI deadline for AI projects is not clearly addressed. According to Zif and McCarthy (1997), the ROI deadline for an R&D project is two years. However, responses from the different interviews show that the timeframe for AI projects is more like one year. However, this difference may be due to the more iterative approach that has been introduced in recent years, which allows the need to be met more quickly.

---

Canhoto, A. I., & Clear, F. (2020). Artificial intelligence and machine learning as business tools : A framework for diagnosing value destruction potential. Business Horizons, 63(2), 183‑193. https://doi.org/10.1016/j.bushor.2019.11.003

Megler, V. M. (2019). Managing Machine Learning Projects Balance Potential with the Need for Guardrails. Amazon White Paper, February.