B2B AI PlatformB2B AI Platform
Making AI Predictive Modeling Work
B2B AI Platform

Making AI Predictive Modeling Work

Emma RobertsWords by Emma Roberts

AI predictive modelling requires a single short page to function: The action a score triggers, the person who possesses that action, and the cost of being wrong in each direction.

That is the precondition. All other parts are located below it. Model selection, platform selection and vendor selection are all reversible, but no choice of these fixes a missing decision. Teams that skip the page end up with a well-engineered route to a dashboard no one opens, and they usually find out when the subscription comes up for renewal.

The precondition, and the two tests that prove it is met

The page has three lines and a signature.

The action is written as a sentence a new starter could follow without asking. The owner is listed as a job title with a name beside it. The cost decreases twice, once for a false positive and once for a false negative, in money or in hours. If those two figures sit close together, the analysis is incomplete.

Three decisions behave differently, and the numbers should look different for each.

The fraud decision has its asymmetry in money. A false negative moves funds out of the business. A false positive costs a review and some customer friction.

The retention decision has its asymmetry in time. A false positive spends an incentive on someone who was staying anyway. A false negative loses the account and every future year of revenue linked to it.

The safety or clinical decision has an asymmetry measured in harm. A false negative can injure a person, and a longer review queue is an acceptable price for catching it.

Now run two tests. First, imagine the score lands tomorrow at 0.8 and no one changes anything. If that is acceptable, stop the project. Second, imagine switching the model off next Tuesday. If no user complains, the output never reached a workflow.

Step 1. Define the outcome, then version the definition

Write the outcome as a definition before you write it as a data field, because definitions move below models.

The road map for clinicians to develop and evaluate AI predictive models, published in BMJ Health Care Inform, makes this its opening stage for a reason. Sepsis is the example it uses: several competing definitions exist, each sitting at a different distance from the underlying syndrome, and the choice changes what the model predicts and how its performance figures should be read. The same applies to a churned customer, a failed payment or a fraudulent claim.

The paper also notes that definitions vary between guidelines and change over time. It treats model upgrading as routine work.

Version the definition with a date. When the definition changes, the model is stale from that date. The drift alarm fires later.

You should have the required materials before the next step: A single sentence, a single version number, and a single date are signed by the person responsible for the outcome.

Step 2. Pick the metric from the error cost

The metric is a business decision wearing statistical clothes, and the wrong one will pass a model that should have failed.

The Domino guide to benchmarking predictive models is blunt on this. It argues that AUC is a weak and often misleading indicator of real-world performance, because it summarises a model across every threshold while production only ever runs one.

So set the threshold from the cost ratio written on the page in the precondition, then choose the metric that behaves at that threshold.

You should have the required assets: the metric name, the threshold value, and both error costs expressed in the same unit.

Step 3. Set the baseline you have to beat

A baseline is the cheapest honest answer to the same question, and it is the number every later model has to justify itself against.

The Domino guide recommends keeping it simple: k-nearest neighbours or Naive Bayes on a minimally prepared dataset for classification, and an exponentially weighted moving average for a time series. These capture little complexity. That is the point. They give you a floor and they exercise the pipeline that will carry the real model.

What you should have: the baseline score, measured on the same data, at the same threshold, with the same metric.

Step 4. Build the dataset and prove it looks like the population you will score

Data quality is the load-bearing wall.

The road map from BMJ Health Care Inform sets out the practical sequence for UK data. Check relevance, size and diversity first. Then check that the proposed dataset represents the population where the model will be used, because a mismatch bakes bias into the result. The Gateway is a single entry point for requesting access to UK health datasets stored by the nine health data research hubs, and it flags de-identification as genuinely hard for demographic variables that are themselves useful predictors. Patient information is protected by the General Data Protection Regulation, and patients can refuse through the national data opt-out service.

Domino's guide adds the failure mode that catches experienced teams: leakage. A naive train and test split on a time series leaks the future into training, and complex models exploit that leak better than simple ones, so the benchmark ranks them backwards. Exclude everything that would not have been known at the moment of prediction.

You should have the required materials: the population definition, the date range, the exclusion list, and a written statement of what the model could not have seen.

Step 5. Choose the model family from the shape of the question

The family follows the outcome. Four are worth knowing, and each is a different answer to a different question.

Regression models predict a continuous number from one or more predictors. Use them when the answer is an amount, a duration or a rate, and when the relationship between inputs and output needs to be readable by the people who act on it.

Classification models assign each case to a class. Use them when the answer is a yes or a no, a risk band or a routing decision, and expect to choose the limit deliberately, because the threshold is where the money is.

Time-series models read data in sequence and forecast the next value from what came before. Use these for demand, capacity and any case where the sequence of observations provides information a row-by-row model would discard.

Clustering models group cases by similarity without labels. Use them when no one has told you what the categories are. Treat the output as an input to further analysis, and let a later stage make the decision.

OutSystems sets out this taxonomy of predictive models, along with the machine learning algorithms behind it, including random forests, gradient boosting and neural networks for problems where the relationships are genuinely complex.

What you should have: the question written in one line, and the family named as the answer to it.

Step 6. Benchmark in a fixed environment, with a seed, and record the operational numbers

A benchmark that measures only accuracy will pick a model that cannot be run.

The Domino guide makes the case for containers. These containers keep the data, the library versions and the compute configuration constant, so runs are comparable and reproducible, and they let you record memory, storage input and output, and processor use alongside the score. That matters, because a model that misses its latency budget has failed the business requirement no matter how accurate it is. The guide also warns about thread oversubscription, where a multi-threaded maths library and a parallel resampling strategy compete for the same cores and wreck the run. Set the thread count explicitly, and measure the run time as a first-class output.

Set the random seed before every run. Most random number generators support it, and without it the benchmark is not reproducible.

What you should have: the container image, the seed, the latency and memory figures, and the benchmark score measured at the production threshold.

Step 7. Present the prediction where the decision is made

A score in a dashboard is a score that no person uses.

The BMJ road map treats presenting and interpreting the prediction as a stage of its own. It notes that much of the difference in whether benefits are realised relates to whether the predictions are given to clinicians in a timely way that enables them to take an appropriate action. The same rule applies in a business process. The prediction has to arrive at the moment of decision, it has to carry the threshold that turns it into an action, and it has to show the reason codes that let a person question it.

Design the alert so that a false positive costs the least possible attention. A score with no linked action trains people to ignore it.

You should have the necessary requirements: A mock-up of the screen or the message shows the action it recommends and the reason codes linked to the top three features.

Step 8. License and maintain the model as an asset

A predictive model is a maintained asset with a retirement date.

The BMJ road map separates the legal right to use a model from its maintenance, and both stages have to be planned before the model is deployed. Licensing covers who may use the model, where it may run, and what happens when the underlying data changes. Maintenance covers drift monitoring, retraining, and performance reporting.

Write the retirement date before you write the launch plan. A model that has no retirement date accumulates technical debt and regulatory exposure.

What you should have: The authorization terms, the person who maintains it, the drift metric and its alarm threshold, and the retirement date.

Step 9. Evaluate the impact after the model is live

Model accuracy and business impact are different measures.

The BMJ road map ends with ongoing evaluation of the impact of the AI predictive model. Its conclusion is direct: the downstream effects on processes and outcomes of AI prediction models vary widely, and it is essential to evaluate the use in clinical practice using an appropriate study design. In a commercial setting, that means a before-and-after comparison or a controlled rollout, not a dashboard of model metrics.

Measure the decision that changed, the cost that moved, and the harm that was avoided. If the model is accurate and the decision is unchanged, the project has failed.

You should have these items: the evaluation design, the baseline period, the primary outcome measure, and the date of the first review.

The page you keep

The precondition page is the asset. The notebook, the model and the dashboard are the disposable parts. Run the steps in the order given. At the end of Step 9, you will have a model that someone uses, a cost that someone can defend, and a retirement date that someone has written down.

Sources