When AI recommends training programmes, what shaped the shortlist?

Steve New

An AI assistant asked for ‘the best online management training’ could return three impressive courses, each with a plausible explanation of what it teaches and who it suits. It might be a useful answer. But the question has not established whether the buyer wants an introduction to leadership, practical help with a first team, or a qualification they can use to demonstrate professional competence.

Those are different reasons to train, and they may lead to different kinds of programme being worth considering. A shortlist drawn mainly from one category can be coherent and informative while leaving another route out of view. If the buyer treats the answer as the relevant field of choice, they may never compare the kind of training that would have addressed their actual need.

This is an illustrative possibility, not a recorded AI response or evidence that anybody has been misled. The question I am interested in comes before the detailed comparison: what shaped the set of options the buyer was shown?

The question leaves more open than it seems

‘Online management training’ names a subject and a delivery channel. It says little about the job the learning needs to do. The person may already be managing people and want help with difficult conversations. They may be preparing for their first supervisory role, need a recognised qualification for career progression, or simply want a manageable introduction before committing to anything longer.

An assistant could sensibly ask which of those aims matters. It could also make a reasonable interpretation and offer a shortlist immediately. The buyer then receives something that looks like a comparison of management training, although the answer may really be a comparison of one particular kind of management training.

That distinction does not depend on knowing the model’s internal selection process. A visible shortlist tells us which programmes were presented, not whether every other programme was examined and rejected. What we can investigate is whether the choice set makes sense for the buyer's purpose.

Three different routes into management learning

The range of published programmes makes the problem concrete. Harvard Business School Online’s Leadership Principles covers personal leadership, team performance, coaching and feedback through an online course with cases and exercises. The Chartered Management Institute’s Level 3 First Line Management and Leadership qualifications are aimed at practising or aspiring supervisors and can be studied through approved centres. Henley Business School’s Coaching Skills for Managers and Leaders programme, offered as an in-house solution for organisations, combines independent online study with scheduled small-group practice and one-to-one mentoring on workplace conversations.

These are real learning routes that overlap in subject matter, and none has a monopoly on practical learning. But they involve different purposes and purchasing arrangements. An individual can enrol in Harvard's online course, while CMI qualifications are delivered through approved centres. Henley describes this programme as an in-house solution for organisations. An individual buying training for themselves and an employer arranging development for managers are not necessarily choosing from the same options.

Imagine the question in the opening produces a list dominated by broad online leadership courses. For somebody looking for an introduction, that might be a useful answer. Now suppose the person is already managing a team and needs to practise difficult feedback conversations with somebody who can observe and respond to what they do. The relevant comparison may shift towards training with repeated guided practice and feedback, not simply a different ranking of general courses. If their employer is arranging development for several managers, an in-house option such as Henley's could enter the comparison. If they are buying on their own, they would need to look for individually accessible practice-based training instead.

The issue is not whether an assistant should have named all three examples. It is whether the buyer’s actual purpose would have changed which kinds of programme deserved a place in the comparison. Sometimes an answer that contains only good courses is still an answer to a narrower question than the person intended.

Why the size of the shortlist matters

In a 2026 survey by UPCEA and Search Influence of 1,057 qualified adults aged 18-60 interested in online and continuing education, 56 per cent reported seriously considering three or fewer programmes before applying. Fifty-four per cent said they were very or extremely likely to use a dedicated AI tool for programme research. Among respondents at least somewhat likely to use AI for that research, half said they would use it to narrow their options.

These figures describe reported consideration and intended research behaviour, not observations showing that AI caused anybody's shortlist to shrink. Nor do they tell us how all professional-training buyers behave. They do, however, put the question in a real setting: many prospective learners seriously consider relatively few programmes, and some say they would use AI to help decide which ones to examine.

A generated list is also not a stable map of a market. In an empirical audit of six large language models across 100 product and service categories, researchers found substantial differences in the breadth and prominence of recommendations. That study was not about training enrolments, but it is a reason not to treat one answer as evidence that the named providers were the only plausible candidates.

There is a strong argument for keeping lists short. An assistant that gives somebody twenty loosely relevant courses may have saved them little work. A person who needs flexible study around shifts has no reason to inspect every programme with compulsory live sessions, and an employer organising a team intervention may reasonably rule out courses designed for individual self-study.

The useful distinction is between narrowing the choices for a reason that reflects the buyer’s needs and narrowing them around an assumption the buyer has not examined. An assistant can ask a clarifying question, explain the kind of programme it is prioritising, or show a couple of materially different routes before comparing providers. A buyer can also ask for those alternatives. Adding more courses of the same kind is not necessarily as valuable as reconsidering which kind belongs on the list.

What does an omitted provider actually tell you?

For a training provider, being absent from a generated shortlist can look like an obvious marketing problem. But that observation does not tell us whether the programme was suitable for the question, whether the system found the relevant source material, whether another phrasing would have produced a different answer, or whether a prospective buyer ever saw the result. It certainly does not establish a lost enrolment.

I would start with the questions suitable prospective learners actually ask, where that evidence is available. Are they searching for a qualification, a different way to learn a skill, help with a workplace problem, or reassurance about whether they are ready for the programme? Then I would examine the kind of training the AI answer puts in front of them and how that changes when the intended outcome is described more precisely.

Suppose a provider's real strength is coached practice with feedback for new managers, but its public materials mostly promise to ‘transform leaders’ and ‘unlock potential’. If plausible questions about learning to handle difficult workplace conversations consistently bring back only general leadership courses, I would look at how clearly the provider explains its own approach. The format, the opportunity to practise, the person it is designed for and the time it requires may not be legible enough. Making those distinctions clearer could help a human buyer judge the offer, whether or not it changes a particular AI response.

A different case might call for no page change at all. The provider may already explain those things clearly, while the assistant draws on other information, the prompt favours a different kind of training, or a few sampled answers simply vary. The programme might genuinely be too intensive, too specialised or otherwise wrong for the person asking. Getting it onto that buyer's shortlist would not improve the decision.

If the possible exclusion looks consequential, I would compare the buyer’s question, the answer they saw and the original descriptions of the relevant programmes. Where possible, I would also find out which alternatives the buyer understood to exist and whether learning about a different route would have changed their consideration. A handful of generated answers may show a mismatch worth pursuing. They cannot show how often it happens or what commercial result would have followed without it.

The response should match what that evidence can justify. It may be worth describing a distinctive learning format more plainly, correcting conflicting information or making it easier to distinguish an introductory course from guided workplace development. It may also be sensible to leave the site alone. A large AI-visibility project built on the fact that one programme was missing from a few lists is not yet a diagnosis.

The choice behind the comparison

A good shortlist should not give every provider a chance to appear. Its value is helping somebody consider a manageable number of genuinely relevant routes. That requires getting close enough to their purpose that the right kinds of training are in view, and being willing to change the frame when the purpose becomes clearer.

Before asking why a particular provider was not recommended, I would first want to know what the buyer was trying to achieve and whether the shortlist left out a materially different way of achieving it. Only then does comparing the named programmes, or investigating an omitted one, become a useful next step.

Selected original sources