Why organisations need fewer perpetual experiments and a stronger capacity to turn technological possibilities into institutional decisions.
The Comfort of Experimentation
One of the most reassuring words in the contemporary language of artificial intelligence is pilot.
It suggests innovation without recklessness, experimentation without excessive commitment and progress without the risks associated with large-scale transformation.
A pilot sounds sensible. Responsible, even.
An organisation identifies a promising technology, selects a limited use case, involves a group of enthusiastic professionals and tests whether the proposed solution works.
The results are presented. Participants share positive experiences. A few productivity improvements are reported. Senior management receives a presentation demonstrating that the organisation is actively exploring artificial intelligence.
Everyone can reasonably conclude that something has been achieved.
And frequently, something has.
But what happens next?
Sometimes very little.
The pilot remains a pilot. Another experiment begins. A different department launches a similar initiative. A new model becomes available, encouraging another round of testing.
The organisation accumulates demonstrations, presentations and technical experience.
Yet its processes, responsibilities, knowledge structures and decision-making arrangements remain substantially unchanged.
Experimentation becomes visible. Transformation remains elusive.
This is not necessarily a failure of technology.
The models may perform well. Users may appreciate the tools. The results may genuinely demonstrate potential.
The difficulty lies elsewhere.
An organisation can become increasingly sophisticated at experimenting with artificial intelligence without becoming particularly capable of deciding what to do with it.
And this distinction deserves much greater attention.
The Difference Between Demonstrating Capability and Creating Value
Over several decades of technological change, from early computing and the Internet to mobile communications, digital platforms and today’s artificial intelligence, one pattern has repeatedly emerged.
Demonstrating that a technology works is not the same as demonstrating that an organisation can benefit from it.
A successful technical experiment establishes that something is possible under particular conditions.
A successful organisational transformation establishes that the capability can be integrated into the way people work, decisions are made, responsibilities are distributed and value is created.
These are fundamentally different achievements.
Consider a generative AI assistant that can summarise documents, answer questions or produce reports.
A pilot may demonstrate that the assistant generates useful responses and reduces the time required for certain activities.
That is encouraging.
But the organisation must still answer a different set of questions.
Where will the information come from?
Who is responsible for its quality?
How will the system handle uncertainty, conflicting evidence or outdated knowledge?
Who will verify its outputs?
What happens when employees begin relying on it?
How will existing procedures change?
Which responsibilities remain human?
And perhaps most importantly, what organisational outcome will actually improve?
A reduction in the time required to produce a document may be useful.
But if the document still passes through the same approval process, requires the same corrections and supports the same decisions, the overall benefit may be considerably smaller than the pilot suggests.
The technology has accelerated an activity.
It has not necessarily improved the system in which that activity takes place.
A pilot proves a capability. Transformation proves its value in context.
Confusing these two achievements is one of the reasons organisations struggle to move from experimentation to adoption.
Figure 1 – Technical capability becomes organisational value through ownership, adoption, governance and redesigned workflows.
When the Pilot Becomes a Comfortable Destination
There are legitimate reasons to conduct AI pilots.
Artificial intelligence introduces uncertainty. Models behave probabilistically, performance varies across tasks and organisational consequences can be difficult to predict.
Testing technologies in controlled environments is therefore an important part of responsible adoption.
The problem begins when the pilot loses its temporary nature.
An experiment should be designed to answer a question.
If the question remains undefined, the experiment can continue indefinitely without producing a meaningful conclusion.
An organisation might begin by asking whether generative AI can improve knowledge retrieval.
But what would count as success?
Faster responses?
More accurate answers?
Reduced dependence on particular experts?
Better access to organisational knowledge?
Lower operational costs?
Improved service quality?
These objectives are related, but they are not interchangeable.
A system might retrieve information quickly while providing insufficient evidence to support its answers.
It might reduce the workload of one team while increasing verification responsibilities elsewhere.
It might improve the experience of experienced users but create new difficulties for those unfamiliar with the underlying subject.
Without an explicit definition of the problem and the desired outcome, almost any positive result can be presented as evidence of success.
This creates what might be called the pilot trap.
The organisation demonstrates activity, celebrates experimentation and postpones the harder decisions required for transformation.
The pilot becomes a comfortable destination rather than a stage in a deliberate process.
And because pilots are generally associated with innovation, questioning their continuation can appear less progressive than launching another one.
Yet responsible innovation also requires the ability to conclude that an experiment has fulfilled its purpose.
Sometimes the correct decision is to scale.
Sometimes it is to redesign.
And sometimes it is to stop.
All three can represent successful organisational learning.
The Missing Decision
The most important moment in an AI pilot may not be the demonstration of the technology.
It may be the decision that follows.
This is where experimentation becomes organisational intelligence.
A pilot should reduce uncertainty sufficiently to support a choice.
It should help the organisation understand what works, under which conditions, at what cost and with which consequences.
It should also reveal what remains unknown.
The decision may be to expand the initiative because the evidence supports broader adoption.
It may be to redesign the process because the technology performs well but the surrounding organisation is not ready.
It may be to continue with a narrowly defined additional experiment because a critical uncertainty remains unresolved.
Or it may be to discontinue the initiative because the expected benefits do not justify the investment, risks or complexity.
What matters is that the decision is explicit and supported by evidence.
Figure 2 – Every AI pilot should generate evidence for an explicit decision: scale, redesign or stop. Further testing should address a clearly defined remaining uncertainty.
An experiment that ends with a justified decision not to proceed is not a failure.
It may have prevented the organisation from making a costly mistake.
By contrast, a technically impressive pilot that never leads to a decision may have created little lasting value.
This distinction is particularly relevant to public institutions, universities and other complex organisations where innovation initiatives must coexist with established procedures, multiple stakeholders and demanding accountability requirements.
The challenge is not simply to demonstrate technological possibilities.
It is to connect those possibilities with institutional priorities, operational realities and legitimate decision-making processes.
The opposite of innovation is not caution. It is experimentation that never changes what the organisation knows or does.
The Illusion of Progress
Artificial intelligence is particularly susceptible to this problem because its capabilities are immediately visible.
A chatbot can answer questions.
A language model can produce a report.
An agent can execute a sequence of tasks.
A generative video system can create material that would previously have required substantial specialist effort.
These demonstrations are compelling.
They produce an immediate sense of progress.
And because the technology evolves rapidly, there is always another capability to explore.
A new model may outperform the previous one. A new tool may offer improved integration. A new agent framework may promise greater autonomy.
The temptation is to continue experimenting with the next technological development rather than confronting the organisational implications of the previous one.
This creates a paradox.
The faster AI evolves, the easier it becomes for organisations to remain permanently in the experimental phase.
The external pace of technological change provides a continuous justification for postponing internal transformation.
Why redesign a process around today’s model when a better one may arrive in three months?
Why establish governance arrangements for a system that may soon be replaced?
Why invest in organisational capabilities when the technology itself is still evolving?
These questions are understandable.
But taken too far, they create a form of institutional paralysis disguised as technological dynamism.
The organisation appears active because it is continuously testing new solutions.
Yet the decisions that would turn those solutions into durable capabilities are repeatedly deferred.
The answer is not to ignore technological evolution.
It is to distinguish between decisions that depend on a particular model and decisions that concern the organisation itself.
Models will change.
The need for reliable knowledge, clear accountability, appropriate human judgement and coherent processes will remain.
An intelligent organisation should be able to experiment with changing technologies while developing stable capabilities for evaluating and integrating them.
What Should an AI Pilot Actually Measure?
The measurement of AI pilots often begins with technical performance.
Accuracy, response time, model quality, user satisfaction and task completion are all relevant.
But they describe only part of the picture.
A pilot intended to support organisational transformation should examine at least five dimensions.
The first is the operational outcome.
What changes in the actual process?
Does the system improve service quality, reduce errors, shorten end-to-end completion times or enable work that was previously impractical?
The important measure is not simply whether an individual task becomes faster, but whether the wider process improves.
The second is human impact.
How does the technology change the experience of employees, professionals, customers or citizens?
Does it support judgement or encourage excessive reliance on automated outputs?
Does it reduce unnecessary work or introduce additional monitoring and verification responsibilities?
Does it make expertise more accessible or diminish opportunities for learning?
The third is economic sustainability.
What is the full cost of operating the solution beyond the experimental environment?
This includes infrastructure, integration, maintenance, security, data preparation, training, monitoring and the organisational effort required to keep the system reliable.
A pilot may appear inexpensive because many of these costs are temporarily absorbed by enthusiastic participants or existing technical teams.
Scaling exposes costs that the experiment did not fully represent.
The fourth is risk and accountability.
Can the organisation explain how the system is used?
Are responsibilities clear?
Can outputs be verified?
What happens when the system produces incorrect, incomplete or misleading information?
How are privacy, security, compliance and operational resilience addressed?
The fifth is organisational readiness.
Does the organisation possess the skills, knowledge structures, governance arrangements and leadership commitment required for sustainable adoption?
A technically successful system may fail to scale because the surrounding organisation cannot support it.
These dimensions should not be examined only at the end of a pilot.
They should inform its design from the beginning.
Otherwise, the organisation may discover that it has collected extensive technical evidence but insufficient information to make a responsible investment decision.
From Proof of Concept to Proof of Organisational Capability
The traditional language of innovation frequently distinguishes between a proof of concept and a production deployment.
A proof of concept demonstrates technical feasibility.
Production establishes that a solution can operate under real conditions.
Artificial intelligence suggests the need for an additional distinction.
We should also examine the organisation’s ability to absorb and govern the capability.
Call it a proof of organisational capability.
The question is not simply whether the AI system works.
It is whether the organisation can make it work responsibly, repeatedly and sustainably.
Consider an AI assistant that provides access to internal knowledge.
The technology may be capable of retrieving relevant documents and generating coherent responses.
But the organisation must still maintain the knowledge base, manage conflicting sources, define information ownership and establish how users should respond to uncertainty.
If these arrangements do not exist, the assistant may perform impressively during a demonstration while remaining unreliable in everyday operations.
The pilot has exposed an organisational requirement.
That finding is valuable even if the immediate technology is not deployed.
It reveals where knowledge management, governance or process design must improve.
This is precisely how experimentation should contribute to organisational intelligence.
The experiment should not merely test the technology.
It should help the organisation understand itself.
What knowledge does it possess?
Where are responsibilities unclear?
Which processes depend on informal practices?
What assumptions have remained unexamined?
Where does human expertise compensate for weaknesses in formal systems?
AI pilots can become unusually effective instruments for discovering these conditions.
But only when their findings are interpreted beyond technical performance.
Who Owns the Decision?
One reason pilots remain suspended between experimentation and adoption is that responsibility changes as the initiative progresses.
The technical team may own the experiment.
An innovation office may sponsor it.
A business unit may provide users.
A supplier may demonstrate the technology.
But who owns the decision to transform the process?
Who is accountable for the resulting operational capability?
Who accepts the risks?
Who provides the resources?
And who decides that the pilot should stop?
These questions cannot be resolved simply by appointing a project manager.
They concern the distribution of organisational authority.
A pilot can be managed as a project.
Transformation must be governed as an organisational decision.
This distinction becomes especially important when AI crosses traditional functional boundaries.
A knowledge assistant may involve technology, legal, human resources, operations and information governance.
An AI agent may affect multiple processes and decision rights.
A predictive system may change how professional judgement is exercised.
The organisational consequences do not necessarily align with the department that initiated the experiment.
Without clear ownership, a pilot can succeed technically while lacking an institutional sponsor capable of authorising broader change.
The result is familiar: positive feedback, encouraging presentations and no decision.
A mature AI portfolio should therefore define decision ownership before experimentation begins.
Not every pilot requires a board-level decision.
But every pilot should have an identifiable authority responsible for determining what happens when the evidence is available.
The Value of Saying No
Innovation cultures often celebrate initiatives that progress to deployment.
Experiments that are discontinued receive much less attention.
Yet the ability to stop an initiative can be an important indicator of organisational maturity.
A pilot may reveal that a proposed AI solution is too expensive, too unreliable, insufficiently useful or incompatible with the organisation’s responsibilities.
It may demonstrate that a simpler process change would produce greater value.
It may show that the underlying problem is not technological at all.
Perhaps the organisation does not need an AI assistant to navigate a complicated procedure.
Perhaps it needs to simplify the procedure.
Perhaps a department does not need a generative system to summarise fragmented reports.
Perhaps it needs a more coherent information architecture.
Perhaps the difficulty attributed to insufficient automation is actually a problem of unclear authority.
In these cases, discontinuing the AI initiative may be the most intelligent outcome.
The organisation has learnt something important about the problem.
The danger arises when innovation is measured primarily through the number of pilots launched or solutions deployed.
Such metrics can create incentives to continue initiatives that should be reconsidered.
A more mature approach would recognise the value of evidence-based termination.
A pilot that prevents the wrong investment may create more value than one that produces an impressive demonstration.
The objective is not to maximise the number of AI systems in use.
It is to improve the organisation’s capacity to achieve meaningful outcomes.
The Institutional Memory of Experimentation
There is another problem that receives relatively little attention.
Organisations often fail to preserve what they learn from experiments.
A pilot concludes, a report is prepared and the project team moves on.
The presentation may remain in a shared folder.
But the reasoning behind the conclusions, the assumptions that were tested and the limitations that emerged are not always retained in a form that others can reuse.
A year later, another department may begin a similar experiment.
It may encounter the same problems, evaluate the same alternatives and repeat the same mistakes.
The organisation has accumulated experience without converting it into institutional knowledge.
This is particularly problematic in artificial intelligence, where experimentation is distributed across multiple functions and technologies change rapidly.
The lessons from one pilot may remain relevant even after the model used in the experiment becomes obsolete.
A particular language model may be replaced.
But the discovery that information ownership is unclear, verification responsibilities are missing or a process depends on undocumented expertise may remain important for years.
A mature organisation should therefore preserve more than pilot results.
It should retain the questions asked, the evidence collected, the assumptions challenged, the alternatives considered and the reasoning behind the final decision.
This creates a reusable institutional memory of experimentation.
Future initiatives can begin from accumulated knowledge rather than rediscovering the same constraints.
And the organisation becomes progressively better at deciding where AI is likely to create value.
This is a different understanding of innovation.
Instead of treating each experiment as an isolated project, the organisation develops a continuous capability for learning from technological change.
The Portfolio Problem
Individual pilots can appear reasonable when considered separately.
Each addresses a potentially useful application.
Each requires a manageable initial investment.
Each demonstrates that the organisation is exploring new possibilities.
But the picture changes when the initiatives are examined together.
An organisation may have dozens of experiments involving document analysis, knowledge assistants, customer interaction, process automation and predictive systems.
Some may use overlapping technologies.
Others may require similar data infrastructure, governance arrangements or professional skills.
Several may attempt to solve different manifestations of the same underlying organisational problem.
Without portfolio-level coordination, experimentation becomes fragmented.
Resources are dispersed.
Knowledge is duplicated.
Technical dependencies multiply.
And decisions about priorities become difficult.
This does not mean that every experiment must be centrally controlled.
Distributed experimentation can encourage creativity and reveal opportunities that a central function might overlook.
But decentralised discovery should be connected to shared organisational learning.
A mature AI portfolio needs visibility across initiatives.
It should identify common requirements, compare evidence, recognise dependencies and distinguish local improvements from capabilities that could benefit the wider organisation.
It should also establish criteria for allocating resources beyond the experimental phase.
The objective is not bureaucratic control.
It is strategic coherence.
An organisation should be able to explain why certain AI initiatives deserve investment, why others remain experimental and why some have been discontinued.
Without that ability, the portfolio risks becoming a collection of interesting technologies rather than an expression of organisational priorities.
Figure 3 – Shared learning, governance and ownership connect isolated experiments to reusable knowledge, better decisions and coordinated action.
The Difference Between Activity and Progress
There is a broader lesson here that extends beyond artificial intelligence.
Organisations have always been capable of confusing activity with progress.
Projects are launched.
Committees are established.
Strategies are published.
Dashboards are produced.
Workshops are organised.
These activities may be necessary.
But their existence does not prove that the organisation has become more capable.
Artificial intelligence makes this distinction particularly visible because the technology can produce impressive results with relatively little initial effort.
A small team can build a convincing demonstration in days.
The same organisation may require months to redesign the processes, responsibilities and knowledge structures needed for sustainable adoption.
The visible technological achievement arrives quickly.
The organisational transformation takes longer.
This difference in speed creates pressure to celebrate what can be demonstrated rather than what has actually changed.
Yet organisational intelligence cannot be measured by the number of technologies tested.
It concerns the capacity to recognise opportunities, evaluate evidence, make decisions, coordinate action and learn from consequences.
An organisation that launches twenty AI pilots but cannot decide which ones deserve adoption is not necessarily more advanced than one that conducts five carefully designed experiments and turns three into sustainable capabilities.
The numbers themselves are not the point.
The quality of the decisions is.
Innovation should be measured not only by what an organisation tries, but by what it learns, changes and deliberately chooses not to pursue.
Designing a Decision-Oriented AI Portfolio
What would a different approach look like?
It would begin by treating every AI pilot as a temporary instrument for reducing uncertainty.
Before the experiment starts, the organisation would define the problem, the expected outcome and the decision the evidence is intended to support.
The experiment would be designed around that decision.
Technical performance would matter, but so would operational consequences, human impact, economic sustainability, risk and organisational readiness.
The pilot would have a clear owner and an agreed review point.
At that point, the organisation would decide whether to scale, redesign, conduct a further narrowly defined test or stop.
The reasoning would be documented.
The lessons would become available to future initiatives.
And the portfolio would be reviewed periodically to identify common capabilities, dependencies and strategic priorities.
None of this requires excessive bureaucracy.
The process should be proportionate to the scale and risk of the initiative.
A small internal productivity experiment does not require the same governance arrangements as an AI system influencing consequential decisions.
But both should have a purpose.
Both should generate evidence.
And both should eventually lead to a decision.
The key is to connect experimentation with organisational responsibility.
Not to discourage innovation, but to ensure that innovation produces something more durable than a demonstration.
Beyond the Pilot: The Organisation That Learns
The future of artificial intelligence in organisations will not be determined solely by the capabilities of the models they adopt.
Those capabilities will continue to evolve.
Models will become more powerful, interfaces more natural and systems increasingly capable of performing complex tasks.
But the central organisational challenge will remain.
How do we translate technological possibilities into meaningful, responsible and sustainable change?
The answer requires more than technical expertise.
It requires leadership capable of defining priorities, governance that connects authority with accountability, knowledge systems that preserve learning and organisational structures that can adapt.
It also requires the courage to make decisions.
To invest when the evidence is convincing.
To redesign when the surrounding organisation is not ready.
To question assumptions when the results are ambiguous.
And to stop when a technology does not create sufficient value.
After decades of technological transformation, I have become increasingly convinced that the most important difference between organisations is not how quickly they discover new technologies.
It is how effectively they turn those discoveries into collective capability.
Artificial intelligence makes this distinction more important than ever.
The technology is evolving at extraordinary speed.
The number of possible experiments is almost unlimited.
But organisational attention, resources and capacity for change are not.
The challenge is therefore not to experiment with everything.
It is to develop the intelligence to decide what deserves to become part of the organisation.
A pilot should never be the destination.
It should be the beginning of a decision.
And perhaps the most important question to ask at the end of every AI experiment is not:
Did the technology work?
But:
What do we now know that allows us to make a better organisational decision?
Because an organisation does not become more intelligent by conducting more experiments.
It becomes more intelligent when experimentation changes what it understands, what it decides and what it is capable of doing.
