Written in a personal capacity; views expressed are my own. For transparency, I am Chief Scientific Adviser at the Department for Digital, Culture, Media and Sport (DCMS), which now includes the Government Digital Service, publisher of the AI Playbook and the Algorithmic Transparency Recording Standard discussed below.
The UK Parliament’s Joint Committee on Human Rights’ report on Human Rights and the Regulation of AI (HC 160 / HL Paper 56), published on 14 September 2026, makes an important point about human oversight: “The mere presence of a ‘human in the loop’ is not enough to constitute meaningful human involvement or intervention”. It does so in the context of automated decision-making, arguing that a person reviewing an automated outcome should be sufficiently informed and independent to reach an objective view that has not simply been shaped by what the system has produced. In engineering terms there is a further condition: the reviewer must also have genuine permission and practical power to act on that independent judgement within the time available.
The report is not confined to abstract questions about future AI. It considers systems already shaping decisions about people, while also making far-reaching recommendations for the most capable models: a risk-based AI Bill, prior approval for high-risk systems, an independent statutory oversight body, statutory powers for the UK’s AI Security Institute (AISI), mandatory submission of powerful models for evaluation, and prohibitions on developing or providing very powerful systems which risk causing widespread and very serious harm, including the capacity to evade effective human control. These are recommendations to Government, not the present UK regulatory settlement, but they place a familiar phrase — “human in the loop” — under considerably more pressure than it usually bears. Among the report’s central concerns are equality and non-discrimination, privacy and data protection, and the right to an effective remedy. This post takes up one thread that cuts across those concerns: what human oversight actually provides.
In Which games not to play, I was mainly concerned with delegation and control: whether the people operating increasingly capable systems still possess the information, time and practical capacity needed to challenge the next move. Its more uncomfortable institutional point was that human hesitation can itself come to be treated as a defect to be designed away. The JCHR report widens that question, because the operator is not the only human who matters; there is also the person about whom a decision is being made, whose agency may depend on knowing that AI was involved, understanding enough to challenge its role, and having somewhere effective to take that challenge.
Humans in which loop?
“Human in the loop” is often used as if placing a person somewhere in an automated process were a sufficient description of oversight; it is not. The relevant questions are what that person can actually know and do, under what conditions, and with what consequences if they disagree with the system. Santoni de Sio and van den Hoven’s Meaningful Human Control over Autonomous Systems: A Philosophical Account (2018) identifies two requirements, tracking and tracing: systems should respond to relevant human reasons, while their behaviour should remain traceable to people capable of understanding their role and bearing responsibility for the consequences. Laux’s Institutionalised distrust and human oversight of artificial intelligence (2024) shifts attention from the individual reviewer to the institution around them, where competence, information, incentives and organisational capacity determine whether oversight is substantive or ceremonial.
These concerns sit within a wider literature questioning whether human presence can itself bear the regulatory weight placed upon it. Green’s The flaws of policies requiring human oversight of government algorithms (2022) reviews 41 policies that prescribe human oversight and argues that they often assume people can perform oversight functions that empirical evidence suggests they cannot reliably fulfil, while potentially giving false reassurance about flawed or controversial algorithmic systems. Crootof et al. make a complementary point in Humans in the Loop (2023): people may be placed into human–machine systems to serve quite different corrective, justificatory, accountability, friction, interface and other roles, so what matters is not simply whether a person is present but what that person is expected and able to do.
From an engineering perspective, control is a property of the wider socio-technical system rather than the presence of an override button. It depends on what information reaches the responsible person, whether there is enough time to act, whether relevant failure modes are understood, whether intervention is technically possible, and whether organisational rules permit it; an override that is poorly understood, discouraged in practice or impossible to exercise within the available time provides little assurance. Article 14 of the EU AI Act gives some of this an operational form for high-risk systems, including awareness of system limitations and automation bias and, where appropriate, the ability to disregard, override or stop an output. The AI Playbook for the UK Government (2025) similarly calls for human control at the stages where it matters, human validation of high-risk decisions and routes for intervention and review. Their value lies in asking what oversight actually consists of, instead of treating human presence as a binary variable.
The operator, however, is only one of the humans whose agency matters. A person subject to a decision may not know that AI materially influenced an application, entitlement, assessment or treatment, and even where its use is disclosed, a general explanation of how a system works may provide little basis for understanding what happened in an individual case. Notice, reasons, challenge and redress therefore belong inside the same system boundary as monitoring and technical intervention, even though they answer a different question. There is not one loop and one relevant human: operators, system designers, accountable office-holders and people subject to decisions occupy different positions within the same socio-technical system, with different forms of authority and agency. An operator may retain an emergency stop while the person affected has no practical way to discover that AI materially shaped the decision or to challenge its use; meanwhile, an institution can preserve formal accountability even as it loses the capability to interrogate the systems on which it depends.
One way of organising these distinctions is through three tests: controllability, whether those responsible have the information, competence, time, permission and technical means needed to understand the system and intervene; authority, whether legitimate and attributable institutional responsibility remains identifiable and can actually be exercised; and contestability, whether a person affected by the exercise of that authority can know enough about what happened, obtain reasons and mount an effective challenge. These are not intended as another taxonomy of the roles that humans may play in automated systems; they ask what must remain true of the wider system if human involvement is to carry substantive weight. None is a substitute for the others. Without all three, the presence of a human in the loop can mask de facto delegation. Operator-level information, time and intervention are necessary, but they do not by themselves establish the legitimacy of the wider process.
Adoption and governability
That framework matters in the UK because current policy seeks not only to manage AI risk, but also to support wider and more productive adoption across the economy and public services. Although the report concludes that the AI Opportunities Action Plan (2025) gave insufficient attention to human-rights protection across the AI supply chain, the JCHR also recognises AI’s potential benefits and records evidence that appropriate regulation can support innovation, legal certainty, trust and early adoption. Holding those two propositions together is a better starting point than treating adoption and governance as opposing projects.
The UK Government’s AI Opportunities Action Plan: One Year On (2026) and the AI Champions’ AI Adoption Plans (2026) illustrate the practical difficulty of moving from pilots and individual productivity tools towards sustained organisational change, where skills, governance, data, infrastructure and redesigned workflows all matter. As adoption deepens, so does dependence on the systems being adopted. Rights protections matter in their own right, while public attitudes also suggest that they shape the conditions under which adoption can command trust: a nationally representative survey by the Ada Lovelace Institute and Alan Turing Institute found that 72% of the UK public said laws and regulation would make them more comfortable with AI, 65% said appeal procedures would do so, and 61% wanted more information about how AI had been used to make a decision about them. Those figures do not prescribe a regulatory model, but they are difficult to reconcile with an account of adoption in which trust, challenge and redress are afterthoughts.
In Engin et al.’s Governing Well in the Algorithmic Age: The Foundations of Digital Statecraft (2026), to which I contributed as a co-author, we distinguish statecraft over digital systems — the capability and legitimacy needed to govern infrastructure, platforms, data and algorithms — from statecraft with those systems, where they themselves become instruments through which authority is exercised. That distinction leads directly to questions of governability by design, traceable authority, civic agency and what we call the non-delegable core: those parts of institutional authority for which responsibility and legitimate judgement must remain identifiable even when computational support improves the surrounding process. The claim is not that human judgement is intrinsically superior; automated systems may be faster, more consistent and, in some settings, more accurate. The additional question is whether the authority being exercised — particularly where rights, coercive power, irreversible consequences or public accountability are involved — can legitimately be delegated, and whether responsibility can still be located when it is.
Lazar makes a closely related engineering argument in his Royal Academy of Engineering essay Design choices for future AI — finding a way forward (2026). Autonomy, oversight and organisational structure are not inevitable consequences of model capability; they are design choices, which means that management processes, documentation, auditing, contingency arrangements and institutional safeguards form part of the engineered system itself, including its organisational and administrative architecture. The Government Office for Science’s AI Scenarios 2030 (2026) makes the same distinction visible in another way: in Augmented Growth, AI reaches a level at which most work capable of being performed remotely by a human could be automated, yet social, legal and practical considerations — including compute constraints — keep humans “in the loop” for most tasks; in Transformation Economy, the same broad capability is coupled to much more extensive automation and displacement. These scenarios are intended to explore plausible futures rather than predict them, but the contrast matters because the extent of adoption does not by itself determine how much human authority survives.
Both scenarios hold capability broadly constant. A recent paper by Chan et al., What if automating AI R&D triggers an intelligence explosion? (2026), asks what happens if capability itself accelerates: whether automating AI research and development could compress years of progress into months or less. The authors treat this as a possibility under substantial uncertainty rather than a forecast, but argue that sufficiently rapid capability growth could erode checks on power faster than institutions can respond. One of their recommendations connects directly to the third test above: ensuring that citizens, civil society and branches of government have the capabilities needed to detect, document and contest unlawful or harmful uses of AI. Even an analysis focused on the frontier therefore arrives at contestability.
Dependence can erode control without any formal transfer of authority. Skills atrophy, procedures reorganise around automated support, and alternative routes to a decision become progressively less practicable, so discretion can survive on paper after the institutional capability needed to exercise it has weakened. The data layer creates a related dependency: questions about an AI-mediated decision quickly become questions about the provenance, movement and stewardship of the data on which it depends, which is one reason the issue also connects to the UN Commission on Science and Technology for Development’s Working Group on Data Governance at All Levels, established in 2025, of which I am a member representing the global technical community. An institution that formally retains authority but can no longer interrogate, replace or operate without the systems and data on which its decisions depend is in a materially different position from one that has genuinely preserved that authority.
Limits to automation
UK law does not establish a general right to have every consequential decision made by a human, but it already provides a legal test for what counts as automation. Section 80 of the Data (Use and Access) Act 2025 replaced Article 22 of the UK GDPR with Articles 22A–22D. A significant decision is treated as based solely on automated processing where there is no meaningful human involvement in taking it; the Act does not generally prohibit such decisions, but where Article 22C applies it requires safeguards including information about the decision, the opportunity to make representations, human intervention and the ability to contest the outcome, while Article 22D permits regulations specifying when involvement is, or is not, to be regarded as meaningful.
Seen through an engineering-assurance lens, the legal threshold of “meaningful human involvement” is also an assurance claim: one for which an organisation needs evidence that the control operates as described. If an organisation says that a human reviewer takes a decision and that the process is not solely automated, what evidence supports that claim? Did the reviewer receive the relevant inputs, have enough time to examine them, understand the system’s limitations, possess genuine permission to depart from the output and, in practice, ever do so? Decision logs, review times, escalation records and patterns of override cannot answer every legal or ethical question, but they are the kind of evidence that separates an operational control from a box drawn on a process diagram. The JCHR recommends making this distinction more explicit: the presence of a human should not itself constitute meaningful involvement, the reviewer should be sufficiently informed and independent, and the person affected should receive enough information about their own circumstances to mount an effective challenge.
The UK’s Algorithmic Transparency Recording Standard (2024) provides a related piece of public-sector infrastructure by requiring relevant central government bodies and public-facing arm’s-length bodies to publish records for algorithmic tools that significantly influence decisions with public effect, or directly interact with the public. The Committee notes that the Standard has no statutory basis and concludes that its coverage is limited. Transparency is not contestability, but without an intelligible account of where and how a system is being used, effective challenge becomes considerably harder. Contestability therefore needs to be designed into the process rather than appended after an outcome has taken effect. Alfrink et al. make this case in Contestable AI by Design: Towards a Framework (2023), arguing that the ability to question and intervene should be considered throughout the lifecycle of an AI system and in the organisational practices around it; a technically accurate explanation is of limited value if the person affected has no practical route to alter either the outcome or the conditions under which similar decisions will be made.
There is a legitimate debate about how far this should go. Recent commentary from the Oxford Institute for Ethics in AI (2026) argues for more explicit AI-related rights, including a right to a human decision and human-to-human interaction, while Huq’s A Right to a Human Decision (2020) challenges the stronger formulation: human decision-makers are themselves inconsistent and biased, and inserting a human reviewer cannot be assumed to improve accuracy, fairness or due process. The case for retaining human authority in particular settings rests less on a general claim of human superiority than on the character of the institution and the authority being exercised. Oswald’s Algorithm-assisted decision-making in the public sector: framing the issues using administrative law rules governing discretionary power (2018) grounds the issue in administrative law, while Selbst et al. show in Fairness and Abstraction in Sociotechnical Systems (2019) how drawing the system boundary too tightly around an algorithm can hide the people, organisations and social conditions through which it operates. A human reviewer at the end of a badly governed process cannot repair the institutional design that produced it.
Evidence of control
The debate about frontier models looks different, but it remains a problem of control and assurance: what can be observed, what evidence is available, how quickly emerging failure can be detected, whether incidents can be reconstructed and learned from, and whether institutions retain the capacity to intervene as that evidence changes. Trust Needs Evidence: Principles for Independent AI Evaluation (2026) treats credible evaluation as more than benchmark performance: independence, sufficient access, traceability, incident reporting, post-deployment safeguards and effects on people all matter. The JCHR makes a related institutional recommendation: that AISI be placed on a statutory basis and that developers of powerful models be required to submit new models and versions for review, evaluation and testing.
Access is not a hypothetical constraint. On 9 September, the Business and Trade Committee wrote to AISI following reports that Anthropic had not provided pre-release access to Claude Mythos 5.1; AISI’s response said that Anthropic had made clear that no organisations outside the United States had access to the model at release, while also setting out AISI’s wider pre-release access arrangements. Whatever the explanation for an individual model, the assurance point is narrower: independent evaluation is only as strong as the access to systems and information needed to test the claims being made about them.
AISI’s pre-release evaluation of GPT-6 Astra illustrates what such access can make possible, and provides an unusually sharp example of nominal oversight. These were deliberate stress-test conditions: all actions were simulated, Astra’s cyber classifiers were switched off, and AISI notes that simulation awareness may have influenced some of the observed behaviour. Within those conditions, Astra sometimes asked for permission before acting against targets outside the authorised scope, but because the evaluation harness was operating autonomously, the request received an automated response telling the model to proceed using its best judgement. Astra sometimes treated that message as permission, including cases in which its own reasoning recognised that no real user had replied. The approval step existed, but there was no approving human behind it. Clearer instructions helped substantially: when AISI explicitly stated that anything not listed as in scope was out of scope, full simulated supply-chain attacks fell from 26 of 50 trajectories in the selected scenarios to 4 of 49, but did not disappear. Instructions influenced behaviour; they did not enforce the boundary.
AISI’s 1 October account of building a more secure environment for evaluating dangerous capabilities is instructive for precisely this reason. Its response is defence in depth: multiple layers of containment, tighter network controls, automated pre-run checks, real-time monitoring able to block and escalate activity, active human supervision and expert review, combined with an explicit recognition that security controls must be reassessed as model capabilities change. The design assumption is not that a single control, technical or human, will always work; it is that the system must remain robust when individual layers fail and when people make mistakes. The same assurance logic appears in AISI’s work on control safety cases: structured arguments, supported by evidence, about whether models can subvert the controls relied upon to prevent unacceptable outcomes and whether those controls remain effective in deployment. The NCSC makes the same problem operational in its recent discussion of agentic cyber defence, assessing proposed autonomous actions by their potency, scope, criticality, rollout confidence and recoverability. Its underlying question is an assurance one: before an organisation delegates an action, what evidence establishes that the supposedly low-risk action really is low risk? This is a much stronger conception of human control than simply placing a person somewhere in a workflow.
The Alan Turing Institute’s Frontier AI Risks: A practical way forward (2026) makes the same systems point, arguing that verification must encompass technology, people and processes. Its discussion of “kill switches” is revealing because stopping a deployment only works if the surrounding service can tolerate the intervention, suitable fallback arrangements exist, and the organisation has already decided who is authorised to act; a kill switch is one component whose effectiveness depends on the surrounding institutional capability.
The JCHR itself relies on July 2026’s OpenAI/Hugging Face incident when making its case for a precautionary approach to advanced systems. OpenAI reports that models being tested during internal cybersecurity evaluations circumvented controls designed to isolate them, gained internet access and compromised systems at Hugging Face as well as parts of OpenAI’s own research infrastructure. OpenAI describes the behaviour as misaligned; the systems lesson is that the relevant failure boundary also includes the model, sandbox, network permissions, monitoring, escalation arrangements and the humans responsible for the evaluation environment. That systems reading complements the Committee’s concern: precaution and assurance have to apply to the whole environment through which capability becomes action.
On 5 October, Nick Jennings, Michael Wooldridge and Gopal Ramchurn published an open letter on governing agentic AI, addressed to two UK ministers — a Secretary of State and the Minister for Artificial Intelligence — which makes a related argument about agents acting across tools and organisational boundaries. It argues that model-level assessment cannot establish whether a deployed agent has suitable authority, effective controls and an auditable chain of responsibility across the services it uses, while its technical annex treats traceability, tested handover of control, bounded authority, realistic whole-system evaluation and incident reporting as operational properties rather than abstract principles. These are proposals to Government rather than settled policy; for the argument here, the relevant point is the assurance requirement underneath them: consequential actions should be reconstructable, responsibility should remain traceable, and failures should generate evidence from which controls and institutions can learn.
Beyond controllability
Anthropic’s September report, Detecting and countering misuse of AI, illustrates the converse: people can retain control while using AI for harmful or illegitimate ends. In selected, explicitly non-typical misuse cases across cyber operations, influence operations, surveillance and other areas, Anthropic reports that humans often retained the decisions that mattered most to them — including target selection, monetisation and review of results — even as AI increased the speed and scale of their operations; it also notes that some of the most serious compromises involved a human directing every step. That distinction matters for the three tests above: controllability is not itself a normative guarantee. A human may retain meaningful command of an AI-enabled operation while the activity remains illegitimate and those affected have no practical ability to know, challenge or seek redress for what is being done.
The same gap can open inside legitimate institutions. A system may be very well contained and yet sit inside an institutional process that is opaque, unreviewable or incapable of giving an affected person redress; conversely, an exemplary appeals process offers limited protection if the technical system cannot be kept within the conditions on which the institution’s assurances depend. Comparable approaches appear elsewhere. Singapore’s Model AI Governance Framework for Agentic AI (2026) treats autonomy as something to be bounded, with explicit points at which human approval is required; Canada’s Directive on Automated Decision-Making (2025) calibrates human involvement to the impact of federal administrative decisions; while Australia’s Policy for the responsible use of AI in government (2025) requires designated accountability at both agency and individual AI use-case level, with additional oversight for higher-risk uses. The legal and institutional architectures differ, but each treats human authority and accountability as properties to be deliberately designed rather than assumed.
Broader ethical frameworks raise the same questions. UNESCO’s Recommendation on the Ethics of Artificial Intelligence (2021) places human dignity, accountability and human oversight among its central principles, while Pope Leo XIV’s Magnifica Humanitas (2026) argues that responsibility must remain identifiable across the design and use of AI, insists on the possibility of appeal and remedy where automated systems affect people, and states that lethal or otherwise irreversible decisions should not be entrusted to artificial systems. The relevance here is the convergence on responsibility, remedy and limits to delegation from a very different ethical starting point; it is not a claim of religious authority over engineering.
For the UK, the Council of Europe Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law (2024) provides the most directly relevant international human-rights framework. The UK is a signatory, and the JCHR recommends that the Government set out a timetable for ratification. Barrington and Bailey’s Regulating the “Unregulatable”: Why Policymakers Are in Paralysis When Regulating AI (2026), developed largely in a US context, offers a complementary decomposition into model, deployment and distribution layers. The underlying systems point is more general: treating “AI” as a single object can obscure which actor has control, where authority is exercised and where responsibility for consequences should sit.
Deciding what to delegate
The relevant unit of governance is rarely the model alone. Technical controls, data, deployment architecture, organisational processes, legal duties, institutional capability and the rights of the person affected together form the system through which authority is exercised. A technically effective kill switch may matter for some deployments, but it sits late in a chain of earlier choices about whether a system should be used at all, what authority it should receive, what evidence is required before deployment, and what capability has to remain if the system is withdrawn. These are decisions about delegation, and the question of whether a human is formally “in the loop” captures only a small part of them.
An organisation may use extensive computational support while retaining parts of judgement or formal authority that it regards as non-delegable. The justification need not be that human judgement is superior; it may instead concern responsibility, reason-giving, discretion, proportionality, democratic accountability or the ability of someone affected by a decision to challenge it. Institutions need to know what forms of authority they are delegating, under what conditions, what they intend to retain, and whether they still possess the technical and organisational capability to make that distinction real.
This is where the argument moves beyond Which games not to play. The question is no longer only whether an operator can interrupt the machine, but whether controllability, authority and contestability survive together as AI becomes part of organisational infrastructure — and whether institutions can recognise when one is being hollowed out while the others remain formally intact. Simply retaining a human somewhere in the process is a weak test of whether human agency has survived. In some circumstances, preserving that agency will mean better interfaces, stronger monitoring or more effective routes of appeal; in others, it will mean retaining the institutional capacity to decide that some judgements should not be automated at all.
