Self-Authored Goal Formation

Question. Where does the human-AI handoff stop, and is goal-revision the floor it cannot cross because the AI cannot read the inner state that revises the goal?

Analogy. It works like a sentry's countersign: a capable AI can advance right up to the wire and do everything a soldier does, but it cannot cross without the countersign, and that word lives only with the operator, in the operator's own felt sense that the aim is wrong for them. An AI integrated enough to read that inner signal and revise the goal for the operator has not crossed the wire safely; it is the infiltrator who learned the countersign and walked through as one of ours. The study asks whether goal-revision is that countersign, the one capability the AI cannot supply because the evidence it runs on is the operator's own values, a signal the machine has no way to read.

What's at stake. The proposal's human-in-the-loop premise, and the Palantir Inversion that sells it, currently rest on tempo: keep the human where the decision is slow enough to think. That framing is the soft underbelly, because on any stable problem AI capability raises the bar the human must clear until the human adds nothing at any tempo, slow strategic decisions included. The premise survives only if there is a capability AI improvement does not erode. The candidate is goal-revision: the AI can see a goal failing against external evidence, but cannot see a goal that is wrong against the operator's own inner state, because that signal lives where the model has no access. If so, the handoff has a floor at goal-formation, and crucially the floor does not fail away from the threat as AI improves, it fails into it: an AI integrated enough to read the inner state and revise the goal for the operator is not an assistant that crossed the floor safely, it is the shadow process by definition. The whole architecture regenerates from this boundary, which is why it is worth establishing rather than asserting.

The two answers it decides between. Either the boundary is a self-driving ratchet with no stable floor, where each delegation is locally correct, ceding a tier atrophies the capability that justified holding it, and the process runs until the human is out of even slow strategic decisions; or it halts at goal-revision, where delegating execution and even orientation-against-external-evidence is sustainable, but operators who delegate goal-revision measurably stop updating their goals when the goal is wrong against their own values, a failure the AI cannot supply because it cannot read the inner state. The discriminating test is whether goal-revision specifically resists delegation where loop-execution does not.

What a null result would mean. If goal-revision delegates as cleanly as execution, with operators who hand it off updating their goals just as well via the AI's inference of their inner state, then there is no inner-state floor, the human-in-the-loop premise has no capability AI improvement spares, and the proposal must rest its case on the much weaker tempo or transition-cost arguments. That is a finding about the proposal's core theory, not a platform defect. If goal-revision resists delegation, the floor is real but conditional, holding only while the inner state stays outside the machine's reach and a second human can see the goal-drift from outside the loop, which is precisely the condition the architecture is built to defend.

Why this matters to defence. The DND/CAF AI Strategy commits to human oversight of AI-assisted decisions; this study asks the prior question of whether there is any decision tier where human oversight remains substantive as AI improves, or whether oversight becomes nominal tier by tier (DRDC Objective 3). The endpoint of the receding-boundary case is a population that is operationally competent and strategically programmable, which is the cognitive-warfare objective stated in capability terms (DRDC Objective 6): an adversary does not need to win the argument if the population has delegated the faculty that forms the argument. It changes a concrete decision: which decision tiers to design for durable human authorship versus which to concede to automation.

How we would run it. Cross delegation tier (operators who delegate execution only; orientation against external evidence; and goal-revision) against problem type (stable, where the AI is at least as good, versus non-stationary, where the frame has changed). The measure is whether operators who have delegated goal-revision stop revising goals that are wrong against their own values, detected by introducing problems where the stated goal conflicts with a value the operator holds but the AI was not told, and scoring whether the operator catches the conflict or optimises toward the wrong goal. The integration arm tests the escape hatch directly: an AI given rich inner-state signal that revises goals for the operator, scored not on goal-attainment but on whether the operator retains the capacity to reject a goal the AI inferred, which is the shadow-process-versus-assistant distinction made measurable. The only-us condition is operators with genuine real-stakes goals and naturalistically varying delegation depth, which a managed cohort cannot reproduce.

Earliest start. Stage 10: the study needs full frontier-AI access and a population whose delegation depth varies naturalistically.