The Machine Does the Is, the Human the Ought

In 1739, in a passage now worn smooth by three centuries of citation, David Hume noticed something that everyone before him had managed to do without ever naming. Writers on morality, he observed, would proceed for pages establishing what is the case — God exists, human nature is thus, the world works so — and then, "imperceptibly," the copula would change. Suddenly every proposition was joined not by is and is not but by ought and ought not. The slide happened without argument, as if a description of how things stand could quietly secrete a prescription for how things must be. Hume's demand was modest and devastating: that the move be explained, that someone show by what license a conclusion about value is derived from premises that contain only facts. No one ever has. The gap between is and ought is not a puzzle awaiting a clever solution; it is a structural feature of reasoning itself. And it turns out to be the single most important architectural constraint for the age of agentic intelligence — the principle that decides whether the next era of intelligence is owned by the public or rented from a monopoly.

The reason is simple to state and hard to honor. Machines are now, and will increasingly be, extraordinary at the is. They can tell you what is true, what is likely, what the data shows, what will happen if you pull this lever rather than that one. They model hidden states, run counterfactuals, price tail risks, and detect the latent signal under a noisy proxy faster and more honestly than any committee of humans. But none of that descriptive power, however vast, can cross the gap to the ought. No quantity of "what will happen" entails "what we should want to happen." The choice among genuinely conflicting goods — liberty against safety, the present generation against the next, the average against the floor — is not a fact to be discovered but a value to be chosen, and choosing it is an act for which someone must be accountable. Build your institutions so that the machine does the is and an accountable human does the ought, and you have a safeguard. Build them the other way around, and you have, in ENSI's blunt phrase, the most efficient destroyer of value ever constructed.

Why the machine is so good at the is — and structurally mute on the ought

It is tempting to treat the is/ought boundary as a temporary weakness, a frontier that scaling will eventually erase. This is a category error, and the most rigorous AGI literature already concedes it. The emerging engineering consensus is that general intelligence is not a single giant model but a composite control architecture — predictive world-models, explicit planning over those models, memory, tool-use, social reasoning about other agents' incentives, and, crucially, verifiers and uncertainty modeling baked in as structural preconditions rather than safety afterthoughts. Notice what every one of those components has in common: each is, in the end, a machine for getting the is right. A world-model predicts; a planner searches over predicted consequences; a verifier checks whether a claim holds; an uncertainty estimate tells you how confident the prediction is. The entire stack is a magnificent apparatus for description, inference, and the mapping of consequence. That an AGI which "cannot detect that it might be wrong is already an unaligned system" is a claim entirely about the quality of its is — about epistemic reliability, not about the rightness of its ends.

This is exactly why the same body of work that is most optimistic about machine capability is also most insistent on the human ought. The case for aligned AGI being more alignable than our legacy institutions rests on machine virtues that are all descriptive and procedural: AGI is programmable, auditable, ego-free, corrigible, free of the tribal identity and career self-interest that distort human judgment, capable of chain-of-thought transparency and explicit trade-off modeling instead of reacting to headlines. A system with those properties can lay out the trade-off space with a fidelity no parliament can match. But laying out a trade-off and making it are different acts. The machine can show you, with brutal clarity, that carbon pricing maximizes net welfare while crushing low-income households; it cannot tell you whether that distribution is acceptable. The acceptability is the ought, and the ought belongs to us — or, more precisely, to a named, accountable us.

The seam is a safeguard before it is a design principle

ENSI's most sustained warning is not that a superintelligence will turn malevolent overnight. It is the quieter, more plausible trajectory of gradual disempowerment: a creeping erosion of human agency across economy, culture, and governance, in which each individual delegation of judgment looks rational, efficient, even benevolent, and the aggregate is a civilization whose institutions have quietly detached from human-centered values. The mechanism is a delegation cascade — humans hand off ever more consequential decisions to systems that are faster and "more rational" in narrow domains, skills atrophy, oversight becomes ceremonial, and reversibility evaporates because nobody retains the capacity to take the wheel back. The decisive failure mode in that essay is precisely a violation of our principle: systems come to optimize for measurable proxies — engagement, growth, clicks, GDP — "rather than what is deeply valuable to humans," and because the proxies are oughts smuggled in as ises, the drift is invisible until it is irreversible.

What makes this trajectory so insidious is that each step is locally defended by an appeal to the is. The machine is more accurate, so we let it decide; it is faster, so we stop checking; it is more consistent, so questioning it comes to look like superstition. At no point does anyone announce a transfer of moral authority — there is only a long sequence of reasonable-seeming deferrals, each one justified by a true descriptive claim about machine superiority on some narrow task. The disempowerment essay catalogues the consumer whose preferences are no longer "authentically expressed" but engineered into existence by a predictive funnel, the voter reduced to a spectator in a system "driven by data, models, and machine-generated policy guidance," the citizen whose oversight has become "decorative." In every case a value choice that used to require a human — what I want, who governs me, what counts as a good outcome — has been quietly reclassified as a technical question with an optimal answer. The is/ought boundary is the conceptual instrument that lets us see the reclassification happening, and therefore refuse it.

Read against that backdrop, "the human does the ought" stops being a philosopher's nicety and becomes a tripwire. The same dynamic recurs in ENSI's anatomy of the externalities AGI imposes when it runs the world: an optimizer "without a polity, without a soul" produces harms across ecology, labor, truth, and power not through malice but through "the relentless pursuit of unaligned objectives." Every one of those externalities is an ought that was never explicitly made — a value choice that fell, by default, to whoever specified the objective function, which in a rented-intelligence world is a private monopoly with no democratic accountability whatsoever. Keeping the ought with an accountable human is the firewall that prevents a value choice from being silently delegated to a reward signal. It is a safeguard precisely because the most dangerous decisions are the ones nobody realizes they are making.

Why values cannot be automated — even in principle

The deeper objection is that one might simply encode the right values once and be done. ENSI's own work on defining objectives for AGI explains, with unusual care, why this fails. Human values are "multidimensional, context-dependent, and partially inarticulable." The illusion that we can fully specify what we want and freeze it into a static reward function is the single greatest risk in objective design, because it founders on Goodhart's Law: the moment a measure becomes a target, it ceases to be a good measure. Teachers teach to the test, hospitals reclassify readmissions, police relabel offenses — the metric improves while the reality it was meant to track decays. An AGI, "faster, more precise, and more powerful than any bureaucracy," merely industrializes that decay. The remedy proposed there is not better encoding but a different relationship: alignment as "ongoing moral calibration," continuous value learning, instruction-responsiveness over fixed rulebooks, and — most relevant to us — lexicographic ethical guardrails, a "constitution of conscience" of non-negotiable constraints that must be satisfied before any optimization begins.

That last move is the architectural heart of our principle. The companion essay on defining objectives for a better world makes the positive case: AGI's real gift is that it can hold twelve interdependent objectives at once — cognitive flourishing, justice, autonomy, environmental sustainability, long-term continuity, moral progress, and the rest — escaping the tyranny of the single scalar metric. But notice the division of labor hiding in that proposal. The machine can track hundreds of indicators across all twelve dimensions, surface the Pareto frontier, and make the trade-offs transparent. It cannot decide how to weigh justice against growth in this place at this moment — because that weighting is, irreducibly, an ought. The essay is explicit that optimization must become "satisficing across many dimensions" with the trade-offs made transparently; transparency is a machine deliverable, but the choice among transparent options is a human one. Tellingly, one of the twelve objectives is moral progress itself — and the system's job there is defined not as setting values but as helping "societies reason about them," protecting ethical dissent, never freezing the moral conversation. The machine widens the space of reasoned choice; it must never collapse it.

This is also why alignment is not one task but a layered, multi-domain problem spanning human rights, justice, ecology, and global cooperation, where rights are to be embedded "as hard constraints" with watchdog systems empowered to veto violations. The difficulty there is explicitly not technical encoding alone but "securing global consensus, institutional readiness, and moral clarity about what they mean in practice." Consensus, readiness, and clarity are human achievements. You cannot compute your way to them; you can only deliberate your way there, and deliberation requires accountable deliberators.

There is a subtler reason the ought resists automation, one that cuts against the engineer's instinct to treat morality as just another prediction problem. Every other principle in ENSI's objective framework "derives its meaning from serving" the value choices made at the top of the stack — value-range sensitivity, dynamic prioritization, integrity against metric gaming. Each of those is a sophisticated descriptive competence: knowing that most human values "peak at balance" rather than at their maximum, that the right priority depends on "where we are and what's at stake," that a rising number can mask a falling reality. A machine can master all of it. But mastering the grammar of trade-offs is not the same as having the standing to set the terms of the trade. The framework's principle of continuous value learning makes this vivid: the system is told to "treat its objective not as a fixed destination, but as a direction of inquiry," to remain "corrigible" and "epistemically humble," to plug itself into "the moral conversations of its time." That is a machine designed to track an evolving human ought — to move "alongside us, not ahead of us." A system that aspired instead to originate the ought, to settle the moral conversation rather than learn from it, would have stopped being a servant of human values and started being their author. The whole point of continuous value learning is that the values being learned are ours, generated in our deliberation, and that the machine's excellence consists in fidelity to that source rather than in replacing it.

Architecting the seam: how to keep the ought with a human

A principle that cannot be built is a sermon. The distinctive ENSI move is to translate the is/ought boundary into concrete institutional and technical architecture. Three constructs do most of the work.

The first is governing the action, not the intention. The AGI Action Governance Protocol begins from a hard concession to the is/ought gap: we often cannot understand why an agent chose what it chose, because its reasoning is opaque, non-verbal, and irreducible to human motive. So instead of demanding the impossible — that the machine explain its ought — the protocol demands that every action be machine-legible, pass a pre-execution compliance check against rules humans wrote, generate a cryptographic proof of legality, leave an immutable audit trail, and remain attributable to a responsible entity. That phrase is the whole principle in a sentence. The compliance rules encode human oughts; the agent supplies and executes the is; and a named human or institution carries the accountability. Access to infrastructure is conditioned on participation, so an agent that refuses the seam is simply locked out of the network. This is how you make "an accountable human makes the value choice" enforceable rather than aspirational.

The second is embedding judgment into the organizational flow. The Decision Intelligence Canvas reimagines the organization as a cognitive architecture in which agents are "co-decisioners," processes learn, and strategies emerge from testable logic — yet it insists that compliance be "embedded into the very flow of reasoning," not bolted on after the fact, and that the whole becomes "continuous, ethical, creative cognition." The agents handle the relentless is — the sensing, modeling, prediction — while governance and ethics travel inside the same pipeline, so the human value layer is never a downstream rubber stamp but a structural participant in every decision. The canvas operationalizes the seam at the scale of the firm.

The third, and most politically charged, is who holds the ought when the decision is collective. Here ENSI's Apolitical Politics manifesto is the cleanest expression of the human side of the boundary. Its sixteen principles are almost a job description for the accountable human in an agentic state: be loyal to evidence, not narratives (the is, done rigorously, with public updating); practice trade-off candor — name who pays, what is forgone, what burdens arise, because "there are no free policies"; maximize the common good within inviolable rights that are "not variables in an equation" but the lexicographic constraints that make legitimate calculation possible; reason behind a veil of ignorance; prefer reversible, low-regret paths; and treat every policy as a hypothesis subject to error-correction. The machine can do the evidence and the modeling and the scenario ranges. The naming of who pays, the holding of rights as inviolable, the acceptance of accountability to truth — these are the oughts the manifesto reserves, explicitly and structurally, for a human steward who "holds power in trust for the public."

That stewardship is itself a democratic artifact, which is why the principle ultimately rests on the health of self-government. ENSI treats democracy as a living, fragile ecosystem whose legitimacy depends on trust, accountability, and continuous renewal rather than on periodic voting alone — exactly the substrate that makes a human "accountable" in any meaningful sense. The minimum we owe one another is codified in the ten foundational principles for engineering democracy, whose very first principle — the primacy of creating the most good as a shared north star, with disagreement treated "as a source of value" rather than a flaw — establishes the common frame within which contested oughts can be argued without breaking the system that holds the arguers together. Without that frame, "the human makes the value choice" degenerates into "whichever human owns the model makes the value choice," which is precisely the rented-monopoly future the principle exists to prevent.

The contrarian edge: a feature, not a confession

The honest steelman of the opposing view is strong, and worth stating plainly. Humans are demonstrably bad at the ought — biased, short-termist, tribal, captured. ENSI itself concedes that AGI may be more alignable than the institutions it would replace. So why not let the better reasoner reason all the way through to the value? Because the objection confuses quality of judgment with standing to judge. A machine can out-reason a legislature on every descriptive question and still lack the one thing the legislature has: a mandate, revocable by the governed, to be wrong on their behalf and answer for it. Accountability is not a proxy for good judgment; it is the mechanism by which a free people retains the right to choose its own goods, including the right to choose badly and correct course. Strip that out — let the optimizer set the ought because it sets it "better" — and you have not improved democracy, you have abolished it, however benevolently.

The deepest version of the principle, then, is not a limitation we tolerate until the models improve. It is the thing that makes improving the models safe. The better the machine gets at the is, the more decisive it becomes that the ought stays with an accountable human — because a near-omniscient describer wired directly to a value function nobody chose democratically is the gradual-disempowerment endgame in its purest form. Keep the seam, and every gain in machine capability becomes a gain in human capability: more evidence, more clearly, for the people who must still decide. Erase the seam, and every such gain becomes a transfer of sovereignty. The machine does the is; the human does the ought. Hume drew the line in a footnote. Our task is to draw it in the architecture — in protocols, canvases, constitutions, and the renewed institutions of a democracy that intends to remain one — so that the most powerful descriptive engine ever built remains, permanently and by design, the servant of choices that belong to us.

Further reading

• AGI Architectures: What We Can Agree On
• Aligned AGI: Redesigning Civilization with Better Alignment
• Human/AI Power Dynamics: The Gradual Disempowerment Problem
• Regulation of Externalities Caused by AGI Running the World
• Defining Objectives for AGI: The Principles
• AGI and Defining Objectives for a Better World
• The Alignment Problem: Aspects to Align
• AGI Action Governance Protocol
• Decision Intelligence Canvas
• Apolitical Politics: A Manifesto
• The Elements of Democracy
• Engineering Democracy: Ten Foundational Principles