AGI Architectures: What We Can Agree On
October 23, 2025
Artificial general intelligence is no longer a speculative abstraction. The last decade of scaling laws, multimodal pretraining, and agentic scaffolding has translated vague philosophical debates into engineering trajectories. What once lived in academic essays now lives in code, in trained models, and in observable failure modes. The remaining question is not whether we will attempt general intelligence, but what structural commitments any such system must satisfy to function in the wild and not collapse under distribution shift, complexity, or society’s constraints.
The emerging picture from theory, systems, and empirical convergences is that AGI is not one trick — not a single “giant model” or a single training recipe — but a composite control architecture. Its core will integrate predictive world-models, explicit planning over those models, and mechanisms for continual self-improvement. Around this core sit layers for memory, tool-use, embodiment, generalization, and social reasoning — not as afterthoughts, but as co-equal conditions for operating in unbounded environments.
The same literature also converges on a second meta-fact: intelligence that does not self-monitor and self-correct is brittle, and brittle intelligence fails catastrophically when scaled. That is why reflectivity, uncertainty modeling, and verifiers are not “safety extras” but structural preconditions for reliability. An AGI that cannot detect that it might be wrong is already an unaligned system.
A third convergence concerns economics, not philosophy: most high-value applications are multi-agent, regulated, and dynamic. That implies that social intelligence — modeling other agents, norms, and institutions — is as central to AGI design as perception or planning. Systems that cannot reason about incentives, constraints, and negotiated equilibria cannot make good decisions in human domains.
A fourth convergence concerns scalability and realism: pure feed-forward reasoning without deliberation collapses under long horizons. Hence, search survives — as MCTS in control, as tree-of-thought in language, as active inference in embodied agents. Planning and search are the prostheses that convert pattern recognition into strategic behavior.
A fifth convergence is compression and composability as the engine of generality. World-models compress reality; hierarchical controllers compress temporal structure; distillation compresses competence; retrieval compresses knowledge. Every scalable subsystem reduces dimensionality while retaining decision-relevant invariants.
A sixth convergence is grounding. Whether through robotics, simulated sandboxes, or controlled tool-interfaces, AGI must close a perception-action loop that allows hypotheses to be tested and corrected. Ungrounded language alone cannot stabilize semantics or enforce causal beliefs.
And finally, a seventh convergence: safety is architectural. Oversight, containment, constitutional constraints, capability gates, and logged deliberation will not be retrofits; they will be first-class components in the system diagram. The design of AGI is therefore indistinguishable from the design of aligned AGI: the two are the same engineering problem.
AGI needs an internal predictive/causal model of the environment
Enables simulation, counterfactuals, planning, and transfer
Implemented via latent dynamics models, structured memory, large-corpus abstractions
Learned heuristics alone are insufficient for long-horizon control
Explicit search (tree search / ToT / ReAct) dramatically improves success
Planning is the source of “non-myopic” intelligence
In-context learning already behaves like meta-learning
Practical AGI must adapt both at inference and across lifetimes
Reflective rewrite (Gödel/Hyperon) is the end-state of self-improvement
Not benchmark-generalization but task / modality / embodiment / domain generality
Reuse of abstractions across transfers is the functional definition of “general”
Embodied & multimodal training appears to boost systemic generalization
Cognition decomposes into reusable modules and time scales
Options/subgoals reduce credit assignment and improve interpretability
Modular stacks allow targeted safety, debugging, and reuse
External tools become extensions of cognition (APIs, search, code, simulators)
Agents must learn when/why/how to call tools and reuse outputs in reasoning
Retrieval is memory; execution is “extended action”
Working, episodic, semantic, and external memory are distinct needs
Episodic caches & retrieval increase sample-efficiency and factuality
Long-form tasks require revisitable, inspectable memory — not pure parametrics
Semantics must be tied to perception and action (physical or simulated)
Embodiment yields causal learning and reduces hallucination
Multi-embodiment training produces transferable competence
Objective design shapes reachable cognitive regimes
RLHF/CAI/DPO = practical methods for norm-compliance
Debate: “reward is enough” vs “scalar reward is insufficient” — unresolved
AGI must know when it does not know (epistemic)
Drives safer action, active exploration, and abstention/escorts to tools/humans
Ensembles, MC-dropout, OOD detection are current workhorses
Intelligence is not only amortized heuristics — search must stay in the loop
AlphaZero/MuZero and ToT/Self-Consistency prove this pattern generalizes
Search introduces correctability and verifiability inside cognition
Abstraction = discarding detail while preserving decision-relevant structure
Scaling laws & compute-optimal training formalize this principle
Distillation transfers competence; bottlenecks enable reuse and control
Systems must critique, verify, and revise their own chains of thought/actions
Debate, verifiers, process-supervision reduce silent reasoning failures
Confidence/abstention enables risk-aware action and corrigibility
Real problems are multi-agent; AGI must model other minds & institutions
Role-based and population training yield robustness and specialization
Cooperation/competition structure drives emergent norms and strategies
Policy filters, verifiers, capability gates, sandboxed tools, audit trails
Supervisory layers sit on the execution path, not post-hoc
Safety is part of the architecture, not an after-training patch
A. Description
An AGI must maintain an internal, compressed causal/predictive model of its environment (a “world-model”) to simulate consequences, abstract regularities, and support planning, tool-use, and transfer across tasks. In practice this is a latent dynamical model that predicts future observations, rewards/utility proxies, and state features. OpenReview+2arXiv+2
B. What most authors agree on (with examples)
Predictive modeling is the core substrate. LeCun’s roadmap explicitly centers a “configurable predictive world model” trained self-supervised, paired with actor/critic heads. (“…autonomous intelligent agents… configurable predictive world model…”) OpenReview
Models should support imagination/rollouts. World Models trains a generative model and shows policies can be trained “entirely inside of [a] hallucinated dream.” arXiv
General algorithms benefit from learning an environment model. DreamerV3 “learns a model of the environment and improves behavior by imagining future scenarios,” then transfers across 150+ tasks, including Minecraft from scratch. arXiv
Even theory targets a universal predictor. AIXI fuses Solomonoff induction with sequential decision theory; the agent plans using a mixture over computable world-hypotheses. arXiv+1
C. Why it’s essential (multiple angles)
Sample-efficiency: modeling latent dynamics reduces trial-and-error cost in long-horizon tasks. arXiv
Counterfactual reasoning: simulating “what-ifs” under interventions is necessary for causal control. OpenReview
Transfer/generalization: abstract state that’s reusable across tasks, modalities, and embodiments. arXiv
Safety hooks: a model that predicts consequences enables constraint checking and risk-aware lookahead. OpenReview
D. How far are we right now
Research platforms: DreamerV3 and successors show strong generality in continuous control, Atari, DM Lab, and open-world Minecraft—without domain-specific tuning. arXiv
Reality gaps remain: world-models still struggle with long-term memory, partial observability at human scales, and complex, multi-agent social worlds. (Imagination is still short-horizon and brittle outside benchmarks.) arXiv
LLMs: text-only LMs implicitly learn world regularities but lack persistent, verifiable latent state and grounded sensorimotor learning by default. LeCun’s critique highlights this gap. arXiv
E. Best architecture so far & how it works
DreamerV3 (model-based RL): learns a stochastic latent dynamics model p(zt+1∣zt,at) plus reward and value heads; improves policy by imagining rollouts in latent space, optimizing actor/critic on imagined trajectories; uses robust normalization/balancing to stabilize across domains. arXiv+1
AIXI (theoretical gold standard): uncomputable Bayes-optimal agent mixing over all computable environments; practical approximations (AIXI-tl/CTW) illustrate the “predict+plan” decomposition, but are far from scalable. arXiv+2hutter1.net+2
A. Description
Planning is explicit deliberation—searching action sequences against the model or external tools to maximize objectives under uncertainty (tree search, beam search over thoughts, look-ahead rollouts, self-evaluation). It complements amortized “reflex” policies. arXiv
B. What most authors agree on (with examples)
Planning + learning beats either alone. AlphaZero/MuZero pair a learned/value policy with tree search; MuZero plans by predicting the quantities most relevant to planning: reward, policy, value. Nature+1
LLMs need deliberative inference. Tree-of-Thoughts argues left-to-right decoding is insufficient; it treats reasoning as search over “thought” states with backtracking/lookahead, yielding large gains. arXiv
Reason–act interleaving helps. ReAct interleaves chain-of-thought with tool actions (search, calculators), letting the plan evolve as evidence arrives. arXiv
C. Why it’s essential
Long-horizon credit assignment: lookahead mitigates myopia and compounding error. arXiv
Exploration under uncertainty: planning enables hypothesis tests and information-gain actions. arXiv
Safety and verification: explicit plans can be inspected, constrained, or simulated before execution. arXiv
D. How far are we right now
Games/Sim: Superhuman planning is solved in perfect-information games (Go, Chess, Shogi) and competitive on many Atari benchmarks. arXiv
LLM planning: Prompt-level planning (ToT, ReAct) reliably boosts reasoning, but is brittle, compute-heavy, and lacks consistent guarantees on real-world tasks. arXiv+1
Open challenges: partial observability, non-stationarity, rich tool chains, and multi-agent coordination at “civilization scale” remain unsolved.
E. Best architecture so far & how it works
MuZero (planning with learned dynamics): learns a compact latent transition model and uses Monte-Carlo Tree Search over latent states; each node stores policy/value estimates from the network, guiding exploration; no explicit environment rules are required. arXiv+1
AlphaZero (planning with policy/value nets): similar MCTS but with known rules; trains by self-play, iterating between improving the net and strengthening the search. arXiv
For LLMs: Tree-of-Thoughts as the current “best-of-breed” inference-time planner—structured branching over thoughts with self-evaluation and backtracking; ReAct when tool-use is integral to planning. arXiv+1
A. Description
AGI will improve itself at multiple levels: (i) fast, in-context adaptation during inference (learning from a few examples/instructions without weight updates); (ii) slow, across episodes via gradient-based meta-learning, finetuning, or architectural rewrites; (iii) reflective, where the system edits its own code/algorithms under guarantees (Gödel-style). arXiv+1
B. What most authors agree on (with examples)
In-context learning ≈ meta-learning. Evidence that Transformers implement a form of gradient-descent-like adaptation internally—“learn in their forward pass.” arXiv+1
Formal self-improvement is a coherent ideal. The Gödel Machine frames a provably optimal self-modifying agent that rewrites itself only after proving net utility gain. (“…self-referential, self-improving, optimally efficient problem solvers…”) arXiv
Practical AGI programs aim for reflective rewrite. OpenCog Hyperon couples a metagraph memory (Atomspace) with a meta-language (MeTTa) designed for reflective metagraph rewriting—i.e., the system can transform its own cognitive procedures. arXiv+2arXiv+2
C. Why it’s essential
Distribution shift resilience: continuous adaptation prevents rapid performance decay off-distribution. arXiv
Data/compute efficiency: reusing priors and learning algorithms accelerates skill acquisition. University of Edinburgh Research
Open-endedness: reflective improvement enables lifelong learning and capability growth without hand-engineering. arXiv
D. How far are we right now
Fast path: strong in-context adaptation in large Transformers is now well-documented (mechanistic links to GD/Bayesian inference continue to firm up). arXiv
Slow path: routine post-training (RLHF/RLAIF, DPO), tool-use augmentation (Toolformer) and dataset-driven “self-refine” loops give steady gains—but are still externally orchestrated. arXiv
Reflective path: Gödel-style provable self-rewrite remains theoretical; Hyperon’s reflective rewriting is an active engineering effort rather than a scaled demonstration. arXiv+1
E. Best architecture so far & how it works
In-context meta-learner (Transformer view): pretraining on broad task mixtures induces mechanisms (e.g., induction heads) that implement implicit optimization during inference; recent analyses show equivalence to preconditioned gradient descent in toy regimes—i.e., the model “learns how to learn” without weight updates. arXiv+1
Reflective program-space AGI (conceptual): Gödel Machine provides the cleanest formal target (proof-guided self-modification); OpenCog Hyperon is the most explicit practical blueprint (MeTTa programs as subgraphs in Atomspace; cognitive processes are themselves rewriteable data). arXiv+2arXiv+2
A) Description
AGI won’t just “fit” a benchmark; it must systemically generalize across tasks, data modalities, embodiments, and objectives with minimal re-engineering—ideally by reusing common abstractions (concepts, skills) and quickly acquiring new ones. This view spans classic AGI (NARS), modern scaling (CLIP/Flamingo), and embodied LLMs (Gato/PaLM-E). arXiv+5cis.temple.edu+5arXiv+5
B) What most authors agree on (with examples)
Cross-task/embodiment reuse is mandatory. Gato trains a single policy across 600+ tasks/modalities/embodiments using one set of weights. arXiv+1
Multimodal pretraining yields broad transfer. Flamingo and CLIP show large gains in few/zero-shot transfer by aligning images↔text at scale. NeurIPS Proceedings+1
Embodiment improves grounding & transfer. PaLM-E interleaves continuous sensory state with language; reports positive transfer from joint multimodal/robotics training. arXiv+1
AGI must work under scarce knowledge/resources. NARS formalizes “AIKR”—operating with insufficient knowledge and resources as a design principle for generality. cis.temple.edu+1
Benchmarks should measure skill-acquisition efficiency, not just skill. Chollet’s ARC reframes “general intelligence” as the efficiency of learning new tasks from limited priors. arXiv+1
C) Why it’s essential
Reality is open-ended: new tasks/ontologies constantly appear.
Data/compute efficiency: reusing abstractions beats per-task finetunes.
Safety & robustness: broader priors reduce brittle shortcut solutions.
Economic value: cross-domain reuse underpins rapid deployment.
D) How far are we now
Strong: zero/few-shot perception generalization (CLIP, Flamingo). Proceedings of Machine Learning Research+1
Promising: policy transfer across embodiments (Gato), grounded multimodal reasoning (PaLM-E). arXiv+1
Gaps: causal/generalizable reasoning across long horizons; out-of-distribution compositionality (ARC-style) remains hard.
E) Best architectures so far & how they work
CLIP/Flamingo (foundation for perception-side transfer): dual encoders (CLIP) or interleaved V-L training (Flamingo) learn shared representations enabling zero/few-shot transfer without task-specific heads. Proceedings of Machine Learning Research+1
Gato (policy-side transfer): a single Transformer policy tokenizes observations/actions across tasks; context decides whether to emit text, torques, or button presses. arXiv
PaLM-E (embodied multimodal LM): encodes continuous robot state + vision into a language backbone; joint training yields positive transfer across V-L-robotics tasks. arXiv
A) Description
AGI will decompose cognition into modules and levels of temporal abstraction: perception → memory → valuation → planning → action, with hierarchical control (slow “manager” setting subgoals; fast “workers” executing). This appears in hierarchical RL (Options, FeUdal Networks), cognitive architectures (LIDA), and modern roadmaps (LeCun). OpenReview+3UMass Amherst+3arXiv+3
B) What most authors agree on (with examples)
Temporal abstraction helps long-horizon tasks. The Options framework formalizes temporally extended actions (options) inside RL. UMass Amherst+1
Manager/worker splits stabilize learning. FeUdal Networks learn high-level goals in latent space (Manager) that a Worker executes at fast timescales. Proceedings of Machine Learning Research
Cognitive cycles require modular stages. LIDA (GW-style architecture) cycles through perception→attention→action selection with distinct memory modules. cse.buffalo.edu+1
Modern blueprints retain modularity. LeCun’s world-model + actor + configurator proposal explicitly advocates hierarchical joint-embedding and intrinsic-motivation modules. OpenReview
C) Why it’s essential
Credit assignment over long horizons via subgoals.
Reusability: learned skills/options become callable primitives.
Interpretability/safety: modular plans and goal interfaces are inspectable.
Scalability: different modules optimize at different timescales.
D) How far are we now
Mature theory & demos: Options/FeUdal show large gains on Atari/DM-Lab and remain standard references. Proceedings of Machine Learning Research+1
Cognitive stacks exist but are narrow: LIDA-style systems run end-to-end but haven’t scaled to web-scale learning. cse.buffalo.edu
Frontier practice: many state-of-the-art systems implement de-facto modularity (separate retrievers, planners, tool-APIs), but interfaces are still ad-hoc.
E) Best architectures so far & how they work
Options framework: represents skills as semi-MDP options with initiation sets, intra-option policies, termination; standard RL learns over both primitive actions and options. ScienceDirect
FeUdal Networks (FuN): a Manager emits goal vectors in latent space at a low frequency; a Worker is rewarded for moving latent state toward that goal—decoupling timescales and easing long-term credit assignment. Proceedings of Machine Learning Research
LIDA (GW implementation): distinct perceptual/episodic/procedural memories and an attention/“broadcast” phase select contents for action selection—i.e., modular control at the cognitive level. cse.buffalo.edu
A) Description
Future AGI will treat external tools (search engines, calculators, code interpreters, databases, robots, simulators) as cognitive extensions—learning when to call which tool with what arguments, and how to fuse results into ongoing reasoning and memory. arXiv+1
B) What most authors agree on (with examples)
Self-taught API use works. Toolformer fine-tunes LMs to decide if/when/how to call APIs in a self-supervised way (few exemplars per API). arXiv+1
Reasoning↔acting must interleave. ReAct interleaves chain-of-thought with actions (e.g., Wikipedia lookups), reducing hallucinations and improving task success. arXiv+1
External memory boosts knowledge tasks. RAG couples a generator with a dense retriever to ground outputs in updatable corpora; RETRO pushes retrieval into both training & inference to rival much larger LMs. arXiv+2NeurIPS Proceedings+2
C) Why it’s essential
Performance: specialized tools (math, search, code) beat parametric recall.
Faithfulness & provenance: retrieval provides citations and updateability.
Sample/compute efficiency: spares the model from memorizing facts.
Scaffolding for agency: tools become “hands and eyes” for planning.
D) How far are we now
Reliable gains on QA, reasoning, and interactive tasks with ReAct/ToT + RAG style agents, though orchestration remains prompt-heavy and brittle. arXiv+1
Scaling lessons: RETRO shows retrieval can substitute parameters at training time (25× fewer params vs. GPT-3 on Pile). arXiv
Open issues: unified routing (which tool when), latency/cost trade-offs, and safety/permissioning.
E) Best architectures so far & how they work
Toolformer (self-supervised API learner): seed a few API exemplars → LM proposes candidate calls in pretraining corpora → filter by utility → fine-tune so the model learns policies for when/what/how to call; integrates results back into next-token prediction. arXiv
ReAct (reason-act interleaving): prompt format induces alternating Thought → Action → Observation loops; tools feed back into the reasoning trace, enabling correction and exploration. arXiv
RAG/RETRO (external memory):
RAG: dense retriever fetches passages from a vector index; generator conditions on them (either fixed per sequence or token-adaptive), improving factuality/diversity. NeurIPS Proceedings
RETRO: retrieval baked into the Transformer at training & inference; looks up nearest neighbor chunks for each context window, achieving GPT-3-level perplexity with far fewer parameters. arXiv
A) Description
AGI needs multiple memory systems with different purposes and time-scales: fast working memory for scratch-space during reasoning; episodic memory for storing/replaying experiences; semantic/long-term memory for stable knowledge; and external memory it can read/write (vector stores, knowledge graphs, databases). In practice this spans differentiable memories (NTM/DNC), episodic caches (NEC/MERLIN), and retrieval systems (RAG/RETRO). arXiv+4arXiv+4Nature+4
B) What most authors agree on (with examples)
Neural nets benefit from explicit external memory.
Neural Turing Machines (NTM) and Differentiable Neural Computers (DNC) couple a controller to an addressable memory matrix, enabling algorithmic tasks (copying, sorting, graph queries) beyond standard RNN/LSTM capacity. Stanford University+3arXiv+3arXiv+3
Episodic memory boosts sample-efficiency.
Neural Episodic Control (NEC) stores value estimates in a fast key–value episodic table, dramatically speeding RL compared to purely parametric value functions. MERLIN adds predictive memory for partially observed tasks. arXiv+2Proceedings of Machine Learning Research+2
Retrieval can substitute params and improve faithfulness.
RETRO conditions generation on retrieved chunks from a massive corpus, matching GPT-3-scale performance with 25× fewer parameters; retrieval also underpins grounding and updatability. arXiv
C) Why it’s essential
Reasoning capacity: scratchpads and memory address long chains of thought.
Sample/compute efficiency: episodic caches re-use experience.
Factuality & updateability: retrieval prevents stale parametric “knowledge.”
Generalization: different stores support different forms of transfer.
D) How far we are
Mature prototypes: NTM/DNC show algorithmic manipulation with external RAM; NEC/MERLIN deliver big data-efficiency gains in RL and long-horizon POMDPs. Nature+2Proceedings of Machine Learning Research+2
At scale: RETRO demonstrates that retrieval can replace parameters while improving knowledge-intensive tasks; RAG-style pipelines are standard in production assistants. arXiv
Gaps: unified memory routing (what to store/where/when), write policies, and lifelong de-duplication remain open research; standardized memory benchmarks are still evolving. arXiv
E) Best architectures so far & how they work
DNC (external differentiable memory): a neural controller learns content- and location-based addressing to read/write a memory matrix; end-to-end differentiable, enabling learned data-structure manipulation and long-term storage. Nature+1
NEC/MERLIN (episodic & predictive memory for RL): NEC keeps a KNN-like table of state embeddings→Q-values for rapid reuse; MERLIN learns a predictive latent model that guides what gets stored and supports long-duration tasks under partial observability. Proceedings of Machine Learning Research+1
RETRO (retrieval-enhanced Transformer): augments each context with nearest-neighbor text during training and inference, attaining GPT-3-level perplexity with a much smaller LM. Ideal blueprint for AGI-grade semantic LTM. arXiv
A) Description
Even if much “thinking” happens symbolically, AGI must anchor symbols to sensorimotor reality (physical or simulated) and act to test hypotheses. Modern systems bind language models to vision, proprioception, and action streams so that words point to manipulable world-state. arXiv
B) What most authors agree on (with examples)
One policy, many embodiments is possible.
Gato trains a single Transformer across 600+ tasks and embodiments (Atari, dialogue, robot arm). Same weights, different output tokens (text, torques, buttons). arXiv+1
Multimodal LMs can become embodied LMs.
PaLM-E injects continuous robot state and visual tokens directly into a language backbone and shows positive transfer from V&L to robotics. arXiv+1
Web-scale VLMs transfer to action.
RT-2 distills knowledge from internet-scale VLMs into Vision–Language–Action policies that control real robots, improving generalization to novel instructions. arXiv+1
Open-ended skill acquisition emerges in rich worlds.
Voyager (Minecraft) builds an ever-growing skill library via automatic curricula and self-verification, then reuses those skills in new worlds. MineDojo provides the benchmark + internet knowledge. minedojo.org+3arXiv+3voyager.minedojo.org+3
C) Why it’s essential
Grounded semantics: tie words to objects/actions/affordances.
Causal learning: interventions/retries → better world models.
Robustness: interactive feedback reduces hallucinations.
Economic value: robotics, UI automation, scientific instruments.
D) How far we are
Evidence of transfer: PaLM-E and RT-2 show text/vision knowledge improving robot control; Gato demonstrates a working multi-embodiment policy. arXiv+2arXiv+2
Open problems: long-horizon autonomy, safe exploration, reliable tool-use in unstructured environments, and affordable real-world data collection.
E) Best architectures so far & how they work
PaLM-E (Embodied Multimodal LM): learn encoders for images and robot state; interleave with text tokens; joint training teaches the LM to plan/manipulate using grounded inputs while retaining general language/V&L skills. arXiv+1
RT-2 (V-L-A policy): start from a large vision–language model, then fine-tune it end-to-end so the same backbone maps observations→action tokens; leverages web knowledge for semantic generalization. arXiv+2robotics-transformer2.github.io+2
Voyager + MineDojo (open-ended skill library): use an LLM to iteratively propose programs, self-verify, and store successful skills in a library; MineDojo supplies tasks + internet knowledge for broad transfer. arXiv+1
A) Description
What AGI optimizes shapes what it becomes. Two contrasting theses dominate: (i) “Reward is enough”—scalar reward maximization can, in principle, produce general intelligence; (ii) “Scalar reward is not enough”—we need multi-objective or preference-based objectives to avoid unsafe shortcut solutions. Modern practice centers human/AI preference learning (RLHF, CAI, DPO). ScienceDirect+1
B) What most authors agree on (with examples)
The debate:
Silver/Sutton et al. argue that maximizing reward can yield most facets of intelligence; Vamplew et al. counter that single-scalar reward is insufficient and risky for AGI, advocating explicit multi-objective formulations. ScienceDirect+1
Preferences are practical signals.
Christiano et al. show deep RL from human preferences can teach complex behaviors with minimal oversight. InstructGPT operationalizes this at scale (RLHF) for instruction-following LMs. arXiv+1
Constitutional supervision reduces human labor.
Anthropic’s Constitutional AI replaces much human feedback with an AI-critique guided by a rule set (constitution). arXiv+1
Simpler alignment objectives exist.
DPO optimizes preferences without explicit reward modeling/RL, matching or beating PPO-based RLHF on several tasks. arXiv+1
C) Why it’s essential
Capability control: objectives/constraints select reachable cognitive regimes.
Safety: mitigates specification gaming & proxy-hacking. Google DeepMind
Scalability: preference learning and constitutions reduce expert reward engineering.
Societal acceptability: encodes norms into otherwise power-seeking learners.
D) How far we are
Industrialized pipelines: RLHF/RLAIF/CAI are standard in frontier LLMs (and new wrappers like constitutional classifiers reinforce them). NeurIPS Proceedings+2arXiv+2
Theoretical questions remain: convergence/robustness under distribution shift, multi-objective trade-offs, and formal guarantees beyond narrow settings; “reward is enough?” remains contested. ScienceDirect+1
E) Best architectures so far & how they work
RLHF / InstructGPT pipeline: collect pairwise human preferences → train a reward model → optimize the base LM with RL (e.g., PPO) regularized toward pretrain distribution; improves helpfulness/harmlessness. NeurIPS Proceedings
Constitutional AI (RLAIF): define a constitution (principles); use an AI to critique and revise model outputs per principles → supervised fine-tune → optional RL phase using AI feedback, reducing human labels. arXiv+1
DPO: cast preference learning as a closed-form policy update (no explicit reward model, no RL loop); optimize a classification-style loss on chosen vs. rejected outputs to align the LM stably and efficiently. arXiv+1
A) Description
AGI must represent and act under uncertainty: epistemic (what the model doesn’t know) and aleatoric (inherent noise). In practice this means well-calibrated predictions, OOD awareness, and decision-making that accounts for belief distributions—not just point estimates. Surveys standardize the taxonomy and methods (Bayesian approximations, ensembles, evidential models, calibration, OOD detection). arXiv+1
B) What most authors agree on (with examples)
Simple baselines work shockingly well. Deep ensembles give strong, calibrated uncertainty and flag OOD inputs better than many Bayesian approximations. arXiv+2arXiv+2
Dropout ≈ Bayesian approximation. Test-time dropout can be read as approximate Bayesian inference, yielding usable uncertainty without architectural surgery. arXiv+2Proceedings of Machine Learning Research+2
OOD detection is a first-class requirement. Generalized OOD surveys argue safety-critical systems must detect distribution shift and abstain / escalate. arXiv+1
C) Why it’s essential
Safer decisions: act conservatively when beliefs are wide.
Exploration: target information gain where uncertainty is high.
Robustness to shift: avoid overconfident errors off-distribution.
Tool routing: choose retrieval / human-in-the-loop when uncertain.
D) How far we are
Strong ingredients: deep ensembles and MC-dropout scale and improve calibration/OOD detection across vision and language. arXiv+1
Ecosystem maturity: multiple up-to-date surveys (UQ & OOD) synthesize methods and gaps; benchmarks are broadening beyond “novel class” only. arXiv+2arXiv+2
Gaps: unified end-to-end uncertainty propagation in agent loops (planning, tool-use, memory writes) is still ad-hoc.
E) Best current architecture(s) & how they work
Deep Ensembles: train KKK independently-initialized nets; at inference aggregate mean/variance. Captures epistemic uncertainty, improves calibration, and flags OOD. arXiv
MC-Dropout: keep dropout active at test time; multiple stochastic passes approximate a posterior predictive. Low-friction retrofit for existing models. arXiv
UQ + OOD stack for agents (pattern): model with ensembles/MC-dropout → calibrate → attach OOD detector → policy/planner uses uncertainty for risk-aware search or abstention. (Framework summarized in the surveys.) arXiv+1
A) Description
AGI won’t be pure feedforward. It will interleave learned heuristics (policies/values in networks) with explicit search/deliberation (tree search, hypothesis branching, self-evaluation). This hybrid shows up from AlphaZero/MuZero in games to Tree-of-Thoughts / Self-Consistency in LLMs. arXiv+1
B) What most authors agree on (with examples)
Search+learning beats either alone (control). AlphaZero couples a policy/value net with Monte-Carlo Tree Search; MuZero learns the model it searches over and predicts policy/value/reward—no rules given. arXiv+3arXiv+3Science+3
Deliberative decoding helps (language). Tree-of-Thoughts frames inference as a search over intermediate “thought” states; Self-Consistency samples multiple chains of thought and votes, yielding big gains on math/logic. arXiv+3arXiv+3arXiv+3
C) Why it’s essential
Long-horizon credit assignment: lookahead reduces myopia.
Systematic exploration: branch & backtrack rather than greedy decode.
Verifiability: plans/thoughts can be inspected, constrained, and simulated.
D) How far we are
Solved niches: superhuman planning in perfect-info games; robust MuZero across Atari, Go, chess, shogi. Science+1
Emergent but brittle in LLMs: ToT / Self-Consistency are powerful prompts, but costy and sensitive to hyperparameters; tool-augmented planning remains orchestration-heavy. arXiv+1
E) Best current architecture(s) & how they work
MuZero: learn a latent transition g(ht,at)→ht+1 and heads for reward/value/policy; perform MCTS over latent states; train by matching search targets. Scales without environment rules. Nature+1
AlphaZero: policy/value net + MCTS + self-play; iteratively improve the net with search-amplified targets. arXiv+1
Tree-of-Thoughts / Self-Consistency (LM inference): structure decoding as branch–evaluate–prune over thoughts; sample diverse chains, then marginalize to the most consistent answer. Drop-in for existing LMs. arXiv+1
A) Description
Across learning theory and practice, compression/abstraction—minimizing description length while preserving predictive/decision utility—appears central to intelligence. Two pillars: the Information Bottleneck (learn representations that compress inputs while retaining task-relevant info) and scaling laws (loss follows smooth power laws in parameters/data/compute; compute-optimal training favors more data, not just more params). Distillation operationalizes compression into smaller models. arXiv+2arXiv+2
B) What most authors agree on (with examples)
Deep nets spend much of training compressing. The Information-Bottleneck view shows layers move toward compressive, task-relevant representations as training proceeds. arXiv
Performance scales predictably with size/data/compute. Kaplan et al. show power-law scaling; Hoffmann et al. (Chinchilla) show many frontier LMs were under-trained on tokens and that compute-optimal training balances params and data. arXiv+3arXiv+3arXiv+3
Knowledge can be compressed. Distillation transfers “dark knowledge” from a large/ensemble model into a smaller one with minimal loss. arXiv+1
C) Why it’s essential
Generalization: compressed features discard spurious detail, keep causal structure.
Efficiency: compute-optimal training and distillation reduce costs.
Systems design: compressed, modular reps travel across tools/memory/agents.
D) How far we are
Well-validated laws: scaling laws and Chinchilla-style training now shape frontier model design and budgets—even as critics (e.g., LeCun) argue scaling alone won’t yield reasoning without world models/planning. arXiv+2arXiv+2
Operational practice: distillation and representation bottlenecks are standard in production; principled MDL/IB objectives in giant models remain active research.
E) Best current architecture(s) & how they work
Compute-optimal LM training (Chinchilla rule): for a fixed compute budget, scale data with params roughly 1:1 (double params → double tokens). Train smaller-but-well-read models for better accuracy and cheaper inference. arXiv+1
Information-Bottleneck-guided reps: train encoders whose intermediate layers maximize I(Z;Y) while minimizing I(Z;X), yielding compact, task-sufficient features; useful design lens for multimodal AGI stacks. arXiv
Knowledge Distillation pipeline: teacher (or ensemble) produces soft targets → student optimizes KL to teacher logits (optionally with hard labels) → deploy smaller, faster agent with comparable competence. arXiv
A) Description
An AGI must continuously assess its own reasoning and actions—estimating confidence, checking intermediate steps, critiquing plans, and revising itself. Reflectivity spans: (i) local checks (verify a proof step, unit-test a function), (ii) global checks (is the plan still on target?), and (iii) meta checks (did my method work; should I switch strategies?).
B) What most authors agree on (with examples)
Critic loops improve reliability. “Reflexion”/self-critique and verifier models reduce reasoning errors by iteratively reviewing and editing outputs.
Process supervision beats outcome-only. Rewarding intermediate steps (proof states, tool traces) trains models to notice and fix local errors.
Debate/adversarial review exposes flaws. “AI Safety via Debate,” multi-agent critiques, and jury/verifier schemes systematically surface wrong steps.
Confidence estimation matters. Calibrated confidence (ensembles, MC-dropout) and abstention thresholds govern when to escalate to tools or humans.
C) Why it’s essential
Prevents silent failures in long chains of thought.
Enables corrigibility: the system knows when it might be wrong.
Supports safe autonomy: reflective checks gate risky actions.
Data efficiency: learning from one’s own critiques accelerates improvement.
D) How far we are
Strong empirical boosts from self-critique, verifier-guided decoding, self-consistency voting, and debate prompts—especially in math/code/QA.
Still brittle: gains can be prompt- and budget-sensitive; verifiers themselves can be fooled; calibration in open-world tasks is uneven.
E) Best architecture so far & how it works
Actor–Critic–Editor loop (ACE):
Actor proposes a solution/plan (with tool calls).
Critic/Verifier tests steps (unit tests, theorem checkers, retrieval grounding, constraints).
Editor revises the trace; loop until time/quality threshold.
Add confidence heads (or ensembles) to decide when to stop/abstain, and process-supervision training so the critic learns to spot granular faults.
Debate-plus-Verifier: two reasoners argue; a separate verifier (or rules/ground truth) adjudicates; winner’s trace trains the policy.
A) Description
Real environments are social. AGI must model other agents’ beliefs, incentives, norms, and commitments, and coordinate/compete in teams, markets, and institutions. Architecturally: (i) theory-of-mind inference, (ii) communication protocols (messages, shared memory), (iii) mechanisms design (contracts, auctions), and (iv) population training (self-play, leagues).
B) What most authors agree on (with examples)
Self-play creates robust skills. AlphaZero/AlphaStar-style leagues cultivate strategies that generalize across opponents.
Agent societies outperform monoliths on complex workflows. Multi-agent frameworks (e.g., role-specialized “planner–solver–reviewer,” CAMEL/AutoGen-style) reliably beat single-agent baselines on decomposition-heavy tasks.
Emergent conventions/norms matter. Large agent populations in sandboxes exhibit coordination conventions and division of labor—useful for planning with/against humans.
ToM/intent modeling is a capability frontier. Reasoning over others’ hidden goals/states raises success in negotiation, assistance, and safety-critical oversight.
C) Why it’s essential
Economic reality: most valuable tasks are team- and market-embedded.
Robustness: diverse agents catch each other’s failures.
Scale: parallel specialization yields throughput and quality.
Alignment: social feedback and norms constrain misbehavior.
D) How far we are
Mature in games/simulations: self-play leagues, population-based training, and curriculum generation are proven.
Promising in tools/software: role-based LLM teams routinely solve harder, longer tasks (codebases, research, analytics) than solo agents.
Gaps: stable communication protocols, reliable intent inference, and cost-aware task allocation in dynamic, real-world contexts.
E) Best architecture so far & how it works
Role-specialized multi-agent stack:
Planner decomposes goals → tasks.
Solvers (domain-specific) execute with tools/memory.
Reviewer/Verifier checks outputs; Mediator resolves conflicts; Memory stores shared artifacts/decisions.
Use self-play and league training in simulations to stress-test strategies; adopt contracts/auctions for task assignment; track reputation for reliability.
Generative-Agents-style workspace: agents with profiles, long-term memory, and message passing; a scheduler coordinates interactions to accomplish projects.
A) Description
As capabilities grow, control layers become architectural features, not afterthoughts. Expect policy models (filters/constitutions), verifier/guard models, capability gating, sandboxed tool executors, provenance logging, evaluation harnesses, and human-in-the-loop (HITL) checkpoints welded into the agent’s control flow.
B) What most authors agree on (with examples)
Preference learning is table stakes. RLHF/DPO/Constitutional methods align objectives with human norms and reduce unsafe outputs.
Guard/Verifier stacks reduce risk. Separate models (or rules) check for policy compliance, prompt injection, data exfiltration, unsafe tools, and hallucination; retrieval provenance is used for audits.
Least-privilege execution. Tools, files, networks, and actuators are permissioned; high-impact actions require multi-stage review or HITL.
Scalable oversight is necessary. Debate, weak-to-strong supervision, and process supervision reduce human labeling load while raising reliability.
Transparent traces help governance. Storing plans, tool calls, evidence, and decisions allows audits and post-mortems.
C) Why it’s essential
Risk management: prevent catastrophic or costly actions.
Regulatory compliance & forensics: produce explainable, reviewable records.
Trust & deployment: enterprises require guarantees and controls.
Technical leverage: verifiers and policies improve capability and safety.
D) How far we are
Production-ready pieces: RLHF/DPO/Constitutional AI; robust retrieval grounding; output and input filters; sandboxed code/execution; red-team/eval suites.
Open problems: jailbreak resistance, cross-tool prompt-injection, long-horizon goal-misgeneralization, and formal guarantees for tool use and autonomy.
E) Best architecture so far & how it works
Layered Safety Controller (LSC) in front of the Agent Core:
Policy layer: input/output filters, constitutional rules, jailbreak detection.
Verifier layer: fact-checkers, tool-call validators, data-loss-prevention, prompt-injection/command-injection detectors.
Capability gate: action scoring (risk, reversibility, blast radius); require HITL or multi-agent approval for high-risk steps.
Sandboxed executors: isolated environments for code, browsing, robots; strict allow-lists and rate limits.
Audit & eval bus: immutable logs of prompts, plans, tool calls, retrieved evidence, and outcomes; periodic adversarial evals; rollback hooks.
Training alignment stack: pretrain → SFT on curated behaviors → process-supervision (reward steps, not just outcomes) → DPO/RLHF/RLAIF → post-training with safety classifiers and guard-rails.
October 23, 2025
October 21, 2025
October 17, 2025