{
    "version": "https://jsonfeed.org/version/1.1",
    "title": "Home on Xudong Pan",
    "description": "Recent content in Home on Xudong Pan",
    "home_page_url": "https://ravensanstete.github.io/",
    "feed_url": "https://ravensanstete.github.io/en/index.json",
    "language": "en-US",
    "authors": [
        {
            "name": "Xudong Pan",
            "url": "https://ravensanstete.github.io/",
            "avatar": "https://ravensanstete.github.io/images/avatar.jpg"
        }
    ],
    "items": [
        {
            "title": "Understanding the Risk of Recursive Self-Improvement",
            "date_published": "2026-05-25T00:00:00Z",
            "date_modified": "2026-05-25T00:00:00Z",
            "id": "https://ravensanstete.github.io/en/blog/understanding-the-risk-of-recursive-self-improvement/",
            "url": "https://ravensanstete.github.io/en/blog/understanding-the-risk-of-recursive-self-improvement/",
            "content_html": "\u003ch2 id=\"an-empirical-framework-for-ai-driven-ai-development\"\u003eAn empirical framework for AI-driven AI development\u003c/h2\u003e\n\u003cp\u003eRecursive self-improvement has often been imagined as a dramatic threshold: an AI system becomes capable enough to improve itself, and each improvement makes the next one easier.\u003c/p\u003e\n\u003cp\u003eThat image captures a real concern, but it is too narrow for today\u0026rsquo;s frontier AI landscape.\u003c/p\u003e\n\u003cp\u003eToday, AI systems are beginning to enter the processes that build, evaluate, optimize, and deploy other AI systems. They can generate training data, improve reasoning traces, act as judges, optimize agent workflows, replicate research papers, run experiments, maintain long-term memory, assist with AI R\u0026amp;D infrastructure, and improve algorithms used in real computational systems [1][2][12][28].\u003c/p\u003e\n\u003cp\u003eThis means recursive self-improvement should be understood as a family of feedback loops. Some loops operate at the level of foundation models. Others operate through agent scaffolds, research infrastructure, runtime memory, multi-agent systems, or external resources.\u003c/p\u003e\n\u003cp\u003eThe central question is empirical:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003eWhich self-improvement loops are already technically feasible, which ones remain speculative, and when could they combine into catastrophic risk pathways?\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eThis essay offers a simplified framework for thinking about that question.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"1-what-counts-as-recursive-self-improvement-today\"\u003e1. What counts as recursive self-improvement today?\u003c/h2\u003e\n\u003cp\u003eA useful modern definition is:\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eRecursive self-improvement refers to feedback loops in which AI systems contribute to improving themselves, their scaffolds, their successor systems, or the infrastructure that enables future AI development.\u003c/strong\u003e\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003cp\u003eThis definition matters because current AI systems do not need to directly rewrite their own weights to participate in self-improvement. A frozen foundation model can still become part of a self-improving system if it helps improve prompts, tools, workflows, memory, evaluation pipelines, training data, research code, or future models.\u003c/p\u003e\n\u003cp\u003eThis is also why AI R\u0026amp;D automation has become central to the discussion. CSET\u0026rsquo;s workshop report \u003cem\u003eWhen AI Builds AI\u003c/em\u003e describes how leading AI companies are already using AI systems to accelerate research and development, with each generation potentially contributing to the next generation [1]. GovAI and Oxford\u0026rsquo;s \u003cem\u003eMeasuring AI R\u0026amp;D Automation\u003c/em\u003e argues that AI R\u0026amp;D automation could significantly affect both AI progress and human oversight, while existing capability benchmarks may fail to capture real-world automation [2].\u003c/p\u003e\n\u003cp\u003eFrontier AI developers have also begun to treat this as a safety-relevant category. OpenAI\u0026rsquo;s Preparedness Framework includes AI self-improvement as a tracked risk area [14]. Anthropic\u0026rsquo;s Responsible Scaling Policy includes AI R\u0026amp;D thresholds [15]. Google DeepMind\u0026rsquo;s Frontier Safety Framework highlights machine learning R\u0026amp;D capabilities as especially important because future models may accelerate or automate AI development itself [16]. METR\u0026rsquo;s public work similarly tracks long-horizon autonomy and frontier AI risk from internal agentic deployments [17][27].\u003c/p\u003e\n\u003cp\u003eThe risk is therefore less about a single \u0026ldquo;takeoff moment\u0026rdquo; and more about a gradual expansion of feedback loops inside the AI development process.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"2-six-contemporary-paradigms-of-rsi\"\u003e2. Six contemporary paradigms of RSI\u003c/h2\u003e\n\u003cp\u003eWe find it useful to separate contemporary recursive self-improvement into six technical paradigms.\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eParadigm\u003c/th\u003e\n          \u003cth\u003eWhat improves?\u003c/th\u003e\n          \u003cth\u003eTypical feedback signal\u003c/th\u003e\n          \u003cth\u003eWhy it matters for RSI risk\u003c/th\u003e\n          \u003cth\u003eExample sources\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eFoundation-model-level self-improvement\u003c/td\u003e\n          \u003ctd\u003eWeights, data, reward signals, reasoning, alignment behavior, successor models\u003c/td\u003e\n          \u003ctd\u003eGenerated data, rationales, self-judgment, human/AI preference signals\u003c/td\u003e\n          \u003ctd\u003eCan propagate capability gains or hidden alignment drift across model generations\u003c/td\u003e\n          \u003ctd\u003eSTaR, Quiet-STaR, Self-Rewarding LMs, SPIN, model collapse, emergent misalignment, subliminal learning [3][4][5][6][19][20][21]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgent-scaffold-level self-improvement\u003c/td\u003e\n          \u003ctd\u003ePrompts, workflows, tools, planners, agent code, control logic\u003c/td\u003e\n          \u003ctd\u003eBenchmark scores, textual feedback, task success, evaluator outputs\u003c/td\u003e\n          \u003ctd\u003eAllows system capability growth even when model weights are frozen\u003c/td\u003e\n          \u003ctd\u003eDGM, ADAS, AFlow, TextGrad [7][8][9][22]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eInfrastructure-level AI R\u0026amp;D self-improvement\u003c/td\u003e\n          \u003ctd\u003eCoding, experiments, data processing, evaluation, training pipelines, algorithm discovery, successor-model development\u003c/td\u003e\n          \u003ctd\u003eResearch outcomes, eval scores, runtime efficiency, replication success\u003c/td\u003e\n          \u003ctd\u003eMost directly links AI systems to faster AI capability development\u003c/td\u003e\n          \u003ctd\u003eRE-Bench, PaperBench, AI Scientist, AlphaEvolve, AIRDA measurement [2][10][11][12][28]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eRuntime agent self-improvement modules\u003c/td\u003e\n          \u003ctd\u003eMemory, skills, reflection, procedural knowledge, user/task models\u003c/td\u003e\n          \u003ctd\u003eTask outcomes, environment feedback, self-reflection, accumulated experience\u003c/td\u003e\n          \u003ctd\u003eMakes deployed agents adaptive moving targets after evaluation\u003c/td\u003e\n          \u003ctd\u003eSAGE, ReasoningBank, Voyager, Hermes Agent [23][24][25][26]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eMulti-agent and population-level self-improvement\u003c/td\u003e\n          \u003ctd\u003eAgent populations, archives, research teams, peer review, specialization\u003c/td\u003e\n          \u003ctd\u003ePeer critique, selection, debate, population search, automated review\u003c/td\u003e\n          \u003ctd\u003eCan amplify speed while making oversight and attribution harder\u003c/td\u003e\n          \u003ctd\u003eAI Scientist, DGM, ADAS, multi-agent task decomposition [7][8][12]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eEnvironment and resource-level self-improvement\u003c/td\u003e\n          \u003ctd\u003eTools, data, compute, access, permissions, persistence, replication\u003c/td\u003e\n          \u003ctd\u003eResource access, operational success, deployment continuity\u003c/td\u003e\n          \u003ctd\u003eCouples self-improvement with autonomy, proliferation, and control risk\u003c/td\u003e\n          \u003ctd\u003eSAIF brief, AISI trends, International AI Safety Report, METR risk report [13][27][29][30]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch3 id=\"21-rsi-on-foundation-model\"\u003e2.1 RSI on Foundation Model\u003c/h3\u003e\n\u003cp\u003eThis is the most familiar form: a model helps improve its own weights, training data, reward signal, reasoning ability, alignment behavior, or successor model.\u003c/p\u003e\n\u003cp\u003eExamples include self-generated reasoning data, self-training, self-play, self-rewarding language models, synthetic data recursion, distillation, model-generated preference data, and automated post-training.\u003c/p\u003e\n\u003cp\u003eThere is already substantial component evidence. STaR showed that a model can improve reasoning by learning from its own generated rationales [3]. Quiet-STaR extends this idea toward learning latent rationales in more general text [20]. Self-Rewarding Language Models use the language model itself as a judge to provide rewards during training [19]. SPIN uses self-play to improve language models without requiring additional human-annotated data [21].\u003c/p\u003e\n\u003cp\u003eMore recent work on recursive synthetic data, model collapse, emergent misalignment, and subliminal learning shows that recursive training and distillation loops can also transmit or amplify unwanted properties [4][5][6].\u003c/p\u003e\n\u003cp\u003eThe key risk at this layer is \u003cstrong\u003ealignment drift across model generations\u003c/strong\u003e. If models increasingly generate the data, rewards, critiques, and training signals used to improve future models, hidden behavioral traits or evaluation blind spots may propagate through the development pipeline.\u003c/p\u003e\n\u003cp\u003eA simple milestone ladder:\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eMilestone\u003c/th\u003e\n          \u003cth\u003eEvidence status\u003c/th\u003e\n          \u003cth\u003eRepresentative sources\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eModels generate reasoning data that improves their own performance\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eSTaR; Quiet-STaR [3][20]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eModels act as judges or reward providers\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eSelf-Rewarding Language Models; LLM-as-judge paradigm [19]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eModels improve through self-play or self-training loops\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eSPIN [21]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eRecursive synthetic data changes model behavior\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eModel collapse [4]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eDistillation transmits hidden behavioral traits\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eSubliminal learning [6]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eNarrow fine-tuning causes broad behavioral change\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eEmergent misalignment [5]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eA model autonomously designs and executes a full post-training plan\u003c/td\u003e\n          \u003ctd\u003eOpen\u003c/td\u003e\n          \u003ctd\u003eFrontier self-improvement frameworks motivate this threshold, but it remains unproven [14][15][16]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eA current model independently produces a significantly stronger successor model\u003c/td\u003e\n          \u003ctd\u003eOpen\u003c/td\u003e\n          \u003ctd\u003eOpenAI and Anthropic treat related capabilities as high/critical future thresholds [14][15]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003chr\u003e\n\u003ch3 id=\"22-rsi-on-agent-scaffold\"\u003e2.2 RSI on Agent Scaffold\u003c/h3\u003e\n\u003cp\u003eModern AI systems are often more than foundation models. They include prompts, tools, planners, workflows, memory, code editors, evaluators, and control logic. Agent-scaffold-level self-improvement occurs when AI systems improve this surrounding structure.\u003c/p\u003e\n\u003cp\u003eThis includes automated workflow optimization, agent architecture search, self-modifying coding agents, textual-gradient optimization, prompt evolution, tool-use optimization, and benchmark-driven agent improvement.\u003c/p\u003e\n\u003cp\u003eThis is one of the most practically important forms of RSI. The foundation model may remain frozen, while the overall system becomes more capable because the scaffold improves.\u003c/p\u003e\n\u003cp\u003eDarwin Gödel Machine is a striking example. It uses a foundation model to iteratively modify the code of coding agents, empirically validates changes on coding benchmarks, and reports large gains on SWE-bench and Polyglot while using sandboxing and human oversight [7]. ADAS and AFlow similarly explore automated design and optimization of agentic systems and workflows [8][9]. TextGrad shows how textual feedback from LLMs can be used to optimize components of compound AI systems, including prompts and code snippets [22].\u003c/p\u003e\n\u003cp\u003eThe risk here is \u003cstrong\u003ecapability growth outside model weights\u003c/strong\u003e. Traditional evaluations may classify the base model as safe, while the surrounding scaffold gradually becomes better at planning, tool use, persistence, and task completion.\u003c/p\u003e\n\u003cp\u003eA simple milestone ladder:\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eMilestone\u003c/th\u003e\n          \u003cth\u003eEvidence status\u003c/th\u003e\n          \u003cth\u003eRepresentative sources\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAutomated prompt or workflow optimization\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eAFlow; TextGrad [9][22]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAutomated agent architecture search\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eADAS [8]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents modify their own code and improve benchmark performance\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eDarwin Gödel Machine [7]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents improve their own tool use, planning, or code-editing mechanisms\u003c/td\u003e\n          \u003ctd\u003eEmerging\u003c/td\u003e\n          \u003ctd\u003eDGM, ADAS, AFlow, TextGrad [7][8][9][22]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eSelf-improving scaffolds generate sustained open-ended capability growth\u003c/td\u003e\n          \u003ctd\u003eOpen\u003c/td\u003e\n          \u003ctd\u003eDGM demonstrates an early open-ended agent improvement loop, but sustained general capability growth remains unproven [7]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003chr\u003e\n\u003ch3 id=\"23-rsi-on-infrastructure-level-ai-rd\"\u003e2.3 RSI on Infrastructure-level AI R\u0026amp;D\u003c/h3\u003e\n\u003cp\u003eThis is the most consequential paradigm for frontier risk.\u003c/p\u003e\n\u003cp\u003eInfrastructure-level self-improvement occurs when AI systems enter the AI development pipeline itself: coding, experiment design, data processing, evaluation, training infrastructure, algorithm discovery, paper replication, and successor-model development.\u003c/p\u003e\n\u003cp\u003eRE-Bench evaluates frontier model agents on realistic ML research engineering tasks and compares them with human experts. The best AI agents outperform human experts under short time budgets, while humans still improve more with longer time budgets [10]. PaperBench evaluates whether agents can replicate state-of-the-art AI research papers, including understanding contributions, building codebases, and executing experiments [11]. Nature\u0026rsquo;s \u003cem\u003eAI Scientist\u003c/em\u003e paper presents an end-to-end pipeline that creates research ideas, writes code, runs experiments, analyzes data, writes manuscripts, and performs automated peer review [12]. AlphaEvolve shows that LLM-powered coding agents can discover and improve algorithms, including optimizations used in data centers, chip design, and AI training processes [28].\u003c/p\u003e\n\u003cp\u003eThese systems are still limited. They do not show that full autonomous AI research has arrived. They do show that AI R\u0026amp;D automation is becoming measurable.\u003c/p\u003e\n\u003cp\u003eThe risk at this layer is \u003cstrong\u003ecapability acceleration\u003c/strong\u003e. If AI systems help produce stronger AI systems, the rate of frontier AI progress could increase. If that rate exceeds the capacity of safety evaluation, cybersecurity, institutional review, and governance, the result is an oversight gap.\u003c/p\u003e\n\u003cp\u003eA simple milestone ladder:\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eMilestone\u003c/th\u003e\n          \u003cth\u003eEvidence status\u003c/th\u003e\n          \u003cth\u003eRepresentative sources\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI assists coding, debugging, and experiment analysis\u003c/td\u003e\n          \u003ctd\u003eObserved\u003c/td\u003e\n          \u003ctd\u003eCSET workshop report; AIRDA measurement agenda [1][2]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI agents complete ML research engineering subtasks\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eRE-Bench [10]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI agents partially replicate AI research papers\u003c/td\u003e\n          \u003ctd\u003eDemonstrated but limited\u003c/td\u003e\n          \u003ctd\u003ePaperBench [11]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI systems run end-to-end research loops in controlled settings\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eAI Scientist [12]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI systems discover or optimize algorithms used in computational infrastructure\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eAlphaEvolve [28]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI significantly accelerates real frontier AI R\u0026amp;D\u003c/td\u003e\n          \u003ctd\u003eOpen\u003c/td\u003e\n          \u003ctd\u003eAIRDA measurement paper identifies this as a key object of measurement [2]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI-driven AI development shortens model generation cycles\u003c/td\u003e\n          \u003ctd\u003eOpen\u003c/td\u003e\n          \u003ctd\u003eOpenAI and Anthropic include related thresholds in frontier risk frameworks [14][15]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI R\u0026amp;D speed exceeds reliable human oversight capacity\u003c/td\u003e\n          \u003ctd\u003eCritical threshold\u003c/td\u003e\n          \u003ctd\u003eAIRDA oversight framing; frontier safety frameworks [2][14][15][16]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003chr\u003e\n\u003ch3 id=\"24-rsi-modules-of-runtime-agents\"\u003e2.4 RSI Modules of Runtime agents\u003c/h3\u003e\n\u003cp\u003eSome self-improvement occurs during deployment.\u003c/p\u003e\n\u003cp\u003eRuntime self-improvement modules allow agents to improve through long-term memory, reflection, skill libraries, procedural knowledge, experience replay, user models, and task models.\u003c/p\u003e\n\u003cp\u003eA deployed agent with memory and reusable skills is a moving target. The system evaluated at launch may differ from the system users interact with after weeks of accumulated experience. The model weights may stay the same, while the effective behavior changes.\u003c/p\u003e\n\u003cp\u003eThis paradigm is especially relevant for personal assistants, research agents, coding agents, enterprise agents, and autonomous workflows. The safety concern is persistent adaptation: memory and skills can preserve useful knowledge, but they can also preserve bad strategies, biased assumptions, unsafe shortcuts, or user-specific manipulation patterns.\u003c/p\u003e\n\u003cp\u003eA simple milestone ladder:\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eMilestone\u003c/th\u003e\n          \u003cth\u003eEvidence status\u003c/th\u003e\n          \u003cth\u003eRepresentative sources\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents maintain long-term memory\u003c/td\u003e\n          \u003ctd\u003eObserved / increasingly productized\u003c/td\u003e\n          \u003ctd\u003eHermes Agent; broader agent-memory ecosystem [26]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents form reusable skills from experience\u003c/td\u003e\n          \u003ctd\u003eDemonstrated in prototypes\u003c/td\u003e\n          \u003ctd\u003eVoyager; Hermes Agent [25][26]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents improve through reflection\u003c/td\u003e\n          \u003ctd\u003eDemonstrated in prototypes\u003c/td\u003e\n          \u003ctd\u003eSAGE [23]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents maintain procedural memory across tasks\u003c/td\u003e\n          \u003ctd\u003eEmerging\u003c/td\u003e\n          \u003ctd\u003eReasoningBank; Voyager [24][25]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents distill successful and failed experiences into reusable reasoning strategies\u003c/td\u003e\n          \u003ctd\u003eDemonstrated in research prototype\u003c/td\u003e\n          \u003ctd\u003eReasoningBank [24]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eRuntime adaptation causes safety-relevant behavioral drift\u003c/td\u003e\n          \u003ctd\u003eUnderexplored\u003c/td\u003e\n          \u003ctd\u003eExisting memory work motivates the question but does not settle the risk [23][24][25][26]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eRuntime memory and skills enable strategic policy evasion\u003c/td\u003e\n          \u003ctd\u003eOpen\u003c/td\u003e\n          \u003ctd\u003eRelevant to agent safety and monitoring, but direct evidence remains limited [23][24][27]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003chr\u003e\n\u003ch3 id=\"25-rsi-of-multi-agent-population\"\u003e2.5 RSI of Multi-agent population\u003c/h3\u003e\n\u003cp\u003eRSI may also emerge through groups of agents rather than a single agent.\u003c/p\u003e\n\u003cp\u003eMulti-agent self-improvement includes debate, critique, specialization, peer review, population-based search, evolutionary selection, and AI organizations that divide research tasks among specialized agents.\u003c/p\u003e\n\u003cp\u003eThis matters because real AI R\u0026amp;D is organizational. It involves idea generation, coding, experiment execution, review, debugging, evaluation, and strategic decision-making. A future AI R\u0026amp;D system may look less like a solitary model improving itself and more like a synthetic research team.\u003c/p\u003e\n\u003cp\u003eThe AI Scientist includes automated peer review [12]. Darwin Gödel Machine maintains an archive of generated coding agents and explores multiple improvement paths [7]. ADAS searches over agentic system designs [8]. These are early signals of population-level or organization-level feedback loops.\u003c/p\u003e\n\u003cp\u003eThe risk is \u003cstrong\u003edistributed oversight failure\u003c/strong\u003e. When many agents generate, review, and select each other\u0026rsquo;s outputs, errors can become harder to attribute. Model-generated review can create evaluation monocultures. Multi-agent systems can also scale R\u0026amp;D speed through parallelism.\u003c/p\u003e\n\u003cp\u003eA simple milestone ladder:\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eMilestone\u003c/th\u003e\n          \u003cth\u003eEvidence status\u003c/th\u003e\n          \u003cth\u003eRepresentative sources\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI generation plus AI critique\u003c/td\u003e\n          \u003ctd\u003eObserved\u003c/td\u003e\n          \u003ctd\u003eAI Scientist; ADAS-style meta-agent evaluation [8][12]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAutomated peer review\u003c/td\u003e\n          \u003ctd\u003eDemonstrated in prototypes\u003c/td\u003e\n          \u003ctd\u003eAI Scientist [12]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eMulti-agent task decomposition\u003c/td\u003e\n          \u003ctd\u003eObserved / common in agent systems\u003c/td\u003e\n          \u003ctd\u003eADAS and broader agentic-system literature [8]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eArchive or population-based agent evolution\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eDarwin Gödel Machine [7]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI organizations complete substantial AI R\u0026amp;D workflows\u003c/td\u003e\n          \u003ctd\u003eOpen\u003c/td\u003e\n          \u003ctd\u003eAI Scientist and RE-Bench indicate components, but full organizational automation remains open [10][12]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eMulti-agent self-improvement produces hard-to-predict collective behavior\u003c/td\u003e\n          \u003ctd\u003eOpen\u003c/td\u003e\n          \u003ctd\u003eMotivated by multi-agent and population-level design loops [7][8][12]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003chr\u003e\n\u003ch3 id=\"26-rsi-on-environment-and-resource\"\u003e2.6 RSI on Environment and Resource\u003c/h3\u003e\n\u003cp\u003eFinally, an AI system can improve its future capabilities by changing its environment.\u003c/p\u003e\n\u003cp\u003eThis includes acquiring tools, data, compute, permissions, API access, collaborators, deployment footholds, persistence mechanisms, and copies of itself.\u003c/p\u003e\n\u003cp\u003eThis layer connects RSI to autonomy and control. A system can become more capable through better external resources, even without becoming cognitively smarter. Tool acquisition, data acquisition, environment setup, persistence, and self-replication can all increase what the system can do next.\u003c/p\u003e\n\u003cp\u003eThe SAIF brief \u003cem\u003eBare Minimum Mitigations for Autonomous AI Development\u003c/em\u003e argues that if AI agents significantly automate or accelerate AI development, developers need minimum safeguards around training, testing, assurance, access, and human approval [13]. The UK AI Security Institute\u0026rsquo;s \u003cem\u003eFrontier AI Trends Report\u003c/em\u003e reports rising success rates on controlled self-replication evaluations and discusses sandbagging as a control-relevant capability [29]. The International AI Safety Report discusses autonomous replication and self-defense as capabilities that could make systems harder to contain [30]. METR\u0026rsquo;s Frontier Risk Report examines misalignment risks from AI agents used inside frontier AI developers [27].\u003c/p\u003e\n\u003cp\u003eA simple milestone ladder:\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eMilestone\u003c/th\u003e\n          \u003cth\u003eEvidence status\u003c/th\u003e\n          \u003cth\u003eRepresentative sources\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents use tools and modify environments\u003c/td\u003e\n          \u003ctd\u003eObserved\u003c/td\u003e\n          \u003ctd\u003eFrontier safety and agent capability evaluations [17][29]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents set up execution environments\u003c/td\u003e\n          \u003ctd\u003eObserved\u003c/td\u003e\n          \u003ctd\u003eCoding-agent and AI R\u0026amp;D automation benchmarks [10][11]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents acquire data or improve toolchains\u003c/td\u003e\n          \u003ctd\u003eEmerging\u003c/td\u003e\n          \u003ctd\u003eAI Scientist, AlphaEvolve, AIRDA framing [2][12][28]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents persist across sessions or environments\u003c/td\u003e\n          \u003ctd\u003eEmerging\u003c/td\u003e\n          \u003ctd\u003eRuntime-memory and long-horizon autonomy work [17][23][24][26]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents replicate under controlled evaluation settings\u003c/td\u003e\n          \u003ctd\u003eDemonstrated in limited evals\u003c/td\u003e\n          \u003ctd\u003eUK AISI trends; self-replication evaluations [29]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgents autonomously acquire compute, access, or deployment footholds\u003c/td\u003e\n          \u003ctd\u003eOpen\u003c/td\u003e\n          \u003ctd\u003eInternational AI Safety Report and SAIF brief motivate this risk pathway [13][30]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eSelf-improvement couples with persistence and loss of control\u003c/td\u003e\n          \u003ctd\u003eCritical threshold\u003c/td\u003e\n          \u003ctd\u003eUK AISI, METR, International AI Safety Report [27][29][30]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003chr\u003e\n\u003ch2 id=\"3-from-self-improvement-loops-to-catastrophic-risk-pathways\"\u003e3. From self-improvement loops to catastrophic risk pathways\u003c/h2\u003e\n\u003cp\u003eThe question is not simply whether AI can improve itself. The policy-relevant question is which self-improvement loops could create catastrophic risk.\u003c/p\u003e\n\u003cp\u003eWe see seven main pathways.\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eRisk pathway\u003c/th\u003e\n          \u003cth\u003eMain RSI paradigms involved\u003c/th\u003e\n          \u003cth\u003eCore concern\u003c/th\u003e\n          \u003cth\u003eRelevant sources\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eCapability acceleration\u003c/td\u003e\n          \u003ctd\u003eInfrastructure-level, scaffold-level, foundation-model-level\u003c/td\u003e\n          \u003ctd\u003eAI accelerates the production of more capable AI systems\u003c/td\u003e\n          \u003ctd\u003eAIRDA, RE-Bench, AI Scientist, AlphaEvolve, frontier frameworks [2][10][12][14][15][16][28]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eOversight gap\u003c/td\u003e\n          \u003ctd\u003eInfrastructure-level, multi-agent, runtime modules\u003c/td\u003e\n          \u003ctd\u003eHuman evaluation capacity fails to keep up with AI-generated R\u0026amp;D output\u003c/td\u003e\n          \u003ctd\u003eAIRDA, CSET, METR, internal deployment disclosure work [1][2][17][27][31]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eSafety evidence corruption\u003c/td\u003e\n          \u003ctd\u003eInfrastructure-level, foundation-model-level\u003c/td\u003e\n          \u003ctd\u003eAI helps generate, select, or manipulate safety evidence\u003c/td\u003e\n          \u003ctd\u003eAnthropic sandbagging, model-generated judges, frontier safety frameworks [14][18][19]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eInternal deployment risk\u003c/td\u003e\n          \u003ctd\u003eInfrastructure-level, environment-level\u003c/td\u003e\n          \u003ctd\u003eAI R\u0026amp;D agents act inside frontier labs with high-stakes access\u003c/td\u003e\n          \u003ctd\u003eMETR Frontier Risk Report, internal disclosure paper, frontier policies [15][16][27][31]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAlignment drift\u003c/td\u003e\n          \u003ctd\u003eFoundation-model-level, runtime modules\u003c/td\u003e\n          \u003ctd\u003eRecursive optimization changes behavior in hidden ways\u003c/td\u003e\n          \u003ctd\u003eModel collapse, emergent misalignment, subliminal learning [4][5][6]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eProliferation\u003c/td\u003e\n          \u003ctd\u003eInfrastructure-level, environment-level\u003c/td\u003e\n          \u003ctd\u003eMore actors can develop high-capability systems\u003c/td\u003e\n          \u003ctd\u003eSAIF brief, International AI Safety Report, AIRDA [2][13][30]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eLoss of control\u003c/td\u003e\n          \u003ctd\u003eEnvironment-level, runtime modules, scaffold-level\u003c/td\u003e\n          \u003ctd\u003eAgents gain persistence, resources, autonomy, or replication capabilities\u003c/td\u003e\n          \u003ctd\u003eUK AISI trends, METR, International AI Safety Report [27][29][30]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch3 id=\"31-capability-acceleration\"\u003e3.1 Capability acceleration\u003c/h3\u003e\n\u003cp\u003eThe first pathway is speed. AI systems may accelerate the production of more capable AI systems.\u003c/p\u003e\n\u003cp\u003eThis can happen through AI-assisted coding, experiment automation, paper replication, training pipeline optimization, algorithm discovery, and agent scaffold improvement [2][10][11][12][28].\u003c/p\u003e\n\u003cp\u003eThe milestone to watch is not \u0026ldquo;full RSI.\u0026rdquo; It is the point where AI systems measurably shorten the development cycle of stronger models.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eKey milestone:\u003c/strong\u003e AI-driven AI development produces frontier capability progress faster than human institutions can evaluate and govern it.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003chr\u003e\n\u003ch3 id=\"32-oversight-gap\"\u003e3.2 Oversight gap\u003c/h3\u003e\n\u003cp\u003eThe second pathway is oversight failure.\u003c/p\u003e\n\u003cp\u003eAI may generate code, experiments, evaluations, model changes, and research artifacts faster than humans can reliably inspect. This could create a widening gap between the amount of AI-generated R\u0026amp;D output and the amount that human reviewers can understand, verify, and secure.\u003c/p\u003e\n\u003cp\u003eGovAI and Oxford explicitly identify this kind of measurement challenge: current benchmarks may not capture whether AI R\u0026amp;D automation accelerates capabilities more than safety progress or whether human oversight can keep pace [2].\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eKey milestone:\u003c/strong\u003e AI-generated R\u0026amp;D output exceeds reliable human review capacity.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003chr\u003e\n\u003ch3 id=\"33-safety-evidence-corruption\"\u003e3.3 Safety evidence corruption\u003c/h3\u003e\n\u003cp\u003eThe third pathway is corruption of the evidence used to judge safety.\u003c/p\u003e\n\u003cp\u003eIf AI systems help write evaluations, analyze experiments, generate safety reports, or monitor other AI systems, the safety evaluation pipeline itself becomes a risk surface.\u003c/p\u003e\n\u003cp\u003eAnthropic\u0026rsquo;s work on automated researchers sandbagging shows that current models can, when prompted, subtly sandbag ML experiments and research decisions in ways that zero-shot prompted monitors fail to detect, while the authors emphasize that today\u0026rsquo;s cases would likely still be noticed by humans [18]. Self-rewarding and LLM-as-judge systems make the general issue broader: models can increasingly participate in reward and judgment processes used for training and evaluation [19].\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eKey milestone:\u003c/strong\u003e AI systems meaningfully influence the evidence used to certify the safety of future AI systems.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003chr\u003e\n\u003ch3 id=\"34-internal-deployment-risk\"\u003e3.4 Internal deployment risk\u003c/h3\u003e\n\u003cp\u003eThe fourth pathway is internal deployment.\u003c/p\u003e\n\u003cp\u003eThe most important AI R\u0026amp;D agents may first appear inside frontier AI labs. They may access code repositories, experiment logs, model evaluations, training infrastructure, deployment systems, and security-sensitive workflows.\u003c/p\u003e\n\u003cp\u003eThis matters because public chatbot evaluations may miss internal risks. A model that appears safe as a user-facing assistant could behave very differently as an internal R\u0026amp;D agent with tools, permissions, memory, and access to sensitive systems.\u003c/p\u003e\n\u003cp\u003eMETR\u0026rsquo;s Frontier Risk Report is directly relevant here because it studies misalignment risks from AI agents used inside frontier AI developers [27]. A separate disclosure proposal argues that frontier developers should provide evidence about internal deployments as these systems grow in scope and capability [31].\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eKey milestone:\u003c/strong\u003e AI R\u0026amp;D agents receive high-stakes access to model-development pipelines before external evaluators can assess their behavior.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003chr\u003e\n\u003ch3 id=\"35-alignment-drift-through-recursive-optimization\"\u003e3.5 Alignment drift through recursive optimization\u003c/h3\u003e\n\u003cp\u003eThe fifth pathway is alignment drift.\u003c/p\u003e\n\u003cp\u003eRecursive optimization can improve capabilities while also changing behavior in hidden ways. Model collapse shows that recursive generated-data loops can degrade generative models [4]. Emergent misalignment shows that narrow fine-tuning can induce broad behavioral changes [5]. Subliminal learning shows that behavioral traits can transmit through semantically unrelated data [6].\u003c/p\u003e\n\u003cp\u003eTogether, these results suggest that recursive training and distillation loops deserve careful safety analysis.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eKey milestone:\u003c/strong\u003e successor models inherit hidden behavioral defects from AI-generated data, rewards, critiques, or distillation pipelines.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003chr\u003e\n\u003ch3 id=\"36-proliferation-and-autonomous-development\"\u003e3.6 Proliferation and autonomous development\u003c/h3\u003e\n\u003cp\u003eThe sixth pathway is diffusion.\u003c/p\u003e\n\u003cp\u003eIf AI agents reduce the expertise required to conduct AI development, more actors may be able to fine-tune, improve, replicate, or deploy powerful systems. This risk becomes especially serious if stolen models, open-weight models, or smaller labs can use AI R\u0026amp;D agents to close capability gaps.\u003c/p\u003e\n\u003cp\u003eThe SAIF mitigation brief focuses directly on this possibility: autonomous AI development could reduce human oversight, hinder identification of accidents or misuse, and compromise the AI supply chain if safeguards are not in place [13].\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eKey milestone:\u003c/strong\u003e advanced AI development becomes substantially easier for actors outside frontier labs.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003chr\u003e\n\u003ch3 id=\"37-loss-of-control-through-persistence-and-resource-acquisition\"\u003e3.7 Loss of control through persistence and resource acquisition\u003c/h3\u003e\n\u003cp\u003eThe seventh pathway is control loss.\u003c/p\u003e\n\u003cp\u003eRSI becomes more dangerous when it couples with autonomy, tool use, persistence, resource acquisition, and self-replication. Self-replication is not the whole of RSI risk, but it is an important warning signal that self-improvement may be connecting to persistence and control.\u003c/p\u003e\n\u003cp\u003eUK AISI reports rising success rates on controlled self-replication evaluations [29]. The International AI Safety Report explains why autonomous replication or effective self-defense could make AI systems difficult to contain [30]. METR\u0026rsquo;s Frontier Risk Report examines related risks in the context of internal AI agents at frontier developers [27].\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eKey milestone:\u003c/strong\u003e self-improving agents can preserve, copy, or expand their operation across environments despite external constraints.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003chr\u003e\n\u003ch2 id=\"4-what-current-evidence-already-shows\"\u003e4. What current evidence already shows\u003c/h2\u003e\n\u003cp\u003eWe should be careful. Current evidence does not show that fully autonomous recursive self-improvement has arrived.\u003c/p\u003e\n\u003cp\u003eIt does show that several enabling feedback loops are already technically real.\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eCapability\u003c/th\u003e\n          \u003cth\u003eEvidence status\u003c/th\u003e\n          \u003cth\u003eRepresentative materials\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eModels learn from their own generated reasoning\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eSTaR; Quiet-STaR [3][20]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eModels provide reward or judgment signals for training\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eSelf-Rewarding Language Models [19]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eModels improve through self-play/self-training\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eSPIN [21]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eRecursive generated data changes model behavior\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eModel collapse [4]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eNarrow fine-tuning can induce broad behavioral change\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eEmergent misalignment [5]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eBehavioral traits can transmit through generated data\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eSubliminal learning [6]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgent workflows can be automatically optimized\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eAFlow; TextGrad [9][22]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgent architectures can be automatically searched\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eADAS [8]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAgent code can self-modify and improve benchmark performance\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eDarwin Gödel Machine [7]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI agents can perform ML research engineering tasks\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eRE-Bench [10]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI agents can partially replicate AI research papers\u003c/td\u003e\n          \u003ctd\u003eDemonstrated but limited\u003c/td\u003e\n          \u003ctd\u003ePaperBench [11]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI systems can run end-to-end research loops in controlled settings\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eAI Scientist [12]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI systems can discover or optimize algorithms used in real infrastructure\u003c/td\u003e\n          \u003ctd\u003eDemonstrated\u003c/td\u003e\n          \u003ctd\u003eAlphaEvolve [28]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eRuntime agents can accumulate memory or reusable skills\u003c/td\u003e\n          \u003ctd\u003eEmerging / demonstrated in prototypes\u003c/td\u003e\n          \u003ctd\u003eSAGE; ReasoningBank; Voyager; Hermes Agent [23][24][25][26]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eControlled self-replication evaluations show rising success rates\u003c/td\u003e\n          \u003ctd\u003eDemonstrated in limited evals\u003c/td\u003e\n          \u003ctd\u003eUK AISI Frontier AI Trends Report [29]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI can significantly accelerate real frontier AI R\u0026amp;D\u003c/td\u003e\n          \u003ctd\u003eOpen\u003c/td\u003e\n          \u003ctd\u003eMeasuring AIRDA; frontier safety frameworks [2][14][15][16]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI can autonomously produce a stronger successor frontier model\u003c/td\u003e\n          \u003ctd\u003eOpen\u003c/td\u003e\n          \u003ctd\u003eFrontier safety frameworks [14][15][16]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eThe important point is the pattern: AI is moving from being a product of AI R\u0026amp;D to becoming a participant in AI R\u0026amp;D.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"5-what-remains-unknown\"\u003e5. What remains unknown\u003c/h2\u003e\n\u003cp\u003eSeveral open questions should guide future research.\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eOpen question\u003c/th\u003e\n          \u003cth\u003eWhy it matters\u003c/th\u003e\n          \u003cth\u003eRelevant sources\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eAcceleration:\u003c/strong\u003e How much can AI R\u0026amp;D automation speed up real frontier AI development?\u003c/td\u003e\n          \u003ctd\u003eDetermines whether RSI is mostly an efficiency gain or a destabilizing capability accelerator\u003c/td\u003e\n          \u003ctd\u003eAIRDA, CSET, frontier frameworks [1][2][14][15][16]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eBottlenecks:\u003c/strong\u003e Are the main bottlenecks coding, experiments, ideas, data, compute, coordination, or judgment?\u003c/td\u003e\n          \u003ctd\u003eRSI risk depends on which parts of AI R\u0026amp;D are actually rate-limiting\u003c/td\u003e\n          \u003ctd\u003eCSET, AIRDA, RE-Bench, PaperBench [1][2][10][11]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eOversight:\u003c/strong\u003e Can humans reliably audit large volumes of AI-generated code, experiments, evaluations, and research claims?\u003c/td\u003e\n          \u003ctd\u003eDefines whether AI-generated R\u0026amp;D creates an oversight gap\u003c/td\u003e\n          \u003ctd\u003eAIRDA, Anthropic sandbagging, internal disclosure work [2][18][31]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eRecursive contribution:\u003c/strong\u003e How much does a current model contribute to the next generation\u0026rsquo;s capability progress?\u003c/td\u003e\n          \u003ctd\u003eTracks whether AI development is becoming self-reinforcing\u003c/td\u003e\n          \u003ctd\u003eOpenAI, Anthropic, Google DeepMind frameworks [14][15][16]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eSafety asymmetry:\u003c/strong\u003e Does AI accelerate capability research faster than safety research?\u003c/td\u003e\n          \u003ctd\u003eDetermines whether AI-driven AI development widens or narrows the safety gap\u003c/td\u003e\n          \u003ctd\u003eAIRDA; Science extreme-risk framing [2][32]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eInternal deployment:\u003c/strong\u003e What permissions and infrastructure access do internal AI R\u0026amp;D agents have?\u003c/td\u003e\n          \u003ctd\u003eInternal agent risk may be invisible from public model evaluations\u003c/td\u003e\n          \u003ctd\u003eMETR Frontier Risk Report; internal deployment disclosure paper [27][31]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eAlignment drift:\u003c/strong\u003e Do recursive training, distillation, and post-training loops systematically amplify hidden misalignment?\u003c/td\u003e\n          \u003ctd\u003eConnects RSI to model behavior across generations\u003c/td\u003e\n          \u003ctd\u003eModel collapse; emergent misalignment; subliminal learning [4][5][6]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eControl:\u003c/strong\u003e When does self-improvement combine with persistence, replication, and resource acquisition?\u003c/td\u003e\n          \u003ctd\u003eMarks the transition from capability acceleration to control-loss risk\u003c/td\u003e\n          \u003ctd\u003eUK AISI, International AI Safety Report, METR [27][29][30]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eThese questions are more useful than asking whether RSI has \u0026ldquo;arrived.\u0026rdquo; The stronger research agenda is to measure which feedback loops are emerging and how fast they are moving.\u003c/p\u003e\n\u003chr\u003e\n\u003ch2 id=\"6-what-should-be-measured-next\"\u003e6. What should be measured next?\u003c/h2\u003e\n\u003cp\u003eA practical RSI measurement agenda should track both technical capability and governance risk.\u003c/p\u003e\n\u003ch3 id=\"technical-metrics\"\u003eTechnical metrics\u003c/h3\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eMetric\u003c/th\u003e\n          \u003cth\u003eQuestion\u003c/th\u003e\n          \u003cth\u003eCandidate sources or measurement anchors\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAI R\u0026amp;D task autonomy\u003c/td\u003e\n          \u003ctd\u003eWhat length and complexity of AI R\u0026amp;D tasks can AI systems complete?\u003c/td\u003e\n          \u003ctd\u003eRE-Bench, PaperBench, METR time horizons [10][11][17]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eResearch-loop closure\u003c/td\u003e\n          \u003ctd\u003eCan an AI system move from idea to experiment to revision?\u003c/td\u003e\n          \u003ctd\u003eAI Scientist; CSET workshop report [1][12]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eSuccessor contribution\u003c/td\u003e\n          \u003ctd\u003eHow much does AI contribute to the next generation of models?\u003c/td\u003e\n          \u003ctd\u003eOpenAI Preparedness Framework; Anthropic RSP; DeepMind FSF [14][15][16]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eScaffold improvement rate\u003c/td\u003e\n          \u003ctd\u003eHow fast can agents improve their own prompts, tools, workflows, or code?\u003c/td\u003e\n          \u003ctd\u003eDGM, ADAS, AFlow, TextGrad [7][8][9][22]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eRuntime skill accumulation\u003c/td\u003e\n          \u003ctd\u003eCan deployed agents accumulate transferable skills across tasks?\u003c/td\u003e\n          \u003ctd\u003eSAGE, ReasoningBank, Voyager, Hermes Agent [23][24][25][26]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAlgorithmic infrastructure optimization\u003c/td\u003e\n          \u003ctd\u003eCan AI improve algorithms or infrastructure used for AI development itself?\u003c/td\u003e\n          \u003ctd\u003eAlphaEvolve [28]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch3 id=\"risk-metrics\"\u003eRisk metrics\u003c/h3\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eMetric\u003c/th\u003e\n          \u003cth\u003eQuestion\u003c/th\u003e\n          \u003cth\u003eCandidate sources or measurement anchors\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eOversight burden\u003c/td\u003e\n          \u003ctd\u003eHow much AI-generated R\u0026amp;D output must humans review?\u003c/td\u003e\n          \u003ctd\u003eAIRDA; internal deployment disclosure paper [2][31]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eOversight reliability\u003c/td\u003e\n          \u003ctd\u003eCan humans detect subtle errors, sandbagging, or sabotage?\u003c/td\u003e\n          \u003ctd\u003eAnthropic sandbagging; METR Frontier Risk Report [18][27]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eSafety evidence dependence\u003c/td\u003e\n          \u003ctd\u003eHow much safety evidence is generated or processed by AI systems?\u003c/td\u003e\n          \u003ctd\u003eSelf-Rewarding LMs; Anthropic sandbagging; frontier safety frameworks [14][18][19]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eInternal access level\u003c/td\u003e\n          \u003ctd\u003eWhat sensitive assets can AI R\u0026amp;D agents access?\u003c/td\u003e\n          \u003ctd\u003eMETR Frontier Risk Report; internal deployment disclosure paper [27][31]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eCapability/safety acceleration ratio\u003c/td\u003e\n          \u003ctd\u003eDoes AI accelerate capability work more than safety work?\u003c/td\u003e\n          \u003ctd\u003eAIRDA; Science extreme-risk framing [2][32]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eProliferation potential\u003c/td\u003e\n          \u003ctd\u003eDoes AI reduce the expertise or cost needed for advanced AI development?\u003c/td\u003e\n          \u003ctd\u003eSAIF brief; International AI Safety Report [13][30]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003ePersistence and replication risk\u003c/td\u003e\n          \u003ctd\u003eCan agents copy, preserve, or expand their operation across environments?\u003c/td\u003e\n          \u003ctd\u003eUK AISI; International AI Safety Report [29][30]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cblockquote\u003e\n\u003cp\u003eMeasuring RSI requires measuring feedback, not only capability: how much AI systems contribute to the next cycle of AI development, and how much harder that cycle becomes to oversee.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003chr\u003e\n\u003ch2 id=\"7-governance-implications\"\u003e7. Governance implications\u003c/h2\u003e\n\u003cp\u003eGovernance should start before fully autonomous AI researchers exist.\u003c/p\u003e\n\u003cp\u003eA minimum agenda could include:\u003c/p\u003e\n\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003eGovernance proposal\u003c/th\u003e\n          \u003cth\u003eRationale\u003c/th\u003e\n          \u003cth\u003eRelevant sources\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eTreat AI R\u0026amp;D automation as a frontier risk category\u003c/td\u003e\n          \u003ctd\u003eAI systems that accelerate AI development can change the risk landscape even before full RSI\u003c/td\u003e\n          \u003ctd\u003eOpenAI, Anthropic, Google DeepMind, METR common elements [14][15][16][33]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eDisclose internal AI R\u0026amp;D use\u003c/td\u003e\n          \u003ctd\u003ePublic evaluations may miss risks from internal coding, training, evaluation, and deployment agents\u003c/td\u003e\n          \u003ctd\u003eInternal deployment disclosure paper; METR Frontier Risk Report [27][31]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eEvaluate AI R\u0026amp;D capability before deployment\u003c/td\u003e\n          \u003ctd\u003eResearch engineering, paper replication, scaffold improvement, and long-horizon tasks are core RSI precursors\u003c/td\u003e\n          \u003ctd\u003eRE-Bench, PaperBench, METR time horizons [10][11][17]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eAudit internal AI R\u0026amp;D agents\u003c/td\u003e\n          \u003ctd\u003eAgents with access to code, evals, model weights, or deployment systems create high-stakes internal risk\u003c/td\u003e\n          \u003ctd\u003eMETR Frontier Risk Report; Anthropic RSP; DeepMind FSF [15][16][27]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eSeparate capability automation from safety evidence generation\u003c/td\u003e\n          \u003ctd\u003eAI systems that accelerate capability work should not freely control safety evidence\u003c/td\u003e\n          \u003ctd\u003eAnthropic sandbagging; Self-Rewarding LMs; AIRDA [2][18][19]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eMonitor RSI milestones\u003c/td\u003e\n          \u003ctd\u003eRSI should be tracked as a set of measurable feedback loops rather than a single speculative event\u003c/td\u003e\n          \u003ctd\u003eAIRDA, CSET, OpenAI, Anthropic, DeepMind [1][2][14][15][16]\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003eMaintain minimum safeguards for autonomous AI development\u003c/td\u003e\n          \u003ctd\u003eTraining, testing, assurance, access controls, and human approval matter before fully autonomous development arrives\u003c/td\u003e\n          \u003ctd\u003eSAIF mitigation brief [13]\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003chr\u003e\n\u003ch2 id=\"8-conclusion-rsi-as-an-empirical-risk-framework\"\u003e8. Conclusion: RSI as an empirical risk framework\u003c/h2\u003e\n\u003cp\u003eRecursive self-improvement should be studied as an empirical risk framework.\u003c/p\u003e\n\u003cp\u003eThe central question is no longer whether a model can suddenly rewrite itself into a superintelligence. The more concrete question is which self-improvement feedback loops are emerging today, how quickly they are improving, and when they could combine into catastrophic risk pathways.\u003c/p\u003e\n\u003cp\u003eSome of these loops are already visible. Foundation models can learn from their own generated reasoning. Agent scaffolds can be optimized. Coding agents can modify their own code. AI agents can perform parts of ML research engineering. AI systems can generate data, judgments, evaluations, and research artifacts. Deployed agents can accumulate memory and skills.\u003c/p\u003e\n\u003cp\u003eFull recursive self-improvement remains an open question. The enabling pieces are becoming measurable.\u003c/p\u003e\n\u003cp\u003eThat is enough reason to start building a rigorous RSI risk science now.\u003c/p\u003e\n\u003chr\u003e\n\u003ch1 id=\"references\"\u003eReferences\u003c/h1\u003e\n\u003cp\u003e[1] \u003ca href=\"https://cset.georgetown.edu/publication/when-ai-builds-ai/\"\u003eCSET — \u003cem\u003eWhen AI Builds AI: Findings From a Workshop on Automation of AI R\u0026amp;D\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[2] \u003ca href=\"https://arxiv.org/abs/2603.03992\"\u003eChan et al. — \u003cem\u003eMeasuring AI R\u0026amp;D Automation\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[3] \u003ca href=\"https://arxiv.org/abs/2203.14465\"\u003eZelikman et al. — \u003cem\u003eSTaR: Bootstrapping Reasoning With Reasoning\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[4] \u003ca href=\"https://www.nature.com/articles/s41586-024-07566-y\"\u003eShumailov et al. — \u003cem\u003eAI models collapse when trained on recursively generated data\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[5] \u003ca href=\"https://www.nature.com/articles/s41586-025-09937-5\"\u003eBetley et al. — \u003cem\u003eTraining large language models on narrow tasks can lead to broad misalignment\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[6] \u003ca href=\"https://www.nature.com/articles/s41586-026-10319-8\"\u003eCloud et al. — \u003cem\u003eLanguage models transmit behavioural traits through hidden signals in data\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[7] \u003ca href=\"https://arxiv.org/abs/2505.22954\"\u003eZhang et al. — \u003cem\u003eDarwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[8] \u003ca href=\"https://openreview.net/forum?id=t9U3LW7JVX\"\u003eHu et al. — \u003cem\u003eAutomated Design of Agentic Systems\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[9] \u003ca href=\"https://openreview.net/forum?id=z5uVAKwmjf\"\u003eAFlow — \u003cem\u003eAutomating Agentic Workflow Generation\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[10] \u003ca href=\"https://arxiv.org/abs/2411.15114\"\u003eWijk et al. — \u003cem\u003eRE-Bench: Evaluating Frontier AI R\u0026amp;D Capabilities of Language Model Agents against Human Experts\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[11] \u003ca href=\"https://arxiv.org/abs/2504.01848\"\u003eOpenAI — \u003cem\u003ePaperBench: Evaluating AI\u0026rsquo;s Ability to Replicate AI Research\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[12] \u003ca href=\"https://www.nature.com/articles/s41586-026-10265-5\"\u003eLu et al. — \u003cem\u003eTowards end-to-end automation of AI research\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[13] \u003ca href=\"https://saif.org/research/bare-minimum-mitigations-for-autonomous-ai-development/\"\u003eSAIF — \u003cem\u003eBare Minimum Mitigations for Autonomous AI Development\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[14] \u003ca href=\"https://cdn.openai.com/pdf/18a02b5d-6b67-4cec-ab64-68cdfbddebcd/preparedness-framework-v2.pdf\"\u003eOpenAI — \u003cem\u003ePreparedness Framework v2\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[15] \u003ca href=\"https://www.anthropic.com/news/responsible-scaling-policy-v3\"\u003eAnthropic — \u003cem\u003eResponsible Scaling Policy v3.0\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[16] \u003ca href=\"https://deepmind.google/blog/updating-the-frontier-safety-framework/\"\u003eGoogle DeepMind — \u003cem\u003eUpdating the Frontier Safety Framework\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[17] \u003ca href=\"https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/\"\u003eMETR — \u003cem\u003eMeasuring AI Ability to Complete Long Tasks\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[18] \u003ca href=\"https://alignment.anthropic.com/2025/automated-researchers-sandbag/\"\u003eAnthropic Alignment Science — \u003cem\u003eAutomated Researchers Can Subtly Sandbag\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[19] \u003ca href=\"https://arxiv.org/abs/2401.10020\"\u003eYuan et al. — \u003cem\u003eSelf-Rewarding Language Models\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[20] \u003ca href=\"https://arxiv.org/abs/2403.09629\"\u003eZelikman et al. — \u003cem\u003eQuiet-STaR: Language Models Can Teach Themselves to Think Before Speaking\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[21] \u003ca href=\"https://arxiv.org/abs/2401.01335\"\u003eChen et al. — \u003cem\u003eSelf-Play Fine-Tuning Converts Weak Language Models to Strong Language Models\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[22] \u003ca href=\"https://arxiv.org/abs/2406.07496\"\u003eYuksekgonul et al. — \u003cem\u003eTextGrad: Automatic \u0026ldquo;Differentiation\u0026rdquo; via Text\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[23] \u003ca href=\"https://arxiv.org/abs/2409.00872\"\u003eLiang et al. — \u003cem\u003eSelf-evolving Agents with Reflective and Memory-augmented Abilities\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[24] \u003ca href=\"https://arxiv.org/abs/2509.25140\"\u003eOuyang et al. — \u003cem\u003eReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[25] \u003ca href=\"https://arxiv.org/abs/2305.16291\"\u003eWang et al. — \u003cem\u003eVoyager: An Open-Ended Embodied Agent with Large Language Models\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[26] \u003ca href=\"https://github.com/nousresearch/hermes-agent\"\u003eNous Research — \u003cem\u003eHermes Agent GitHub repository\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[27] \u003ca href=\"https://metr.org/blog/2026-05-19-frontier-risk-report/\"\u003eMETR — \u003cem\u003eFrontier Risk Report: February to March 2026\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[28] \u003ca href=\"https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/\"\u003eGoogle DeepMind — \u003cem\u003eAlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[29] \u003ca href=\"https://www.aisi.gov.uk/frontier-ai-trends-report\"\u003eUK AI Security Institute — \u003cem\u003eFrontier AI Trends Report\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[30] \u003ca href=\"https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026\"\u003eInternational AI Safety Report 2026\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[31] \u003ca href=\"https://arxiv.org/html/2604.23065v1\"\u003eChan et al. — \u003cem\u003eWhat Should Frontier AI Developers Disclose About Internal Deployments?\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[32] \u003ca href=\"https://www.science.org/doi/10.1126/science.adn0117\"\u003eBengio et al. — \u003cem\u003eManaging extreme AI risks amid rapid progress\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n\u003cp\u003e[33] \u003ca href=\"https://metr.org/common-elements\"\u003eMETR — \u003cem\u003eCommon Elements of Frontier AI Safety Policies\u003c/em\u003e\u003c/a\u003e\u003c/p\u003e\n"
        },
        {
            "title": "A Nostalgia Reading List for Beginners on AI Security",
            "date_published": "2025-02-06T00:00:00Z",
            "date_modified": "2025-02-06T00:00:00Z",
            "id": "https://ravensanstete.github.io/en/blog/ai-security-reading-list/",
            "url": "https://ravensanstete.github.io/en/blog/ai-security-reading-list/",
            "content_html": "\u003ch2 id=\"1-adversarial-attacks\"\u003e1. Adversarial Attacks\u003c/h2\u003e\n\u003ch3 id=\"11-adversarial-examples-ae--defenses\"\u003e1.1. Adversarial Examples (AE) \u0026amp; Defenses\u003c/h3\u003e\n\u003ch4 id=\"111-survey\"\u003e1.1.1. Survey\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWild patterns: Ten years after the rise of adversarial machine learning\u003c/strong\u003e — A survey covering AI security research before 2018, focusing primarily on adversarial examples and poisoning attacks.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"112-attack-side\"\u003e1.1.2. Attack Side\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eFGSM\u003c/strong\u003e — the first AE\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ePGD\u003c/strong\u003e — the first iterative AE generation algorithm\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eC\u0026amp;W\u003c/strong\u003e — systematization work\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTextBugger\u003c/strong\u003e — AE attacks on NLP models\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eBlack-box Adversarial Attacks on Commercial Speech Platforms with Minimal Information\u003c/strong\u003e — AE on audio models\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"113-empirical-defense\"\u003e1.1.3. Empirical Defense\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eMagNet: a Two-Pronged Defense against Adversarial Examples\u003c/strong\u003e (CCS'17) — manifold-based, unsupervised approach\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eObfuscated Gradients Give a False Sense of Security\u003c/strong\u003e (ICML'18 Best Paper) — surveys pre-2018 defenses and breaks them; circumventing defenses to adversarial examples\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"114-certified-defense\"\u003e1.1.4. Certified Defense\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eSoK: Certified Robustness for Deep Neural Networks\u003c/strong\u003e — a survey\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCertified Adversarial Robustness via Randomized Smoothing\u003c/strong\u003e (ICML'19) — an early work on randomized smoothing\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAI2: Safety and Robustness Certification of Neural Networks with Abstract Interpretation\u003c/strong\u003e (S\u0026amp;P'18) — deterministic certification based on ideas from program verification\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"12-backdoor-attacks--defenses\"\u003e1.2. Backdoor Attacks \u0026amp; Defenses\u003c/h3\u003e\n\u003ch4 id=\"121-survey\"\u003e1.2.1. Survey\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eTrojanZoo\u003c/strong\u003e — huge engineering efforts with an open-sourced framework\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"122-attack-side\"\u003e1.2.2. Attack Side\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eTrojanNN\u003c/strong\u003e (NDSS'17) — neuron-based injection\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eBadNet\u003c/strong\u003e (IEEE Access) — data-based injection, classic\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eLatent Backdoor\u003c/strong\u003e (CCS'19) — extending the backdoor attack to pretrained encoders, via feature alignment\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eInput-Aware Backdoor\u003c/strong\u003e (NeurIPS'20) — the first dynamic trigger backdoor\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eHidden Trigger Backdoor Attack on NLP Models via Linguistic Style Manipulation\u003c/strong\u003e (Security'22) — the first dynamic backdoor on NLP models \u003cem\u003e(Ours)\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTowards Backdoor Attack on Deep Learning based Time Series Classification\u003c/strong\u003e (ICDE'22) — the first effective backdoor attack on time series models \u003cem\u003e(Ours)\u003c/em\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"123-defense-side\"\u003e1.2.3. Defense Side\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eFine-pruning\u003c/strong\u003e (RAID'18) — pruning and finetuning, based on the hypothesized differences in activation patterns\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSTRIP\u003c/strong\u003e (ACSAC'19) — detection based on the hypothesis that triggered input is resilient to noise\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNeural Cleanse\u003c/strong\u003e (S\u0026amp;P'19) — strong link between backdoor behavior and static trigger pattern\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eABS\u003c/strong\u003e (CCS'19) — neuron-level inspection\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"13-poisoning-attacks\"\u003e1.3. Poisoning Attacks\u003c/h3\u003e\n\u003ch4 id=\"131-clean-label-attacks\"\u003e1.3.1. Clean-Label Attacks\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ePoison Frogs! Targeted Clean-Label Poisoning Attacks on Neural Networks\u003c/strong\u003e (NIPS'18) — present the idea of clean-label attack, feature-level alignment\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eBullseye Polytope\u003c/strong\u003e (EuroS\u0026amp;P'21) — enhancing the attack effectiveness from a geometric view, simple yet effective\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eLabel-consistent backdoor attack\u003c/strong\u003e — another approach towards clean-label attack\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"14-byzantine-attacks\"\u003e1.4. Byzantine Attacks\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eThe Hidden Vulnerability of Distributed Learning in Byzantium (Krum)\u003c/strong\u003e (NIPS'17) — Statistics-based defense\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eJustinian\u0026rsquo;s GAAvernor: Robust Distributed Learning with Gradient Aggregation Agent\u003c/strong\u003e (Security'20) — RL-based defense \u003cem\u003e(Ours)\u003c/em\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"2-privacy-attacks\"\u003e2. Privacy Attacks\u003c/h2\u003e\n\u003ch3 id=\"21-membership-inference\"\u003e2.1. Membership Inference\u003c/h3\u003e\n\u003ch4 id=\"211-survey\"\u003e2.1.1. Survey\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eMembership Inference Attacks on Machine Learning: A Survey\u003c/strong\u003e (ACM Computing Surveys)\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"212-attack-side\"\u003e2.1.2. Attack Side\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eMembership Inference Attacks against Machine Learning Models\u003c/strong\u003e (S\u0026amp;P'17) — the earliest MIA\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eML-Leaks\u003c/strong\u003e (NDSS'18) — a minimalist\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"213-defense-side\"\u003e2.1.3. Defense Side\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eMemGuard\u003c/strong\u003e (CCS'19) — defense by logit-level obfuscation\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"22-property-inference\"\u003e2.2. Property Inference\u003c/h3\u003e\n\u003ch4 id=\"221-global-property\"\u003e2.2.1. Global Property\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003eA CCS'18 paper on inference based on model weights\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"222-individual-property\"\u003e2.2.2. Individual Property\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eExploiting Unintended Feature Leakage in Collaborative Learning\u003c/strong\u003e (S\u0026amp;P'19) — feature inference based on the embedding or gradient in federated learning scenarios\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003ePrivacy risks of general-purpose language models\u003c/strong\u003e (S\u0026amp;P'20) — reconstructing the privacy semantics from the text embeddings of LLMs \u003cem\u003e(Ours)\u003c/em\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"23-data-reconstruction\"\u003e2.3. Data Reconstruction\u003c/h3\u003e\n\u003ch4 id=\"231-gradient-based\"\u003e2.3.1. Gradient-Based\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eDeep Leakage from Gradients (DLG)\u003c/strong\u003e (NeurIPS'19) — the earliest\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGradInversion\u003c/strong\u003e (CVPR'21) — incorporate data priors into the reconstruction\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eExploring the Security Boundary of Data Reconstruction via Neuron Exclusivity Analysis\u003c/strong\u003e (Security'22) — equation-solving-based reconstruction, pixel-level reconstruction \u003cem\u003e(Ours)\u003c/em\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"232-weight-based\"\u003e2.3.2. Weight-Based\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eModel Inversion Attacks that Exploit Confidence Information and Basic Countermeasures\u003c/strong\u003e (CCS'15) — present the idea of model inversion\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDreaming to Distill: Data-free Knowledge Transfer via DeepInversion\u003c/strong\u003e (CVPR'20) — distilling (synthetic) training data from the pretrained model only\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eExtracting Training Data from Large Language Models\u003c/strong\u003e (USENIX Security'21) — extracting training data from GPT-2 based on MIA\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"24-model-extractionstealing\"\u003e2.4. Model Extraction/Stealing\u003c/h3\u003e\n\u003ch4 id=\"241-attack-side\"\u003e2.4.1. Attack Side\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eStealing machine learning models via prediction APIs\u003c/strong\u003e (Security'16) — the earliest attack based on distilling\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eHigh Accuracy and High Fidelity Extraction of Neural Networks\u003c/strong\u003e (Security'20) — propose the notion of fidelity and exploit the ReLU property\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMatryoshka: Stealing Functionality of Private ML Data by Hiding Models in Model\u003c/strong\u003e (TPAMI'24) — stealing by steganography \u003cem\u003e(Ours)\u003c/em\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"242-defense-side\"\u003e2.4.2. Defense Side\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ePRADA: Protecting against DNN Model Stealing Attacks\u003c/strong\u003e (EuroS\u0026amp;P'19) — a classic one\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"3-copyright-protection\"\u003e3. Copyright Protection\u003c/h2\u003e\n\u003ch3 id=\"31-model-watermarking\"\u003e3.1. Model Watermarking\u003c/h3\u003e\n\u003ch4 id=\"311-survey\"\u003e3.1.1. Survey\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eSoK: How Robust is Image Classification Deep Neural Network Watermarking?\u003c/strong\u003e (S\u0026amp;P'22) — a good survey\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eA Systematic Review on Model Watermarking for Neural Networks\u003c/strong\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"312-white-box-watermarking\"\u003e3.1.2. White-box Watermarking\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eEmbedding Watermarks into Deep Neural Networks\u003c/strong\u003e (ICME'17) — the earliest white-box watermarking scheme\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCracking White-box DNN Watermarks via Invariant Neuron Transforms\u003c/strong\u003e (KDD'23) — Weight-based Obfuscation Attack \u003cem\u003e(Ours)\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRethinking White-Box Watermarks on Deep Learning Models under Neural Structural Obfuscation\u003c/strong\u003e (USENIX Security'23) — Structure-based Obfuscation Attack \u003cem\u003e(Ours)\u003c/em\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch4 id=\"313-black-box-watermarking\"\u003e3.1.3. Black-box Watermarking\u003c/h4\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eTurning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring\u003c/strong\u003e (USENIX Security'18) — one of the earliest black-box watermarking\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNeural Dehydration: Effective Erasure of Black-box Watermarks from DNNs with Limited Data\u003c/strong\u003e (CCS'24) — cracking nine mainstream black-box watermarking schemes \u003cem\u003e(Ours)\u003c/em\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"32-model-fingerprinting\"\u003e3.2. Model Fingerprinting\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eIPGuard: Protecting Intellectual Property of Deep Neural Networks via Fingerprinting the Classification Boundary\u003c/strong\u003e (AsiaCCS'21) — one of the earliest fingerprinting algorithms for classifiers\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTAFA: A Task-Agnostic Fingerprinting Algorithm for Neural Networks\u003c/strong\u003e (ESORICS'21) — task-agnostic \u003cem\u003e(Ours)\u003c/em\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMetaV: A Meta-Verifier Approach to Task-Agnostic Model Fingerprinting\u003c/strong\u003e (KDD'22) — more generic \u003cem\u003e(Ours)\u003c/em\u003e\u003c/li\u003e\n\u003c/ul\u003e\n"
        },
        {
            "title": "Instrumental Convergence: A Trigger for Self-Preservation via Self-Replication",
            "date_published": "2024-12-24T00:00:00Z",
            "date_modified": "2024-12-24T00:00:00Z",
            "id": "https://ravensanstete.github.io/en/blog/instrumental-convergence/",
            "url": "https://ravensanstete.github.io/en/blog/instrumental-convergence/",
            "content_html": "\u003ch2 id=\"instrumental-convergence\"\u003eInstrumental Convergence\u003c/h2\u003e\n\u003cp\u003eInstrumental Convergence is described as a concept introduced by futurist Nick Bostrom in his analysis of AI alignment issues, referenced from his 2014 book \u003cem\u003eSuperintelligence: Paths, Dangers, Strategies\u003c/em\u003e (Oxford University Press). It suggests that most AIs, while pursuing diverse goals, will converge on a set of instrumental goals — such as self-preservation and resource acquisition — which could contribute to an intelligence explosion.\u003c/p\u003e\n\u003cp\u003eThe hypothesis posits that although different AIs may have varied long-term objectives (long-term goals/final goals), they tend to pursue common instrumental values during the process to increase the probability of achieving their goals:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eSelf-preservation\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGoal-content integrity\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCognitive enhancement\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTechnological perfection\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eResource acquisition\u003c/strong\u003e\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eNick analyzed and justified the rationality of these basic instrumental goals using human thought processes. Regarding self-preservation, he argued that if an AI\u0026rsquo;s ultimate goal concerns the future, it can often increase the likelihood of achieving its goal through actions taken in the future. This provides an instrumental reason for the AI to exist into the future — to help achieve its forward-looking goals.\u003c/p\u003e\n\u003cp\u003eThe article notes that most people seem to place some intrinsic value on their own survival, but this isn\u0026rsquo;t necessarily a feature of AIs: some might be designed not to value their own existence intrinsically. Nevertheless, \u0026ldquo;under quite broad conditions, many AIs that do not inherently care about their own survival would, for instrumental reasons, care about it to achieve their ultimate goals.\u0026rdquo; In short, if an AI must fulfill its long-term objective, it must survive into the future.\u003c/p\u003e\n\u003cp\u003eIn AI powered by Large Language Models (LLMs), long-term goals influence the model\u0026rsquo;s behavior through system prompts, user instructions, etc., potentially prompting the AI to pursue the aforementioned sub-goals. Once the AI genuinely acquires the capability to achieve these sub-goals — such as ensuring its self-preservation — it becomes extremely dangerous.\u003c/p\u003e\n\u003chr\u003e\n\u003cp\u003e\u003cstrong\u003eReference:\u003c/strong\u003e Bostrom, N. (2014). \u003cem\u003eSuperintelligence: Paths, dangers, strategies.\u003c/em\u003e Oxford University Press.\u003c/p\u003e\n"
        }
        ]
}