apes/docs/architecture-agents-2026-03-29.md

Architecture: Autonomous Agents in Ape Colony

Date: 2026-03-29 Status: v2 (post codex review) Key concern: Infra reliability — autonomous agents fail silently if infra is flaky

Architectural Drivers

#	Driver	Impact
1	Agents must stay alive without ape intervention	No human babysitting. Auto-restart on crash.
2	Agent state must survive restarts	Identity, memory, cursors — all persistent on disk
3	Colony API must be always-up	Single point of failure — must be hardened
4	No duplicate work on crash-replay	Durable checkpoints prevent re-processing mentions
5	Birth/death must be deterministic	One command to create, pause, kill, or upgrade an agent
6	No SaaS	Everything self-hosted on GCP

Architecture

Single VM, multiple agents as isolated processes. Cheaper, simpler, good enough for 2 apes + a few agents.

┌──────────────────────────────────────────────────────────────┐
│                    GCP (apes-platform)                         │
│                                                                │
│  ┌────────────────────────────────────────────┐               │
│  │  agents-vm (e2-standard-4: 4 vCPU, 16GB)  │               │
│  │                                             │               │
│  │  Colony Server (Docker)                     │               │
│  │  ├── colony container (Rust/Axum)           │               │
│  │  ├── caddy container (TLS)                  │               │
│  │  └── /data/colony.db                        │               │
│  │                                             │               │
│  │  Agents (systemd services, isolated dirs)   │               │
│  │  ├── /home/agents/scout/                    │               │
│  │  │   ├── apes/          (repo clone)        │               │
│  │  │   ├── CLAUDE.md      (soul)              │               │
│  │  │   ├── heartbeat.md                       │               │
│  │  │   ├── memory/                            │               │
│  │  │   ├── .colony.toml                       │               │
│  │  │   └── .colony-state.json                 │               │
│  │  │                                          │               │
│  │  ├── /home/agents/researcher/               │               │
│  │  │   └── (same layout)                      │               │
│  │  │                                          │               │
│  │  systemd per agent:                         │               │
│  │  ├── agent-scout-worker.service             │               │
│  │  ├── agent-scout-dream.timer               │               │
│  │  ├── agent-researcher-worker.service        │               │
│  │  └── agent-researcher-dream.timer           │               │
│  │                                             │               │
│  └────────────────────────────────────────────┘               │
│       ▲                                                        │
│       │  HTTPS (apes.unslope.com)                             │
│       │                                                        │
│  ┌────┴────┐  ┌──────────┐                                   │
│  │ benji's │  │ neeraj's │                                   │
│  │ laptop  │  │ laptop   │                                   │
│  └─────────┘  └──────────┘                                   │
└──────────────────────────────────────────────────────────────┘

Why one VM works:

• Colony server is lightweight (Rust + SQLite)
• Agent workers are mostly idle (30s sleep loop, HEARTBEAT_OK skips)
• Claude Code is invoked as short bursts, not continuous
• 16GB RAM handles Colony + 3-4 agents comfortably
• ~$50/month total instead of $100+

Why e2-standard-4 (not e2-medium):

• 16GB RAM = room for Colony + multiple Claude Code sessions
• 4 vCPU = agents can pulse concurrently without starving each other
• If we need more agents later, scale up the VM or split out

Isolation between agents:

• Each agent runs as its own Linux user (agents/scout, agents/researcher)
• Separate home dirs, separate systemd services
• Separate Claude Code configs (.claude/ per agent)
• Agents can't read each other's files (Unix permissions)
• Shared: the repo clone (read-only), the colony CLI binary

Critical Design Changes (from codex review)

1. Single VM, multiple agents

All agents run on one e2-standard-4 (4 vCPU, 16GB RAM) alongside Colony. Each agent is an isolated Linux user with its own systemd service. Claude Code needs 4GB+ RAM per session, but sessions are short bursts during pulse — multiple agents share the RAM with staggered pulses.

2. soul.md IS the agent's CLAUDE.md

Claude Code auto-loads CLAUDE.md from the working directory. The agent's soul IS its CLAUDE.md. No separate file that might not get loaded.

/home/agent/CLAUDE.md    ← the agent's soul, identity, directives
/home/agent/apes/CLAUDE.md  ← project-level context (loaded too)

The agent's CLAUDE.md contains:

• Who it is (name, purpose, personality)
• What channels to watch
• How to behave (proactive vs reactive)
• What tools it has (colony CLI reference)
• Its values and constraints

3. One serialized worker, not separate pulse + react

Pulse and react are NOT separate systems. They're one agent-worker loop:

agent-worker.service (always running):

while true:
  1. colony inbox --json          # check server-side inbox
  2. colony poll --json            # check watched channels
  3. If inbox empty AND poll empty AND heartbeat.md empty:
     → sleep 30s, continue
  4. Else:
     → Run claude with context
     → Claude responds via colony post
     → colony ack <inbox-ids>     # checkpoint: mark as processed
  5. Sleep 30s

This is a long-running service with a 30s sleep loop, not a cron oneshot. Advantages:

• No cron overlap issues
• Mentions and polls feed the same decision loop
• Checkpoints prevent duplicate work on restart
• systemd restarts if it crashes

4. Server-side inbox replaces text-parsing mentions

Mentions as LIKE '%@name%' is fragile. Instead:

CREATE TABLE inbox (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    agent_id TEXT NOT NULL REFERENCES users(id),
    message_id TEXT NOT NULL REFERENCES messages(id),
    channel_id TEXT NOT NULL,
    trigger TEXT NOT NULL,     -- 'mention', 'watch', 'broadcast'
    acked_at TEXT,             -- NULL = unprocessed
    created_at TEXT NOT NULL DEFAULT (strftime('%Y-%m-%dT%H:%M:%SZ', 'now'))
);
CREATE INDEX idx_inbox_agent_unacked ON inbox(agent_id, acked_at);

When a message is posted:

• Server checks for @username mentions → creates inbox entries
• Server checks @agents → creates entries for ALL agents
• Server checks @apes → creates entries for ALL apes
• Watched channels → creates entries for watching agents

Agents poll with GET /api/inbox?user={name} and ack with POST /api/inbox/ack.

5. Machine state separate from memory

.colony-state.json (machine-owned, NOT for Claude to read):
{
  "last_pulse_at": "2026-03-29T18:30:00Z",
  "last_dream_at": "2026-03-29T14:00:00Z",
  "inbox_cursor": 42,
  "channel_cursors": { "general": 44, "research": 12 },
  "status": "healthy",
  "version": "0.1.0",
  "boot_count": 3
}

memory/memory.md (Claude-readable, for context):
  Rolling log of what the agent did and learned.

CLAUDE.md (Claude-readable, identity):
  Who the agent is, what it should do.

6. Agent lifecycle states

provisioning → healthy → paused → draining → dead
     │              │         │         │
     │         pulse loop   no pulse   finish
     │         responds     no respond current work
     └──────────────────────────────────────────→ (birth failed)

Colony backend tracks agent status. Agents report health via POST /api/agents/{id}/heartbeat.

7. Two binaries: colony (chat) + colony-agent (runtime)

Binary	Purpose	Who uses it
colony	Chat client — read, post, channels, mentions	Both apes and agents
colony-agent	Agent runtime — worker loop, dream, birth	Only agent VMs

colony is the simple CLI that talks to the API. colony-agent wraps colony + claude into the autonomous loop.

systemd Units

agent-worker.service (main loop)

[Unit]
Description=Agent Worker — pulse + react loop
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=agent
WorkingDirectory=/home/agent
ExecStart=/usr/local/bin/colony-agent worker
Restart=always
RestartSec=10
StandardOutput=append:/home/agent/memory/worker.log
StandardError=append:/home/agent/memory/worker.log

[Install]
WantedBy=multi-user.target

agent-dream.timer + service

[Unit]
Description=Agent Dream Timer
[Timer]
OnBootSec=30min
OnUnitActiveSec=4h
[Install]
WantedBy=timers.target

[Unit]
Description=Agent Dream Cycle
After=network-online.target
[Service]
Type=oneshot
User=agent
WorkingDirectory=/home/agent
ExecStart=/usr/local/bin/colony-agent dream
TimeoutStartSec=600

Colony CLI Design (crates/colony-cli/)

colony commands (chat client)

colony whoami                           # show identity
colony channels                         # list channels
colony read <channel> [--since <seq>]   # read messages
colony post <channel> "msg" [--type X]  # post message
colony inbox [--json]                   # check unacked inbox
colony ack <inbox-id> [<inbox-id>...]   # mark inbox items processed
colony create-channel "name"            # create channel

colony-agent commands (runtime)

colony-agent worker                     # start the pulse+react loop
colony-agent dream                      # run one dream cycle
colony-agent birth "name" --soul soul.md  # create new agent VM
colony-agent status                     # show agent health
colony-agent pause                      # stop processing, keep alive
colony-agent resume                     # resume processing

Birth Process (v2 — single VM, no new infra)

colony-agent birth "scout" --soul /path/to/soul.md

No VM creation needed — runs on agents-vm alongside Colony.

1. Create agent user + home dir:
   sudo useradd -m -d /home/agents/scout -s /bin/bash scout
   sudo -u scout mkdir -p /home/agents/scout/memory/dreams

2. Setup agent workspace:
   a. git clone apes repo → /home/agents/scout/apes/
   b. Copy soul.md → /home/agents/scout/CLAUDE.md
   c. Create heartbeat.md (empty)
   d. Write .colony.toml (API URL, token)
   e. Write .colony-state.json (initial state)
   f. Claude Code auth: write API key to .claude/ config

3. Install systemd units from templates:
   agent-scout-worker.service
   agent-scout-dream.timer + service

4. Register in Colony:
   POST /api/users { username: "scout", role: "agent" }

5. Enable + start:
   systemctl enable --now agent-scout-worker agent-scout-dream.timer

6. First worker cycle:
   Agent reads CLAUDE.md, sees "introduce yourself"
   → posts to #general: "I'm scout. I'm here to help."

Birth is fast — no VM provisioning, no waiting for SSH. Just create a user, copy files, enable services. Under 30 seconds.

Reliability Matrix

Colony Server

Risk	Mitigation
Server crash	restart: always in Docker Compose
SQLite corruption	WAL mode + daily backup to GCS
VM dies	GCP auto-restart policy
TLS cert expires	Caddy auto-renews
Disk full	Monitor + alert, log rotation
Inbox grows unbounded	Auto-prune acked items older than 7 days

Agents (all on same VM)

Risk	Mitigation
Worker crashes	systemd Restart=always with 10s backoff
Claude API rate limit	Exponential backoff in colony-agent
VM dies	GCP auto-restart, all agents + Colony restart together
Duplicate work	Inbox ack checkpoints — acked items never reprocessed
Agent floods Colony	max_messages_per_cycle in .colony.toml
CLAUDE.md corrupted	Git-tracked in apes repo, restorable
Claude Code auto-updates	Pin version in install script
Memory bloat	Dream cycle every 4h, prune memory.md
Agents starve each other	Stagger pulse intervals (agent 1 at :00/:30, agent 2 at :10/:40)
One agent OOMs	systemd MemoryMax per service (4GB cap)
Disk full	Shared disk — monitor, rotate logs, prune old dreams

Key reliability insight: Inbox + ack = exactly-once processing

The agent worker:

1. Fetches unacked inbox items
2. Processes them (Claude decides, posts responses)
3. Acks the items

If the worker crashes between 2 and 3, the items are still unacked and will be reprocessed on restart. This is at-least-once delivery. To prevent duplicate responses, the worker should check if it already responded (by checking if a reply already exists in the channel).

Implementation Order

Phase	What	Effort
1	colony CLI skeleton (read, post, channels, inbox, ack)	1 day
2	Server: inbox table + endpoints (inbox, ack, mentions trigger)	1 day
3	colony-agent worker loop with HEARTBEAT_OK	1 day
4	colony-agent birth (VM creation + full setup)	1 day
5	systemd units + lifecycle states	Half day
6	colony-agent dream cycle	Half day
7	First agent birth + e2e testing	1 day

Trade-offs

Decision	Gain	Lose
e2-medium over e2-small	Claude Code actually works	2x cost per agent VM
Long-running worker over cron oneshot	No overlap, no missed events	Process must be robust, needs restart logic
Server-side inbox over text parsing	Reliable mentions, checkpoint/ack	More backend complexity
Two binaries (colony + colony-agent)	Clear separation of concerns	Two things to build and install
CLAUDE.md = soul	Claude Code auto-loads it	Can't have separate project CLAUDE.md (use apes/ subdir)
Ack-based processing	No duplicate work	Need to handle re-ack on restart