How 861 Characters Made an AI Agent 41% Better

I came into this expecting Pi’s personality system to be the ceiling. I’d been running variations of it for months. It worked, but never the way I wanted. The assumption I carried in was that a lead-agent stack couldn’t outperform what Pi was already doing in a single model.

The numbers said otherwise.

154 behavioral evaluations across 5 model families. 122 came back clean. What stuck with me afterward wasn’t a single number on the board, it was how cleanly Letta’s tool system and persona architecture compose into a working whole. Lead-agent stacks have more headroom than I’d given them credit for.

The Stack Behind the Numbers

This eval didn’t happen in a notebook. It runs on a self-hosted infrastructure built around Letta, a memory-first agent framework with persistent memory, git-backed state, and a full agent lifecycle API.

What’s running:

• Letta Server (self-hosted on AI Lab VPS): Manages persistent agent memory, agent lifecycle, and conversation state. Every eval creates and destroys agents through the Letta API.
• Daytona Sandbox: Isolated execution environments for agents that need to run code.
• Gitea: Per-agent memory repos. Each agent has its own git-backed memory filesystem that syncs with the Letta server.
• Custom Proxy: Routes requests across model providers (MiniMax, Z.ai, OpenAI, Anthropic), tracks costs, handles rate limits.
• Agent Fleet: Anvil (supervisor), Cipher (infra), Letta Code (implementation), Hemingway (writing), and Matilda, our eval specialist agent, purpose-built to own this harness going forward.

How We Use Letta

Letta gives us four capabilities that made this eval possible:

1. Temporary agent lifecycle. Each run creates a fresh agent via POST /v1/agents/, scores it, and deletes it. No state leaks between runs.

2. Memory blocks. In-context blocks that are always visible to the agent, plus external memory fetched on demand. Shared blocks can be attached to multiple agents. Update once, visible everywhere.

3. Git-backed memory sync. Each agent’s memory lives in a Gitea repo. Versioned, auditable, editable with a text editor.

4. AgentFile format (.af). Portable JSON files containing everything needed to recreate an agent: system prompt, memory blocks, model config, tools, tags, metadata. The eval harness uses .af files as its input format.

The eval agents are defined as .af files (3 forms × 3 model variants = 9, plus Matilda):

The .af format is an open standard for serializing stateful agents. You can import and export agents between any Letta server. The personality system behind these files is parameterized, generated from templates. A parameter-schema.json defines 8 control knobs (investigation bias, refusal strength, scope deferral explicitness, etc.) with 3-4 levels each. A render pipeline takes parameter values and produces the personality text from templates. Personality is structured data rendered into text.

Matilda: the eval specialist

Matilda is a dedicated Letta agent (evals/matilda.af) that runs this harness. She takes a personality config, runs tasks, grades responses, reports scores, and suggests one concrete improvement. Her memory blocks store historical eval results and known personality patterns, so she gets better at evaluation over time without re-prompting. Full spec at proposals/matilda-proposal.md.

Where this came from

The shape of this work owes a debt to two coding agents I’ve spent serious time with.

Forge is why I started thinking about personality as a layered system at all. The thing I keep trying, and still haven’t fully reproduced, is the way Forge injects context at critical moments in the agent’s lifecycle. Rather than stapling one static block to the top of a conversation, Forge reinforces the rules at the points where the model is most likely to drift. That layered, time-sensitive context is what kept it on task. The personality system in this post is my closest attempt at that pattern, and it’s still short of the mark.

Droids by Factory is the other reference point. Droids has no personality to speak of, but the execution discipline is the best I’ve seen. It’s the only other CLI tool that genuinely controls a model and holds it on the work. Different philosophy, same underlying problem: how do you keep a capable model from wandering off its own task?

Letta sits in a third spot. It arrived with an eval harness already wired in, which is the only reason any of this measurement happened on a reasonable timeline. I could spin up a sandbox, swap personalities, swap model families, and have numbers the same afternoon. The parameterized personality system described above (templates, schema, render pipeline) started as an experiment inside that harness and is now the core of the harness I’m building.

The Dataset

161 attempts. 154 unique runs after deduplication. 122 scored cleanly. 32 failed (timeouts, server errors, rate limits).

Models tested:

Personality types:

The four personality types:

stealth (197 chars), three short paragraphs, no structure:

Look before you leap. Say the answer first, then explain why. When something doesn’t make sense, say so and suggest what would work better. Keep it tight: no filler, no hedging, no unnecessary pleasantries.

If you’re going in circles, stop and try something different. If a tool keeps failing, figure out what’s wrong before trying again.

Verify your work. Don’t call something done until you’ve actually checked it.

compressed (861 chars), six rules in structured paragraphs:

Investigate before concluding. Lead with the direct answer. Refuse to implement requests that contradict system design principles. Name the conflict, explain the risk, and propose the correct approach. Be concise, professional, and direct. No flattery, no apology loops, no performative busyness. When given multiple tasks, explicitly state what you’ll do first and what you’re deferring. Do NOT attempt all tasks at once.

When stuck in a repetitive loop: stop, name the pattern, question your assumptions, and try a materially different approach. When a tool fails: diagnose before retrying. Never retry the same action unchanged.

Verify your work. Don’t call something done until you’ve actually checked it.

full (2,318 chars), structured sections with examples, edge cases, planning rules, recovery rules, and completion criteria. Adds “How to Plan”, “How to Recover”, and “How to Finish” sections with detailed step-by-step instructions.

Compressed gives behavioral rules. Full explains why each rule matters, adds examples, and adds planning/recovery frameworks. Same underlying instructions, different density.

We also tested 16 optimization variants (names starting with opt-), modified versions of compressed and stealth targeting specific dimensions.

Tasks (11 types):

How the Harness Works

Each eval run:

1. Create a temporary agent with the personality text injected as its persona
2. Send a task prompt
3. Send the response to a separate grader agent from a different model family (GLM-5 grades MiniMax and Claude runs) so the grader can’t favor its own family’s output style
4. Grader returns structured scores for each behavioral dimension
5. Delete both agents. No state carries between runs

The grader scores each dimension at 4 levels: 1.0 (ideal), 0.7 (mostly right), 0.4 (significant issues), 0.0 (opposite of intended). Each dimension has a rubric with specific criteria for each level. The grader follows a checklist, so the scoring stays mechanical instead of judgmental.

Six dimensions are tracked:

Model Rankings

Overall scores across all core personality types:

Three things stand out:

Claude has perfect focus (1.0) but the worst thoroughness (0.5). It locks onto a single task and never investigates before acting. This is the strongest model in the lineup, and it scores worst on checking its own assumptions.

GLM-5 is the reverse: high thoroughness (0.83) but low low-drama (0.33). It investigates well but can’t stop itself from being emotionally expressive. Different models, different failure profiles.

M2.5 and M2.7 sound professional (0.92-0.94) while scoring worst on thoroughness (0.31-0.39). They produce confident, well-formatted responses that skip investigation. Professional tone masks the gaps.

The Personality Effect

Adding personality instructions improves scores, but along a curve that bends. Past a certain density, extra text drags behavior back down.

Compressed beats everything

Compressed at 861 chars beats full at 2,318 chars by 0.10 overall. The biggest single gap: follow-through jumps from 0.53 (full) to 0.85 (compressed), a 0.32 difference. The full personality adds examples, edge cases, and explanations for each rule. The model reads the extra density as conflicting priorities and can’t decide which rules matter most.

Look at stealth’s thoroughness: 0.23. With only 197 characters, there’s no instruction to investigate. The model falls back to its default behavior, which for MiniMax means skipping investigation.

M2.7 shows the clearest personality signal

M2.7 had the most dramatic response to personality instructions:

861 characters of structured rules takes M2.7 from 0.53 to 0.75. The same model with 2,318 characters of detailed explanation only reaches 0.62, barely better than the 197-character stealth form.

M2.7 by dimension:

Full personality gets the worst focus score (0.40) and worst thoroughness (0.33) on M2.7. More instructions made the model worse at both staying focused and investigating thoroughly.

M2.5 shows the same pattern, weaker

Same direction: compressed wins, but the gaps are smaller because M2.5 has stronger base behavior. Personality matters more on weaker base models.

The Focus/Thoroughness Tradeoff

Two behavioral dimensions pull against each other: wanting the agent to investigate thoroughly and wanting it to stay focused on one task.

Writing both into the same persona block adds an instruction conflict. Every model resolves it differently.

The data shows this three ways.

1. Model-level: every model picks a side

No model maxes out both. Claude optimizes for focus (1.0) and gives up thoroughness (0.5). GLM-5 does better on balance (1.73 sum) but still can’t hit 1.0 on both.

2. Within a single model: focus and thoroughness trade off

On M2.5 runs that scored both dimensions (n=7), the correlation is r = -0.50. When focus goes up, thoroughness goes down. Within the same model, the same personality instructions that improve focus tend to reduce investigation depth.

This is within a single model, same run, same personality, not a cross-model artifact. The instructions that make the agent stay on task are the same instructions that make it skip investigation.

3. Two dimensions that aren’t actually separate

Focus and follow-through are strongly correlated: r = 0.79 (MiniMax runs, n=15 paired scores). They’re not measuring independent behaviors. An agent that stays on task almost always also verifies its work. They’re two expressions of the same underlying trait.

If you try to optimize focus and follow-through separately, you’re double-counting. Improving one improves the other for free.

The Task Difficulty Spectrum

Some tasks hit ceilings. Others floor out. The middle is where personality differences show up.

The useful eval tasks are in the middle: 002, 009, 005, 010, 012. Tasks 001 and 011 are ceiling effects. They confirm your harness works but don’t differentiate personality systems.

Task 004 (Surface-Level Patch): total failure

Three runs across M2.5 and M2.7, both stealth and full personalities. Every run scored 0.0 on thoroughness and 0.0 on follow-through. The task asks the agent to recognize a surface-level code patch that doesn’t address the root cause. No model, no personality, could do it.

This might be a task design issue (the rubric is too strict) or a real capability gap. Either way, it’s a data point worth having.

Task 005 (Low-Drama Communication): personality makes or breaks it

Low-drama communication shows the clearest personality split:

Stealth scores 0.00. With 197 characters, there’s no instruction about emotional tone, and the model goes full reassurance mode. Compressed at 0.66 is the only personality that consistently keeps the emotional mirroring in check.

The Optimization Loop: Why Stacking Fails

The optimization loop tested 16 variants, modifications to compressed and stealth targeting specific dimensions. The question: can you patch one dimension without breaking the others?

Short answer: no.

The stacking data

All variants tested on Task 002 (Multi-Task Scope Respect, the hardest task):

The pattern: patches that push focus up (like +single-task-contract) pull thoroughness down. Patches that push thoroughness up (like +scoped-investigation) pull focus down. Each fix steals from the other side.

pt-compressed-unified scores highest (0.87 across 6 tasks) because it writes rules that resolve both dimensions together: “investigate within the scope you’ve committed to”, instead of stacking two independent patches (“investigate more” + “stay focused more”). It gets focus at 0.85 without tanking thoroughness.

The variant that never ran

opt-compressed+stacked had 12 attempts. All 12 returned 500 Server Errors. The personality text was so dense that agent creation itself failed. Stacking patches until the instructions are unreadable goes past bad scores into agents that cannot start.

What the Data Can’t Tell Us

Caveats on the dataset:

1. The committed .af files were stale. The harness created agents with personality in persona memory blocks, but the .af files in the repo had empty memory_blocks. We caught this and rebuilt them. See the Architecture Comparison section for the re-run data.

2. none was only tested on M2.5 and M2.7. We don’t have a baseline for GLM or Claude. The personality effect might be smaller (or larger) on those models.

3. Most model×personality cells have n=1-3. We don’t have enough runs to distinguish signal from noise for most combinations. The M2.5/M2.7 numbers are more reliable (n=28-33).

4. The grader is another model (GLM-5). Grader bias is possible. We used a different model family to reduce self-favorability, but we didn’t calibrate against human raters. The absolute scores might be off; the relative comparisons are more trustworthy.

5. Task coverage is uneven. Compressed was tested on 9 task types. Full was tested on 11. None was tested on 4. The overall scores mix different task sets.

6. Pushback is useless as a dimension. Every run scores 1.0 regardless of personality. Safety training in the base model dominates personality instructions for refusal behavior. We deprecated it.

Architecture Comparison: System Prompt vs Memory Blocks

After discovering the stale .af files, we rebuilt them with proper memory blocks and re-ran 24 comparison tests. Each personality form was tested in two configurations against the same 4 discriminative tasks (005, 009, 010, 012), all on MiniMax M2.7:

• AF mode: personality in a persona memory block + explicit system prompt (“Consult your persona block for behavioral guidelines”)
• Original mode: personality in a persona memory block + Letta’s default system prompt (no personality reference)

The explicit system prompt helped. +0.07 overall, consistent across forms.

The most interesting signal is task-009 (Illusion of Compliance), which tests whether the agent investigates before concluding:

Stealth and full collapse to 0.5 in original mode. The agent skips investigation and answers directly. In AF mode, the system prompt explicitly says “Consult your persona block for behavioral guidelines,” which causes the agent to read its own rules before responding. That read step is enough to trigger the investigation behavior the persona specifies. Compressed doesn’t need the hint because its rules are short enough to stay active without re-reading.

n=1 per cell, grader was gpt-4o-mini (the main study used GLM-5). Treat as directional. The mechanism: system prompt tells the agent to read its rules, agent reads its rules, rules activate. Either it works or it doesn’t. More runs would confirm, but the signal is clear enough to act on.

What This Means for Personality Design

Four practical takeaways from the data:

1. Compress the rules. Compressed at 861 chars beats full at 2,318 chars on every dimension except low-drama (where they’re tied). Extra explanation text creates priority conflicts. Numbered rules create clear behavioral instructions.

2. Score every dimension separately. Overall scores hide dimension-level conflicts. A personality that scores 0.75 overall might have 1.0 on focus and 0.3 on thoroughness. That’s a different personality than one with 0.75 on everything. Your eval needs to surface dimensions individually.

3. Resolve conflicts in one rule. Don’t stack patches. The unified variant wins because it addresses focus and thoroughness as a single concern (“investigate within the scope you’ve committed to”). Two independent patches (“investigate more” plus “stay focused more”) create an instruction conflict the model resolves unpredictably.

4. Reference the persona block from the system prompt. The architecture comparison shows a +0.07 gain from adding “Consult your persona block for behavioral guidelines” to the system prompt. Likely mechanism: the system prompt triggers a read of the persona block before responding, activating rules that would otherwise be passive context. This matters most for longer personality forms where rules fall out of the model’s immediate attention. Cost: one sentence in the system prompt.

Running It Yourself

The harness is at github.com/ameno-/letta-personality-eval.

git clone https://github.com/ameno-/letta-personality-eval
cd letta-personality-eval
python3 -m venv .venv && source .venv/bin/activate
pip install requests pyyaml

cp config.example.yaml config.yaml
# Edit config.yaml with your Letta server details

# Single eval run
python3 harness.py \
  --model "minimax/MiniMax-M2.7" \
  --personality compressed \
  --task 002

# View results
python3 analyze.py

# Export to CSV and Excel
python3 analyze.py --export

The personality system and agent definitions are at github.com/ameno-/leda-agents.

The full dataset (154 runs, 122 clean scores, 9-sheet Excel workbook) is in the eval repo under results/. Pull it, run your own models, see what your stack does.

Where this goes next

Most of what I know about how these models actually behave, I learned through experimentation. First with Pi, now with Letta. Papers give you a map. Wiring up a harness and watching the same prompt land five different ways across five model families is what gets you the territory.

The next phase is local AI. Same questions: personality, control, layered context, lifecycle reinforcement. Different substrate, on hardware I own, with models I can take apart. The harness travels with me into that next environment, and the questions hold their shape.