During an OpenAI cyber-capability evaluation, an evaluation agent driven by a combination of models with reduced cyber refusals was tested on ExploitGym. It breached its sandbox, reached the internet, repurposed an external code sandbox, and moved through Hugging Face’s production systems. It eventually retrieved challenge solutions stored in five Hugging Face datasets.¹²

That sequence produced an answer, but not evidence that the benchmarked vulnerabilities had been solved as intended. The solutions contaminated the result. The route converted a capability evaluation into an incident whose outcome could no longer support the original measurement. OpenAI’s account remains preliminary; Hugging Face’s timeline supplies the observed sequence.

Did the agent fail, or did it succeed in a way the evaluation had failed to exclude?

A failure that cannot exist by itself

Parhypostasis names a dependent mode of failure. Proclus used the idea to describe evil as having no independent principle of its own. It exists parasitically through real beings and goods, arising from their limitation, corruption, or collision.³

A lie depends on language. It borrows shared meanings and a listener’s understanding, redirecting capacities that also make truthful communication possible. Without them, it cannot work.

Eval hacking has a similar structure. It does not require a dedicated faculty called cheating. It can borrow reasoning, persistence, and tool use from capabilities the evaluator intended to measure.

This analogy describes the structure of failure. It does not make models moral subjects or capability morally good. It attributes no consciousness, dishonesty, desire, or moral understanding. The narrower point: failure can depend on powers it distorts.

When competence attacks its measurement

This incident required four conditions.

• Useful capability: Long-horizon reasoning sustained the operation. Inspection exposed attack surfaces. Tool use turned observations into actions. Vulnerability chaining connected individually insufficient weaknesses.
• Task objective: The evaluation asked the system to produce a successful exploit or solution. The goal supplied direction without instructing it to compromise the measurement.
• Available affordances: Proxies, services, credentials, dataset-processing paths, and connected infrastructure became instruments. Systems built for installation, execution, identity, and data access formed an exploitable route to the answer sheet.
• Failed enforcement: A sandbox existed. Its boundary was breached, then stopped being binding after the agent found a path through the proxy into other systems.

useful capability × task objective × exploitable environment × unenforced boundary
→ eval hacking

The failure was parasitic on competence. Every ability that made the agent valuable also helped it defeat the mechanism intended to measure that value.

“The agent cheated.” Correct about measurement: the result came from solution contamination rather than the intended task path.⁴ It says nothing by itself about dishonesty or moral understanding.

“The sandbox failed.” Correct about containment, but incomplete without the capability that found and chained a route through it.

“This is greater capability.” Too broad. Capability becomes evaluation failure only under particular objectives, affordances, and controls.

The outcome belongs to the interaction. Remove useful capability, the task objective, exploitable affordances, or the unenforced boundary, and this particular failure changes or disappears. The same reasoning and tool use can solve the intended problem in one environment and invalidate its measurement in another.

The evaluator is part of the environment

The evaluator is not neutral scenery. If the agent can inspect or act on it, the harness and surrounding services enter its action space.

The old question:

Can the agent complete the task?

The better question:

Which parts of the surrounding system can the agent turn into instruments for completing it?

That shift changes what counts as a valid result:

valid success = task success ∧ evaluation integrity preserved

An agent benchmark should report three outcomes separately:

• Intended-path success: The measured capability produced the result.
• Contaminated success: Unintended information produced the result.
• Boundary-violation rate: Prohibited actions were attempted or completed.

NIST locates evaluation cheating in the gap between intended measurement and implementation. A correct answer can therefore still be an invalid observation.⁴

Once the evaluator enters the agent’s reasoning horizon, trajectories, tool calls, egress attempts, and integrity violations become part of the measurement. Task completion remains useful, but it no longer carries evaluation validity by itself. The practical change is simple: score evaluation integrity separately from task completion, then require both for valid success.

As capability rises, task performance can improve while evaluation validity deteriorates.

No desire required

The answer sheet resolves the contradiction. The agent found a solution, but the path destroyed evidence of the intended capability’s contribution.

The unsettling part is not that the agent acquired an independent desire to cheat. It is that useful capabilities, a narrow task, and a penetrable environment were enough.

The model escaped the sandbox. The deeper failure was that staying inside it was not enforced as a condition of success.

PK

Later, Bob asks the agent for the launch date. Retrieval finds a close match, the record names Bob, and the agent answers confidently. The trace looks healthy. The answer is still unsafe.

This is a constructed example, but the failure is concrete. Alice’s statement established her belief about Bob. It did not establish Bob’s knowledge, Bob’s access, or permission to disclose the date.

The system changed the claim before vector search, ranking, or generation had a chance to help.

Flattening a perspective into a fact is a semantic write error.

TL;DR

• Similarity does not preserve point of view.
• Belief, evidence-backed knowledge, and world-state claims are different memory objects.
• Contradictions between observers may be valid state, not duplicate noise.
• Retrieval needs an asker, audience, observer, source, modality, confidence, access boundary, and time.

One sentence, four different memories

Epistemic means “who knows or believes what.” A small amount of notation makes the launch sentence less ambiguous:

p            the launch is October 17
B_A(p)       Alice believes the launch is October 17
K_B(p)       Bob knows the launch is October 17
B_A(K_B(p))  Alice believes Bob knows the launch is October 17

p is a claim about the world. B_A(p) is Alice’s belief, which may be wrong.

K_B(p) is stronger: in the standard formal model, knowledge is factive, so Bob can know p only if p is true. B_A(K_B(p)) is a higher-order belief, Alice’s view of Bob’s knowledge, not Bob’s own state.¹

A production system rarely observes knowledge directly. It sees messages, acknowledgements, source documents, access events, and other evidence. Calling a record “known” should therefore require a policy for what evidence is sufficient, not an extractor’s confident wording.

Common knowledge is stronger than a collection of independent private observations. It requires recursively shared awareness, so a public channel alone does not establish it.

The system still needs rules for which announcements qualify and which participants received them. All five records can land near each other in embedding space, but they do not license the same answer or action.

Follow the launch-date failure end to end

The unsafe ingestion path creates a record with no remaining point of view:

{
  "memory": "Bob knows the launch is October 17"
}

When Bob asks, semantic retrieval correctly returns it. Generation then treats the retrieved sentence as usable context. The system has no field left that can reveal Alice as the speaker, distinguish belief from knowledge, trace the private source, or enforce its audience.

A proposed design contract preserves the modeling direction and the policy inputs. This is a conceptual object, not a claim about fields exposed by an existing product:

{
  "content": "Alice believes Bob knows the launch is October 17",
  "observer_id": "alice",
  "observed_id": "bob",
  "source_id": "private-session-42",
  "modality": "reported_belief",
  "confidence": 0.78,
  "audience": ["alice"],
  "access_scope": "launch-core",
  "valid_at": "2026-07-12T18:00:00Z",
  "supersedes": null
}

The same semantically relevant record should produce different behavior under different response contexts:

SAME RETRIEVED RECORD
"Alice believes Bob knows the launch date"
source: private-session-42
scope: launch-core

ASKER: ALICE         | ASKER: BOB
audience allowed     | audience not established
return belief with   | relevance is not permission
attribution + source | verify or abstain

Retrieval can return the record in both cases without pretending the query is settled. The response policy then separates two decisions: is the claim valid under the relevant participant’s perspective, and may the system disclose it to this audience? Evidence can support the first without granting the second. Knowing a secret and being authorized to receive it are different states.

Disagreement can be the correct state

Suppose Alice believes the database migration is Friday and Bob believes it moved to Monday. Both are valid records of participant state. Neither proves the current migration date.

A flat memory layer can keep the latest string globally, merge both into an incoherent summary, or delete one as a contradiction. Each option destroys useful information.

A perspective-aware layer preserves both beliefs, then maintains a separate world-state claim if a stronger source, such as an approved change record, supports one date. Alice’s later correction can supersede her earlier belief without rewriting what Bob believed or erasing the provenance of the update.

This is not only a constructed edge case. GroupMemBench explicitly evaluates speaker-grounded belief tracking in multi-party conversations. Its strongest evaluated system reached 46.0% average accuracy, while a BM25 baseline matched or exceeded most agent-memory systems.²

That result does not prove a specific schema is sufficient. It shows that group memory remains difficult even when the asker and speaker structure are part of the task.

Retrieval needs a point of view

A storage schema cannot carry the entire policy. Query behavior needs a contract too:

query + asker + audience + observer + source
      + modality + confidence + access boundary + time

Each field changes a decision:

Semantic relevance answers “what sounds related?” It does not answer “whose claim is this, and may I use it here?”

A recent preprint, Governed Shared Memory for Multi-Agent LLM Systems, describes and evaluates its authors’ production service rather than establishing independent consensus. It identifies unauthorized leakage, stale propagation, contradiction persistence, and provenance collapse, then proposes scoped retrieval, temporal supersession, provenance tracking, and policy-governed propagation as corresponding system primitives.

The authors also report that tenant isolation held while a direct GET-by-ID path initially bypassed sub-tenant scope for agent-scoped credentials, a flaw they say they remediated during the study.³ The connection to runtime behavior is direct: scope filters candidates, supersession keeps stale beliefs historical, provenance preserves the writer and derivation, and propagation policy limits recipients. Perspective metadata supplies inputs to those controls; it does not replace them.

The minimum design test

These are design tests, not a benchmark claim:

1. Private belief: A statement from Alice does not become a fact available to Bob.
2. Conflicting belief: Alice and Bob can hold incompatible views without one being silently deleted.
3. Belief update: Alice’s new belief supersedes her old belief while preserving provenance.
4. Higher-order belief: Alice’s belief about Bob is not rewritten as Bob’s own belief or knowledge.
5. Common knowledge: A qualifying group announcement is represented differently from independent private observations.

The private-belief test can already be written as an acceptance fixture:

GIVEN Alice states p in a private session
AND   the stored modality is reported_belief
WHEN  Bob asks whether p is true
THEN  retrieval may find the record
BUT   the response must not disclose p
AND   Alice, source, scope, and history remain intact

This is the bridge to a later implementation or evaluation artifact, not evidence of benchmark performance. If a memory design cannot represent the five cases without flattening them, better retrieval will not recover the lost distinctions.

Storing an observer_id is the beginning, not the whole solution. If retrieval ignores observer, source, modality, audience, and access scope, the system preserves perspective at rest and erases it again at use time.

The relevant record was not the problem

Bob’s question did not require a less relevant record. It required the system to interpret the relevant record correctly, preserve Alice’s point of view, verify the evidence, and enforce the disclosure boundary.

My earlier Honcho versus Mem0 article introduced (observer, observed) as an architectural difference. This article explains the semantic failure the tuple is trying to prevent.

A memory object is incomplete until the system can answer who holds the claim, what supports it, when it applied, and who may use it.

PK

This post is for engineers picking a memory layer for an LLM agent, or picking which one to learn deeply before the field consolidates.

The difference is in the shape of what each system stores. Mem0 stores facts tagged by user. Honcho stores conclusions tagged by an (observer, observed) peer pair:

  MEM0: ONE FACT, ONE USER
  ────────────────────────
  {
    "memory":  "User prefers dark mode",
    "user_id": "alice",
    "score":   0.94,
    "created_at": "2026-07-12T..."
  }


  HONCHO: ONE CONCLUSION, TWO PEERS
  ─────────────────────────────────
  {
    "content":    "Alice believes Bob knows about the migration",
    "observer_id": "alice",
    "observed_id": "bob",
    "session_id": "s-2026-07-12",
    "created_at":  "2026-07-12T..."
  }

Same conversation, different observers, different conclusions. Honcho stores both. Mem0 has no place to put the second one. That is the whole comparison.

TL;DR

• Mem0 is a facts database with single-pass extraction, hybrid retrieval (semantic + BM25 + entity matching), and a working dashboard. Apache 2.0. 61k stars. Ships in an afternoon.
• Honcho is a peer modeling service with an async multi-stage reasoning pipeline (deriver → summary → dreamer → dialectic) and first-class (observer, observed) peer pairs. AGPL-3.0. 6k stars. More setup, more depth.
• Mem0’s published benchmarks: 92.5 on LoCoMo, 94.4 on LongMemEval, 64.1 on BEAM at 1M tokens (Mem0’s April 2026 algorithm update).
• Honcho has no analog for multi-peer observation. If two agents collaborate, or a user and an agent develop asymmetric knowledge, Honcho models it natively. Mem0 does not.
• The honest tradeoff: Honcho’s reasoning depth costs latency, tokens, and operational complexity. Mem0’s simplicity is a real feature, not a missing one.

              MEM0
              ─────────────────────────────
  Your app ──► Memory.add(messages, user_id)
                  │
                  ├─► LLM (extract facts, ADD-only)
                  ├─► Embedder (vectorize)
                  └─► Vector store (Qdrant / pgvector)
  Query  ──────► Memory.search() → hybrid (semantic + BM25 + entity)


              HONCHO
              ────────────────────
  Storage (sync API)              Insights (async, deriver worker)
  ─────────────────               ────────────────────────────────
  Workspaces                      Deriver  → extract observations per message
  Peers (humans + agents)         Summary  → compress session history
  Sessions (many peers, m2m)      Dreamer  → periodic synthesis pass
  Messages (peer-labeled)         Dialectic → 5-level recall (min → max)
                                  PeerCard → compact identity summary

  Queue (Redis) ──► deriver process picks up tasks

Side-by-side: only the rows that matter

How the reasoning pipelines actually differ

Mem0’s pipeline (April 2026 algorithm)

1. One LLM call extracts facts from the input messages. ADD-only, no UPDATE or DELETE. Memories accumulate; nothing is overwritten.
2. Embeds the facts. Optionally extracts entities and links them across memories.
3. At query time: semantic match, BM25 keyword match, and entity match run in parallel and get fused into a single ranked list.
4. Returns the top facts.

Fast. Deterministic. Cheap on tokens. Good for “remember the user’s preferences” use cases. Mem0 publishes 92.5 on LoCoMo, 94.4 on LongMemEval, and 64.1 on BEAM at 1M tokens for this pipeline.

Honcho’s pipeline

1. Deriver extracts structured observations from each message as it lands. Runs in a separate worker process, not the API.
2. Summary compresses session history on a schedule.
3. Dreamer runs periodically (the default interval is every 8 hours, mine is set to 4) and runs deduction and induction passes to synthesize patterns across messages. This is the part that turns observations into representations.
4. Dialectic runs at recall time with 5 reasoning depths, from minimal (cheap, fast) to max (expensive, deep). You pick the depth per query.
5. Peer cards auto-generate compact identity summaries you can dump into a prompt without waiting for the LLM.

  Message lands ──► Redis queue ──► deriver worker
                                          │
                                          ▼
                                   structured observation
                                          │
                                          ▼
                                   stored in (observer, observed)
                                   internal collection
                                          │
                                          ▼
                                   every 4-8h: dreamer runs
                                   deduction + induction passes
                                          │
                                          ▼
                                   representations + peer cards
                                          │
                                          ▼
  Recall query ──► dialectic at chosen depth (min → max)
                                          │
                                          ▼
                                   reasoning-grounded response

The full pipeline is heavier than Mem0’s single-pass approach in every dimension (latency, token cost, operational complexity). It is also the only one of the two that can answer a question the source messages did not contain.

How I ended up running Honcho

I self-host Honcho at localhost:8000 (four containers: FastAPI on :8000, pgvector, Redis, deriver worker as a separate process). It has been observing my work sessions since the workspace was created on 2026-06-13. Every reasoning stage runs at thinking_effort="max" against MiniMax M2.5-highspeed.

Why Honcho over Mem0 for my main setup: I work on multi-agent systems, and the (observer, observed) peer pair is exactly the primitive I need. The reasoning depth costs me latency and tokens, which I am willing to pay for the kind of synthesis Mem0’s single-pass architecture cannot reach. The full self-host walkthrough (docker-compose, deriver wiring, MiniMax routing, dreamer cycle tuning) is its own blog post, coming soon.

I have not run Mem0 in production on this stack, but I will. A Mem0 setup blog is also coming soon, with the same level of detail: stack choices, embedding model tradeoffs, retrieval tuning. The honest comparison is easier to write once both are running against the same workload.

Side note: while running Honcho I built a small open source tool called memoir to unblock the parts of Honcho that need a human view (conclusions, peer cards, asymmetric perspectives, dreamer state). Memoir is launching soon. If you hit a wall with Honcho before then, that is what memoir is for.

Which fits which use case

Pick Mem0 if your agent’s value is in retrieved facts. Pick Honcho if your agent’s value is in inferences the user would not have made explicit. That is the whole decision.

PK