A cache can be correct in every answer and broken in every useful sense. It can return the proper records, hold its locks with exemplary care, satisfy the race detector, and pass a formidable suite of tests. Meanwhile, the database is doing all the work.

This was such a cache. It had been designed to distrust itself whenever an in-flight refresh might be obsolete—a prudent rule in a system that helped coordinate moving vehicles. The trouble was that its test for obsolescence watched the entire fleet. A change to one unit could make the cache doubt an unrelated unit and, once doubt entered the system, doubt proved difficult to dislodge.

The defect was not in the guard’s logic. It was in the guard’s jurisdiction.

Two goroutines can each behave correctly and still tell a subscriber the wrong story. One publishes immediately after an entity changes; the other sweeps all entities once a second. If the sweep reads first and sends last, the subscriber sees the new state followed by the old one. Its view moves backward.

The race calls for serialization. The obvious lock-per-entity solution works, then quietly creates a second problem: locks now have identities and lifetimes that must follow an unbounded, externally influenced set of entity IDs. A fixed array of striped mutexes enforces the ordering without acquiring that registry and its failure modes.

A clock can be wrong by doing exactly what it was designed to do. In a Go service coordinating moving equipment, the wall clock had to accept corrections so its outbound timestamps agreed with other machines. The same service also used time to decide whether equipment was still responding. A backward correction could make a silent unit appear newly alive.

The hazard was easy to describe and easy to reintroduce. The durable fix was to move it out of documentation and into the type system, where a tired engineer or a coding agent would meet a compiler error instead of a cautionary comment. The service did not need a better now(). It needed two different questions.

A design review named the hazard. The specification measured the hot path to the nanosecond. Test coverage reached 97.7 per cent, and the mutation score was 0.92. None of those facts prevented a read-through cache from containing the outline of a database outage.

The defect was caught on a feature branch and never merged. It is worth examining because the first review found exactly half the problem. Its remedy removed a concurrent query spike and replaced it with a serial, unending one: a thundering herd became a siege.

Most writing about agents begins with the prompt, the most visible and least durable part of the system. In production, the consequential decisions belong to the harness: the code between a user and a model loop that decides who may run what, against which files, on whose budget, and with what durable record afterward.

yc-software/qm is an unusually legible example: a headless TypeScript core built with Fastify and PostgreSQL, driving interchangeable agent runtimes across Slack, a web application, and scheduled jobs. Its sandboxes, memory, and credentials are isolated by scope. The code is a dense catalog of production decisions; ten of them transfer well even to a single-user command-line agent.

Discrete-event simulation has an odd relationship with parallelism. A study may contain thousands of independent replications, an almost comically parallel workload, yet the model inside each replication is often built around a deliberately sequential event loop. Make the outside parallel and seeding, serialization, and reproducibility become awkward. Make the inside parallel and causality becomes the problem.

llmsim is a parallel discrete-event simulation (DES) library for Python 3.14 and later. It keeps the generator-as-process model that SimPy established and builds a typed, __slots__-based core around a share-nothing rule: each event loop owns its state, and parallel components communicate only at explicit boundaries. The core and all three parallelism tiers are implemented and gated by determinism tests in CI as the project approaches its 1.0 release.

Despite the name, the core library has nothing to do with calling LLMs. The name reflects two bookends: one of the two examples simulates an LLM-serving agentic workload, and the post-1.0 roadmap adds LLM-powered scenario generation — strictly at design time, never inside a running simulation.

An architecture document is organized to make a system understandable. A delivery plan is organized to make the system buildable. The distance between the two is where elegant designs often acquire vague epics, hidden dependencies, and a final “hardening” sprint containing all the risk the plan failed to name.

A coding agent can produce a remarkable amount of software before anyone has agreed on what the software is for. That speed is exhilarating during the first afternoon and expensive during the third month, when an architectural decision survives only in an old chat, a validation rule has become folklore, and two apparently sensible features no longer fit together.

Spec-driven development, or SDD, is my way of slowing down the decisions while allowing the implementation to remain fast. It creates a versioned contract between developers and coding agents: the specification records the what and the why; the implementation supplies the how. The method is adapted from DeepLearning.AI’s short course on the subject and its course materials, then altered by using it on real projects.

Every old codebase contains a history of reasonable decisions that stopped being reasonable together. A deadline introduced an async void callback; a later feature shared its collection across threads; an emergency credential became permanent configuration. The system kept running, which made each compromise look survivable, until the compromises began to interact.

An LLM cache can be fast, cheap, and wrong. Record a prompt and its answer, and an exact-match cache works beautifully until a user changes reset my password to password-reset help. Add semantic matching, and the cache recognizes the question — but may now retrieve yesterday’s answer after the policy document behind it has changed. The optimization has preserved the prose and discarded its truth.

Reverb is a Go library and standalone service designed around both failure modes. It combines a two-tier cache (exact SHA-256 match, then embedding-cosine similarity) with knowledge-aware invalidation: every cached entry tracks the source documents it was derived from, and a change-data-capture pipeline evicts entries by causality when their sources change. TTLs become a backstop, not the primary correctness mechanism.

A new agent and an unreliable one can receive the same trust score. Give the newcomer no history and the veteran ten thousand evenly split outcomes, and a typical scalar model assigns both a 0.5. The arithmetic is tidy; the meaning is not. One score says, “there is not enough evidence.” The other says, “there is plenty, and it is contradictory.” A system deciding who may review a report, call a tool, or control a workflow ought to know the difference.

MultiTrust keeps that distinction visible. It represents trust as a Subjective Logic opinion — belief, disbelief, and uncertainty, constrained to sum to one — and exposes the model through an MCP server. A Model Context Protocol-aware agent can therefore ask for a trust decision as an ordinary tool call, while an operator can still inspect the evidence behind it.

The most unnerving agent failure is not a crash but a pause that never ends. No exception is raised. Each participant is still alive. The workflow simply stops producing tokens, and the operator is left with a spinner and a question: is the system slow, or will it wait forever?

From the standpoint of concurrency, a multi-agent workflow is a small distributed system. Its participants hold state, exchange messages, and wait on one another. The old failure modes have followed the new vocabulary.

Tangle is a small Python library that catches this class of failure in real time for LangGraph workflows (and, via an HTTP sidecar or OpenTelemetry, for anything else). It reuses an idea that has been sitting in operating-systems textbooks since 1972 — the Wait-For Graph — and applies it at the agent layer, where the same topology has quietly reappeared.