Four Agents, No Ground Truth
Four Agents, No Ground Truth
I have been using LLMs to help me code for about a year now, and I am not going to pretend it didn’t help much. It feels like I 10×’ed.
Despite early appearances of taking my prompt and doing what I want automagically, LLMs often very much do not. If I have LLM code something from scratch, I often find myself hand-holding the chat to the end of time before what I had envisioned materalizes. However, because they are extremely good at connecting my half-formed intentions to already-existing solutions: plotting libraries, packaging tools, literate programming frameworks, website generators, notebook cleanup, GitHub workflows, when the LLM reaches for the right known tool instead of hard-coding a tiny castle in the sand, I can move very fast.
This has been especially powerful for the kind of work I do: scientific software, analysis notebooks, visualization, teaching materials, and and especially small web artifacts around research. (Claude and Codex built this entire website without me looking at them too much.) I am usually not trying to build a giant production system. I am trying to make something that helps a scientist think: a plot, a demo, a tutorial, a figure, a clearer explanation of an analysis. The whorlmap figure repo (https://github.com/sangyu/whorlmap-paper) was build with Claude Code (albeit with a lot of hand-holding, but that’s for another post). While amazed by the increase in productivity, I do also wonder if all this steering is necessary.
But the “human in the loop” part is not a slogan to me. It is the actual work.
LLMs are very good at producing plausible next steps. They are also very good at producing work that looks finished before it really is. The gap between “this ran” and “this looks exactly right” is still where the human scientist, designer, editor, and domain expert matters.
I find myself most needed in two places: design and goal-keeping.
By design, I mean both algorithmic design and graphical design. Does this abstraction actually fit the problem? Does the API expose the right conceptual object? Does the plot make the comparison easier, or is it merely decorative? Should this be a heatmap, a paired plot, a swarm plot, a bootstrap distribution, a state-space trajectory, or no plot at all? On many occasions LLMs want to code things from scratch, those typically don’t end well. Something to be said about human researchers’ and programmers’ often years of work going into packages and solutions.
These are not purely technical questions. They are human-facing questions. The final audience is a person: a reader, a reviewer, a student, a collaborator, a future version of myself. A human eye is not an optional aesthetic layer added after the “real” computation. It is part of the computation’s purpose.
The second place is goal-keeping. This sounds odd because at first LLMs appeared to me astonishingly good at goals. Sometimes I give a model a messy project folder and a loose objective, and it breaks the task down better than I would have. It writes checklists. It finds missing files. It reminds me that the thing needs documentation, examples, tests, and a README. It even fights with me to achieve my goals.
But over longer workflows, goals drift. Context gets compacted. A side issue becomes the main issue. A spurious input gets over-weighted. The model becomes attached to a premature framing. A coding agent satisfies the local instruction while damaging the global purpose. The work keeps moving, but not always toward the thing I actually care about.
So in practice, I steer a lot. No, don’t use that package. No, we need to consider one more thing. No, this figure is not legible; this other figure is. No, do not promote an implementation detail into the biological conclusion. No this description is not human friendly.
This works surprisingly well in a chat window. It does not scale very well.
It is also not very reproducible. A long chat with an LLM can contain a great deal of expertise, but much of it evaporates into the transcript. The next run does not necessarily inherit the judgment that was slowly negotiated in the previous run.
This is why I have become interested in spec-oriented development. I already use specs, memory files, skills, hooks, and persistent project documents. These add a more durable layer on top of the rather elusive context window. Instead of repeatedly explaining what a project is, I try to write down what must remain true: the protected claim, the central design, the main means of reaching the graphical output, the figure logic, the things the model may change, and the things it must not smooth away.
Making figure aside, I couldn’t help but wonder, can LLM’s generate scientific writing with sound logical reasoning? For one thing they seem really comfortable spitting out authoratative sentences. But how do you make sure the facts are factsing, the citations are real and the significance is not overclaimed? For scientific writing, I think there is another missing layer: reasoning rubrics. (To be clear, this is not about submitting “AI manuscripts” to traditional journals. My explorations of LLM-assisted scientific writing is at its current stage, an experiment.)
Scientific writing is often taught through repeated correction rather than explicit instruction. Senior scientists develop a strong intuitive sense for what makes a paper work. They can tell when the claim is disproportionate, when the Introduction is solving the wrong problem, when a figure is not carrying its argumentative weight, when a limitation is honest versus performatively anxious.
A lot of this judgment is tacit. You learn it by drafting, being corrected, revising, watching reviewers misunderstand something, watching reviewers correctly identify a weakness you hoped they would not notice, and slowly absorbing what counts as a strong scientific argument.
Journal guidelines do not really capture this. They tell you word limits, section formats, citation styles, figure requirements, reporting checklists, and data availability statements. These are useful, but they mostly tell authors how a paper should look. They rarely tell authors how a paper should think.
That is the object I want to make more explicit.
A reasoning rubric captures the deeper editorial and scientific judgments that experienced scientists apply all the time but rarely formalize. Does the manuscript give paragraph-level weight to major claims and sentence-level weight to minor caveats? Is the central claim scientifically defensible without becoming so cautious that it loses force? Does each figure prove a necessary point in the manuscript’s logic? Does the paper read like the kind of paper it is trying to be? What objection would a careful reviewer raise, and has the manuscript answered it in the right place?
These are not style rules. They are reasoning rules.
This is where I think LLM workflows become very interesting. The useful thing is not just that a model can draft prose or imitate a reviewer. The useful thing is that making the model useful forces the human expert to externalize judgment. It’s almost a form of transfer learning, from humant to machine. Every time I correct an LLM, I am doing a small act of tacit-to-explicit transfer. “This is too broad” can become a claim-safety rule. “This belongs in Methods, not the Introduction” can become an article-type-fit rule. “The figure does not yet prove the point” can become a figure-as-argument criterion. “This sounds impressive but is not actually the contribution” can become a novelty-framing repair pattern.
Over time, the workflow becomes more than a writing assistant. It becomes a structured record of my taste.
This is the idea behind Lanting, an experimental manuscript-writing loop I have started building.
The name comes from the Lantingji Xu (兰亭集序), Wang Xizhi (王羲之)’s 353 CE preface to a gathering of scholars composing poetry by a winding stream. The original no longer exists; every version we know is a copy. I love this as a metaphor for scientific manuscripts, and maybe for LLM-assisted writing in general. There is no pristine original hiding somewhere. There is no single ground-truth manuscript waiting to be uncovered. There are drafts, critiques, revisions, copies, distortions, corrections, and eventually, if we are lucky, the truest version we can make.
Lanting is currently a small experimental effort: an error-correcting loop for scientific prose. It is now compiled as an explicit state graph, so every step is a named node and every branch is an inspectable edge. A section is drafted, then attacked — first by cheap deterministic checks (protected claim present? banned phrases? citations that resolve to real references? invented technical terms?), then by four LLM judges reading their own rubrics: scientific integrity, article-type fit, adversarial review, and rhetorical hierarchy. The judges are deliberately walled off from one another and from the drafter’s context, so each one argues from its own evidence rather than agreeing with the room.
Their objections are not thrown away between passes. They are normalized into a persistent registry that survives across loops, so an objection a judge rephrases on the next pass is recognized as recurring rather than new, and one that disappears is marked resolved. The drafter then revises against exactly those objections — repair, not resampling; it is told to leave alone everything the critics did not name. The loop stops when the draft stabilizes across two consecutive clean passes, when a call budget runs out, or when a safety invariant fires — the protected claim vanishing, a fabricated term, a forbidden phrase — and it halts for a human instead of quietly writing around the problem.
The reason the plumbing is this elaborate is that Lanting is simultaneously a writing tool and the apparatus for an experiment about writing tools. The hypothesis is that a smaller, faster model inside a structured error-correcting loop can match or beat a larger model used in a single forward pass, at lower cost. So every LLM call is metered — tokens, latency, dollars, which node spent them. On the runs so far, about 85% of the spend is critique, not drafting. The writing is cheap; being sure the writing is right is where the money goes. That ratio is itself a finding, and I could only see it because the loop refuses to guess at a number it does not actually have.
The important part is not the automation itself. I do not want a machine that confidently produces mediocre papers while I sleep. The important part is the interface between human judgment and machine repetition.
For the first phase, I am targeting software application notes, because they are constrained enough to be tractable but still require real scientific taste. A software note has to explain what the tool does, who it is for, what gap it fills, and why the implementation matters without letting implementation details swallow the argument.
This is a working in progress. It is not a claim that LLMs can replace scientific mentorship. If anything, it is the opposite claim. The better the model gets, the more valuable the human steering layer becomes, because the failure shifts from “cannot produce anything” to “can produce many plausible things, only some of which are scientifically right.”
That is where I want to put myself: not outside the loop, and not merely rubber-stamping the loop, but designing the loop so that expert judgment becomes inspectable, reusable, and teachable.
The AI bosses may be rediscovering that humans are needed on the job. Good. Some of us have been here the whole time, squinting at the plots, demoting the overclaims, moving the caveats, and trying to teach the machine not just to write, but to reason like a careful scientist.
Lanting is my attempt to write that down.
Experimental repo: sangyu/lanting