I've designed and used quite a few eval harnesses by now. Here's what I keep coming back to: stronger models need less scaffolding. Pile on more constraints and you just throttle them. A simple circuit breaker does the job.
Weaker models need the opposite — more harness to keep them on track. But that raises the real cost question: you pick a cheap model to save money, it fails more, and then you wrap it in complex harness logic anyway. Per-task cost ends up not far from just using a strong model.
I've designed and used quite a few eval harnesses by now
More posts
I Built local-rag
Former company's knowledge base went down because Google changed the NotebookLM API. The local digital worker lost connection overnight.I Built local-rag to fix it. Lightweight local RAG, no server, no API keys.Stack: CLI → Chroma PersistentClient (local SQLite, no daemon) → embeddinggemma-300m (768d, runs on low-spec hardware)Install: "Give yourself this skill: haoyiyin/local-rag"Drop docs in, search by semantic meaning. 20+ formats including scanned PDFs, images (tesseract OCR), audio (whisper). Archives auto-extracted.
Google sent me the Workspace free AI trial email today
. Their main pitch is Nano Banana Pro, which honestly isn't nothing. Free users can also grab it in Flow — labs.google/fx/tools/flow — daily quota is actually generous. Enough for some quick productivity images without paying.
Claude appeal rejected.
Won't refund the subscription either. I'm half convinced the appeal email is itself phishing, because I'm pretty sure they log the IP. DHH came to Shenzhen recently and showed how he uses Claude Code over there. That's the safest way now, honestly: run Claude Code on an overseas machine, connect through Tailscale, never use it locally. So I need to rent an overseas box with a residential IP, and register and log into Claude there.
Sharing my current mobile only workflow.
Until late September it was the usual setup: parked in front of my laptop, terminal open, bossing pi around, fanning out subagents, then eyeballing diffs myself and filing the PR. Glued to the chair all day, every day. Late September I moved the whole thing to my phone, Hermes runs the show, pi takes the orders. My back is very grateful. Three weeks in, still no regrets. Stack: Hermes on flagship brains (gpt-6-astra / opus-5.5), pi on mid-tier (ds-v4.1-flash / sonnet / terra) with as few plugins as possible, the two talking over a2a. Hermes reads the issues, slices them into small tasks, spins up a worktree for each one, and dispatches a pi into it. When a pi finishes it pings Hermes, who decides: ship the PR or bounce it back for rework. I get back from a full day of being a tourist, or just wake up, and a dozen PRs are sitting there done. Pretty sweet. The catch is how black-box the whole thing is. My emotional arc has been a rerun of early vibe coding, except back then I at least read the diffs. Now even that goes to a flagship model. Boxes all the way down. If you don't have data anxiety, grok-bot and chatgpt-dot basically ship this out of the box, grok-cli and codex built in, native agent-to-agent comms. Downside: your code lives on their servers, everything's moderated, and one ban takes it all with it.
