TL;DR — Once agents raised my throughput, the bottleneck stopped being writing code and became review: more pull requests in flight, more comments arriving, and every one of them pulling me out of deep work. The fix wasn’t reviewing faster. It was a skill built on one rule and one ordering. The rule: a review comment is a hypothesis until you have reproduced it against the code at the pull request’s head. The ordering: machine reviewers first, human attention last. The part that took longest to learn is where the loop ends — not when the bot goes quiet.
The bottleneck moved, and it moved somewhere worse
Removing one bottleneck exposes the next. Writing the code stopped being the slow part, so more work reached review — more pull requests open at once, and longer ones. Review, in both directions, became the constraint.
What made it worse than the constraint it replaced is the shape of the interruption. Writing code is work you schedule. Review comments arrive on someone else’s clock, one at a time, across several pull requests you have already mentally closed. Each one is small; each one costs a context switch back into code you finished on Tuesday. I was being yanked out of deep work by notifications, not by difficulty.
So the goal was never “handle comments faster”. It was to stop handling them one interrupt at a time.
A comment is a hypothesis, not an instruction
The rule the whole workflow rests on: an automated review comment is a claim to be tested, not a task to be done.
Bot reviewers raise frequent false positives and often flag their own low confidence. Treating each comment as an instruction means changing working code to satisfy a machine that was guessing — which is worse than ignoring it, because now the change needs review too. So a comment counts only once you have reproduced it in the real code path, at the pull request’s head commit — not against the main branch, because the reviewer was looking at the head.
The symmetry matters as much as the rule: a comment is never dismissed on vibes either. “Invalid” has to cite the guard, the test, or the framework default that makes the concern moot. Both verdicts carry evidence, and both carry a permalink back to the original comment, so the verdict can be checked rather than trusted.
One round made the point better than any argument could. Nine comments on a single pull request: four were real and I fixed them, and the other five were the same comment five times — every shell script in the change was missing set -e, so a failing command would be ignored and the run would carry on with junk.
Reading the scripts’ own control flow inverted it. Each script creates test records, runs its checks, then deletes those records at the end — and nothing runs that cleanup if the script exits early. Under set -e the first failing command would abort before the cleanup and leave those records behind in a shared environment, which is the exact harm the comment wanted to prevent. I also wrote the checks to run all the way through and report at the end, which set -e would cut short at the first red. Five rejections, carried one at a time as the rule below demands, each citing the same evidence — plus a note naming the fix that would actually help, which is a cleanup step that runs on exit, not set -e.
The tempting move was to treat five identical comments as five signals. They were one: five copies of a hypothesis is still a hypothesis, and repetition is not corroboration.
That single rule converts the queue from a list of chores into a list of claims — and claims can be batched, because they no longer each demand a decision from me at the moment they arrive.
One at a time, and never a wall
The obvious way to batch is to dump every finding at once. It doesn’t work: a wall of parallel verdicts is unreadable, and a human can only act on one decision at a time anyway.
So the skill carries each comment all the way through — verify, propose the fix, get approval, reply, resolve — before it looks at the next. The only exception is when several comments are genuinely trivial and share one root cause, and then the first line has to say why they’re grouped. The skill never groups anything needing real judgment, however small it looks.
Every change is proposed before it is committed: the agent can’t verify that its fix is what I intended, so commits, pushes, and tracker updates wait for an explicit approval. And each proposal leads with a recommendation rather than a bare menu — the agent did the verification, so it owes me a judgment, not a shrug. Any fix that touches code arrives with its test, through the skill from the last post — a review comment is not an excuse to skip the discipline.
The skill is public: github.com/rdok/address-pr-comments-skill — the workflow above, phase by phase, with the exact gh and GraphQL calls behind it. The organisation and checkout paths are the only things you configure.
The ordering is what makes batching land
The refinement that finished the loop was deciding who goes first.
The skill runs the machine reviewers first and asks for my attention last — once it has proven the queue out and stripped the noise. That inverts who does the triage: by default every bot comment interrupts a human, who then sorts signal from noise one notification at a time. Now the agent does that sorting against the actual code, and I take the decisions in one pass, on my schedule.
Worth being honest about what that does and doesn’t change: I still decide nearly every comment individually, so the number of decisions barely moves. What changed is who picks the moment. That’s the quality-of-life win, and it’s bigger than the time saved — review feedback stopped being an interrupt.
Knowing when to stop is a judgment, not a rule
Here is the part no documentation tells you: the bar can never be “iterate until the bot reports zero”.
A bot reviewer with a real false-positive rate will always surface one more nit. Chase zero and you loop forever on decreasing value, and — worse — you start making changes to satisfy noise.
So the end of each round is an explicit judgment about whether another machine pass is worth it, argued from what this round actually found:
- Another round is worth it when the round produced a real, reproduced bug fixed with a behavioural change — that fix is new code the first pass never saw.
- It isn’t worth it when the round produced only nits, rejections, or cosmetic edits — nothing behavioural changed, so there’s little new to catch. Same when the reviewer starts flip-flopping, re-flagging code it accepted earlier: that’s noise, not signal.
Rule of thumb: one re-review after a batch of substantive fixes; a second only if the first found another reproduced bug; by the third you’re in nit territory. Stop and ship.
Those numbers are mine, not a law — they come from watching which rounds actually caught anything in the repositories I work in, and a noisier reviewer or a riskier change would move them. The part I’d defend anywhere is the shape: decide from what this round found.
The bot’s silence is not done
The failure this workflow exists to prevent is subtle enough that it took a while to see: “the bot found nothing more” reads exactly like done, and it is not.
A pull request can clear every automated round, satisfy every nit, reach acceptance testing — and never have been read by a second human. The bot’s quiet feels like a green light because it arrives in the same place a green light would.
So the moment the machine loop ends, the very next question is the human one: has anyone actually reviewed this? If no human review exists and none is pending, the skill asks — right then, at the moment the false sense of completion appears — whether to request one. It suggests a name only when the repository answers for it, through CODEOWNERS or GitHub’s list of people who recently touched those files; otherwise it asks who, because guessing at people is not its job. Requesting the review is mechanical, so the agent does that part; the only thing needed from me is a yes.
Done is a human approval plus green tests. Never a bot’s silence.
The cost of skipping that question isn’t theoretical. On an earlier round I accepted a comment about a job that deletes archived records — correctly, it was a real bug. Then I applied its suggested remedy as worded. The old gate deleted a record unless one particular status said otherwise; the remedy inverted that, so a record was deleted only when every attached status was one the comment had named as safe. Tests covered exactly those statuses, and everything went green.
Then a human reviewer asked about a status the comment never mentioned: rows recording an abandoned attempt, written whenever someone starts an operation and never finishes it. Nothing ever clears them. Under the new gate, one abandoned attempt would have blocked its record from being deleted for good — and abandoned attempts are common by construction. The remedy had made things worse than the bug it fixed.
Accepting a bug report is not accepting its remedy — the remedy is a second hypothesis, and it needs verifying like the first. The uncomfortable part, given that the last post argued tests are the guardrail: tests written from the reviewer’s framing inherit its blind spot and then launder it as confidence. Green proved the gate did what the comment described. It could not show that what the comment described was the wrong rule — which narrows the last post’s claim, rather than contradicting it. Tests carry correctness against the rule you hand them.
The human-approval rule assumes a second human exists. On a solo repository there isn’t one, and branch protection doesn’t substitute — it enforces that someone approved, and on a solo repository that someone is you. What actually caught the deletion gate wasn’t a second person so much as a second framing: a reviewer who had never read the original comment, and so wasn’t standing inside its assumptions. That part is reproducible alone — a fresh pass, deliberately made without re-reading the thread that produced the fix. It is weaker than another pair of eyes, and worth naming as weaker rather than pretending the gate is intact.
Trade notes
The pattern underneath this one generalises past code review: every bottleneck an agent removes has a next bottleneck behind it. And the honest question at the end of any automation is not “did the machine finish?” but “what does finished actually mean here, and who says so?” In this case the answer was a person, and the workflow’s job was to make sure that question got asked at the exact moment it was easiest to skip.
If you’ve built something similar — especially if you’ve found a better stopping rule than the one in this skill — I’d like to hear it. The comments are open.