Steering the Vibe: Review

This is the fourth post in a series about steering AI code assist (code-assist) towards producing maintainable code. This particular post will look at a workflow outside the core loop of generating code and gives insight into how the tools we’re already using can be leveraged to improve more than just writing code. Specifically this post will explore improved pull request (PR) processes.

Claude Code PR Review

Out of the box claude-code installs the code-review plugin and aliases it to /review. You can use this to review an open PR by passing it the number, for example:

/review 123

The plugin uses gh to fetch the PR details and you can see the definition of the command that enables this here.

GitHub Copilot PR Review

GitHub Copilot also has a PR review feature and you might be intimately familiar with it if your repo has it enabled automatically. When I first started interacting with it, some months ago, I found the output noisy and often wrong. However, over time the feature has improved and I find that its success rate is now much higher - typically over 50%, and occasionally all comments are actionable.

This however has changed my PR workflow - as well as getting review comments from colleagues, there are now ~2-5 wordy comments supplied by Copilot. When stepping through many PRs a day, there’s now often double the comments I’d usually have to read through, and so I find a lot more time is spent specifically reviewing whether the Copilot comments were correct or not. I still want to look at them though because at least some of them do point to changes I should make. So, I improved my code-assist workflow to help with this.

Review PR Comments Command

I added a /review-comments command to my code-assist workflow automation tool, assist, and calling it from Claude Code while on a PR branch will trigger the workflow. The workflow is not entirely trivial. To review comments on a PR, you need to:

Check for pr-level comments, and for each comment:
- Make a decision whether a change/commit is required
Check for code-level comments, and for each comment:
- Make a decision whether a change/commit is required and;
- (Optional) Post a comment with either the fix or why you didn’t want to apply it and;
- Resolve the comment thread (required if the branch ruleset requires all comments to be resolved before merging)

A bunch of the PR data can be obtained via standard gh pr commands, but some of the data is only available via the graphql API and this includes mutations like posting a comment or resolving a thread.

# resolve thread 123
gh api graphql -F query="mutation($threadId: ID!) { resolveReviewThread(input: {threadId: $threadId}) { thread { isResolved } } }" -f threadId="123"

Now this is non-trivial enough to raise a few problems immediately:

Putting this workflow in a skill as natural language and hoping the tool will construct the requests correctly is questionable and error prone
Some of the requests are write operations (on my behalf), and I don’t really want any accidental writes to any part of the GraphQL API
The commands to read pr-level or comment-level comments are different, and we need a bunch of different data (like threadId) to complete the workflow

Enforcing determinism and permissions

To solve this, we take the complicated but codifyable parts out of the hands of the LLM. This is achieved by implementing commands in assist:

assist prs list-comments
- Gets all the comments on a PR, their status, their type, other relevant data, and saves it to disk for reference within the workflow
assist prs fixed <comment-id> <sha>
- Verifies that the sha exists and;
- Posts Fixed in <sha> to the comment thread and;
- Resolves the thread
assist prs wontfix <comment-id> <reason>
- Posts the reason for not fixing to the thread and;
- Resolves the thread

Now the LLM doesn’t need to know how or which of the many gh pr view and gh api commands to call and instead need only call the three commands above. The syntax for the above commands is also simple enough that it works with claude-code permissions. I can specify list-comments and fixed commands to be allowed as these do not include any free-form input. Leaving the wontfix command out of the allowed commands means I get to see the text that will be posted and can make a decision to let it go or alter it.

{
  "permissions": {
    "allow": ["Bash(assist prs list-comments:*)", "Bash(assist prs fixed:*)"]
  }
}

Improved workflow

Now the number of steps that I need to perform is reduced. For each unresolved comment on the PR, I’m shown a summary and a recommendation where I either choose the recommendation, or if I disagree choose the other.

In practice this has reduced the time I spend reading through both Copilot (or real human!) comments and even in cases where I disagree with the recommendation, I still find the information surfaced in this workflow to result in a faster, more confident decision.