Build an AI Code Review Agent for GitHub

This is part of the AI Automation Engineer Roadmap series.

TL;DR

An AI code review agent automates first-pass pull request review by listening for GitHub events, fetching diffs, evaluating changed files with an LLM, and posting actionable inline comments back to the PR. The right architecture uses GitHub webhooks for triggers, a diff parser for chunking, deterministic prompts with coding standards context, and guardrails that prevent noisy or low-confidence comments.

Why This Matters

Code review is one of the highest-leverage places to apply AI in software teams because it sits directly on the path to production. Every pull request already has structure: a diff, changed files, author metadata, test signals, and a review workflow. That makes it a much better automation target than open-ended tasks with fuzzy success criteria.

A good AI code review agent does not replace human reviewers. It handles the repetitive first pass:

• ›obvious bugs
• ›missing null checks
• ›error-handling gaps
• ›security footguns
• ›naming inconsistencies
• ›violations of team conventions

That gives human reviewers more time to focus on architecture, trade-offs, domain logic, and product impact.

The important distinction is this: a useful review agent is not "an LLM that reads a diff." It is a pipeline that prepares the right context, scopes the review correctly, and only comments when confidence is high enough to justify interrupting a developer.

Core Concepts

What an AI Code Review Agent Actually Does

At a high level, the agent follows this flow:

1. ›GitHub emits a webhook when a pull request is opened, synchronized, or reopened.
2. ›Your service validates the webhook signature and fetches the PR diff.
3. ›The diff is split by file and optionally by hunk for large changes.
4. ›Each unit is sent to an LLM with coding standards, repository context, and review instructions.
5. ›The model returns structured findings with severity, rationale, and suggested fixes.
6. ›Your service filters low-value findings and posts the rest back to GitHub as inline comments or a summary review.

That pipeline matters because the quality of the review depends less on "what model is best" and more on how well you package the review task.

Review Scope: Full File vs Diff-Only

One of the first design choices is whether to review only the diff or review the full file with diff context.

Diff-only review is cheaper and faster, but it can miss issues caused by surrounding code. A null check might look unnecessary in the diff and required in the full file. A refactor can break a call site that the diff alone does not explain.

Full-file review with highlighted diff context is generally better. The model can reason about imports, helper functions, existing patterns, and consistency within the file. The trade-off is more tokens and slower review time.

For most teams, the pragmatic approach is:

• ›use diff-only for very small changes
• ›use full-file review for modified source files
• ›skip generated files, lockfiles, snapshots, and binaries

Not All Findings Deserve a Comment

The fastest way to make developers hate your agent is to make it noisy.

Your agent should avoid commenting on:

• ›formatting that Prettier or ESLint already handles
• ›speculative style opinions
• ›low-confidence "maybe this is wrong" guesses
• ›comments without actionable fixes

Instead, focus on issues like:

• ›correctness
• ›security
• ›performance regressions
• ›missing validation
• ›missing error handling
• ›test gaps
• ›violations of explicit team rules

The bar should be: "Would a strong senior reviewer be comfortable leaving this comment?"

Architecture

Recommended System Design

For a production-grade code review agent, use four logical components:

1. ›

   GitHub webhook handler

   • ›verifies webhook signatures
   • ›filters relevant PR events
   • ›creates a review job
2. ›

   Diff and file context collector

   • ›fetches changed files
   • ›ignores unsupported file types
   • ›gathers full file contents where useful
   • ›chunks oversized files
3. ›

   LLM review engine

   • ›applies prompt templates
   • ›injects coding standards and repository policies
   • ›requests structured JSON output
4. ›

   Review publisher

   • ›deduplicates comments
   • ›maps findings to specific lines when possible
   • ›posts inline comments or a summary review back to GitHub

This separation matters because each component has different failure modes. Webhook verification failures are security issues. Diff parsing failures are ingestion issues. Model hallucinations are evaluation issues. Comment publishing failures are GitHub API issues.

Hands-On Implementation

Step 1: Listen for Pull Request Webhooks

Start with a minimal webhook endpoint:

// app/api/github/webhook/route.ts
import { NextRequest } from "next/server";
import crypto from "node:crypto";
 
function verifySignature(body: string, signature: string | null, secret: string) {
  if (!signature) return false;
 
  const expected = `sha256=${crypto
    .createHmac("sha256", secret)
    .update(body)
    .digest("hex")}`;
 
  return crypto.timingSafeEqual(
    Buffer.from(signature),
    Buffer.from(expected),
  );
}
 
export async function POST(req: NextRequest) {
  const body = await req.text();
  const signature = req.headers.get("x-hub-signature-256");
 
  const isValid = verifySignature(
    body,
    signature,
    process.env.GITHUB_WEBHOOK_SECRET!,
  );
 
  if (!isValid) {
    return new Response("Invalid signature", { status: 401 });
  }
 
  const event = req.headers.get("x-github-event");
  const payload = JSON.parse(body);
 
  if (event !== "pull_request") {
    return Response.json({ ignored: true });
  }
 
  const action = payload.action;
  if (!["opened", "synchronize", "reopened"].includes(action)) {
    return Response.json({ ignored: true });
  }
 
  // Queue a review job here
  return Response.json({
    accepted: true,
    pullRequest: payload.pull_request.number,
  });
}

Do not perform the full review inside the webhook request path. Queue the work and return quickly. GitHub expects a fast response, and LLM review latency can easily exceed safe webhook timing windows.

Step 2: Fetch and Filter Changed Files

Use the GitHub API to fetch PR files and immediately filter noise:

interface PullRequestFile {
  filename: string;
  status: string;
  patch?: string;
  raw_url: string;
}
 
const IGNORED_PATTERNS = [
  /\.lock$/,
  /^package-lock\.json$/,
  /^pnpm-lock\.yaml$/,
  /\.snap$/,
  /^dist\//,
  /^build\//,
  /\.min\.js$/,
];
 
function shouldReviewFile(filename: string) {
  return !IGNORED_PATTERNS.some((pattern) => pattern.test(filename));
}
 
async function getReviewableFiles(files: PullRequestFile[]) {
  return files.filter(
    (file) =>
      shouldReviewFile(file.filename) &&
      (file.filename.endsWith(".ts") ||
        file.filename.endsWith(".tsx") ||
        file.filename.endsWith(".js") ||
        file.filename.endsWith(".jsx")),
  );
}

This is not a trivial optimization. If you send lockfiles, generated bundles, or snapshots to the model, your reviews get slower, more expensive, and less accurate.

Step 3: Ask the Model for Structured Output

Avoid free-form review prose. Ask for structured JSON:

import { z } from "zod";
 
const ReviewFindingSchema = z.object({
  file: z.string(),
  line: z.number().optional(),
  severity: z.enum(["high", "medium", "low"]),
  category: z.enum([
    "bug",
    "security",
    "performance",
    "maintainability",
    "testing",
  ]),
  title: z.string(),
  explanation: z.string(),
  suggestion: z.string(),
  confidence: z.number().min(0).max(1),
});
 
const ReviewResponseSchema = z.object({
  summary: z.string(),
  findings: z.array(ReviewFindingSchema),
});

And a prompt like:

You are a senior software engineer performing a pull request review.
 
Review the changed code for:
- correctness bugs
- security issues
- performance regressions
- missing validation or error handling
- missing tests
 
Do NOT comment on formatting or subjective style preferences.
Do NOT invent problems without clear evidence.
Only include findings that are actionable.
 
Return JSON with:
- summary
- findings[]
 
Repository standards:
{codingStandards}
 
Changed file:
{filename}
 
Patch:
{patch}
 
Full file context:
{fullFile}

This is the difference between "the model said some things" and "the model produced a machine-usable review artifact."

Step 4: Filter Before Posting Comments

Never publish raw model output directly. Add a post-processing layer:

function filterFindings(findings: z.infer<typeof ReviewFindingSchema>[]) {
  return findings.filter((finding) => {
    if (finding.confidence < 0.75) return false;
    if (finding.severity === "low") return false;
    if (!finding.suggestion?.trim()) return false;
    return true;
  });
}

You can also collapse duplicate findings across adjacent hunks and downgrade comments that are better placed in a top-level summary instead of inline review annotations.

Step 5: Post Review Comments Back to GitHub

Once findings are filtered, map them into GitHub review comments:

async function createReviewComment({
  owner,
  repo,
  pullNumber,
  commitId,
  path,
  line,
  body,
}: {
  owner: string;
  repo: string;
  pullNumber: number;
  commitId: string;
  path: string;
  line: number;
  body: string;
}) {
  await fetch(
    `https://api.github.com/repos/${owner}/${repo}/pulls/${pullNumber}/comments`,
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${process.env.GITHUB_TOKEN}`,
        Accept: "application/vnd.github+json",
        "Content-Type": "application/json",
      },
      body: JSON.stringify({
        body,
        commit_id: commitId,
        path,
        line,
      }),
    },
  );
}

A useful comment template is:

Potential bug: missing null handling when `result.data` is undefined.
 
Why it matters:
This path can throw at runtime if the API returns an empty response.
 
Suggested fix:
Guard the access before reading nested properties and return a safe fallback.

That format is concise, defensible, and actionable.

Production Considerations

Rate Limits and Cost Control

Reviewing every file with a premium model can get expensive quickly. Practical controls:

• ›skip files above a token threshold
• ›use a smaller model for low-risk files
• ›reserve stronger models for large or security-sensitive diffs
• ›cap review frequency on repeated force-pushes
• ›cache unchanged file reviews when rebasing or updating branches

The right system is cost-aware, not just model-aware.

Repository-Specific Standards

Generic review comments are weaker than repository-aware review comments.

Add context like:

• ›framework conventions
• ›error-handling patterns
• ›testing expectations
• ›naming rules
• ›security boundaries
• ›package architecture rules

For example, if your repo requires:

• ›Zod validation on all external input
• ›no raw SQL outside data-access modules
• ›structured logging in API handlers

then the prompt should say so explicitly.

Security Guardrails

Be careful with untrusted pull requests, especially in public repos.

Important protections:

• ›never execute PR code during review unless isolated
• ›never expose privileged tokens to untrusted workflows
• ›separate read-only review from CI secrets
• ›sanitize prompt inputs if PRs can contain prompt-injection content

Yes, prompt injection applies here too. A malicious contributor can literally place instructions inside a diff that try to manipulate the reviewing agent.

Confidence and Escalation

A good agent knows when not to comment.

Use confidence thresholds and escalation rules like:

• ›high-confidence correctness issue: inline comment
• ›medium-confidence concern: include in summary only
• ›uncertain issue: suppress

That preserves trust in the system over time.

Common Pitfalls

Reviewing Generated or Vendor Files

This wastes tokens and generates junk comments.

Asking for "all issues"

That prompt shape encourages hallucination. Ask for high-value findings only.

No Structured Output

Without schemas, your downstream pipeline becomes brittle and harder to evaluate.

No Evaluation Loop

You should sample reviews and manually evaluate:

• ›precision
• ›false positives
• ›missed issues
• ›comment usefulness

If you do not measure review quality, you cannot improve it.

A Better Incremental Rollout

Do not start by posting inline comments on every PR.

Use this rollout:

1. ›

   Shadow mode

   • ›generate reviews internally
   • ›do not post publicly
   • ›compare against human reviews
2. ›

   Summary-only mode

   • ›post one top-level review summary
   • ›no inline comments yet
3. ›

   Inline mode for high-confidence findings

   • ›only correctness/security/performance issues
4. ›

   Repository-wide adoption

   • ›after precision is acceptable

This rollout avoids the reputational damage of shipping a noisy reviewer too early.

Final Recommendations

If you are building your first AI code review agent, optimize for precision and trust, not maximum comment count. A reviewer that catches one real bug every three pull requests is useful. A reviewer that posts five weak comments on every PR gets ignored immediately.

The winning design is usually:

• ›webhook-triggered
• ›queued asynchronously
• ›file-filtered
• ›full-file aware
• ›structured-output driven
• ›confidence-gated
• ›repository-policy aware

That is what turns an LLM demo into a production automation.

Next Steps

Once you have basic code review working, the next useful upgrades are:

• ›test gap detection
• ›security-specific review mode
• ›architectural policy checks
• ›repository memory for common patterns
• ›feedback loops from dismissed vs accepted comments

Those features let the agent adapt from a generic reviewer into a team-specific engineering assistant.

If you are following this AI automation roadmap, the next step after building a code review agent is to think about how agents coordinate across multiple tools and systems. That is where multi-tool agents with MCP become relevant.