anoman
Resep · Bot Code Review

Bot review PR GitHub.

Claude Sonnet 4.6 + tool calling + webhook GitHub. Mem-posting komentar inline dengan kategori severity. ~250 baris Python.

Arsitektur

Yang kita bangun

  1. GitHub mengirim webhook pull_request saat PR dibuka / disinkronkan.
  2. Handler FastAPI kita memverifikasi signature GitHub + memulai review di latar belakang.
  3. Loop review memanggil Claude Sonnet dengan diff PR + dua tool: read_file dan post_review_comment.
  4. Model berulang: membaca file, berpikir, mem-posting komentar, ulangi sampai puas (atau mencapai batas 15 langkah).
  5. Guardrail Anoman memeriksa setiap prompt + respons. Kebijakan tool memastikan bot hanya bisa memanggil dua tool yang kita deklarasikan.

1. Definisi tool

Apa yang bisa dilakukan bot

Dua tool. Skema dikirim ke model pada setiap panggilan. Set tool_choice: "auto" dan biarkan model yang memutuskan.

# tools.py — what the model can do
TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "read_file",
            "description": "Read a file from the PR being reviewed.",
            "parameters": {
                "type": "object",
                "properties": {
                    "path": {"type": "string", "description": "Path relative to repo root."},
                    "start_line": {"type": "integer"},
                    "end_line":   {"type": "integer"},
                },
                "required": ["path"],
            },
        },
    },
    {
        "type": "function",
        "function": {
            "name": "post_review_comment",
            "description": "Post an inline comment on a specific line of a file in the PR.",
            "parameters": {
                "type": "object",
                "properties": {
                    "path":     {"type": "string"},
                    "line":     {"type": "integer"},
                    "severity": {"type": "string", "enum": ["nit", "concern", "blocker"]},
                    "body":     {"type": "string"},
                },
                "required": ["path", "line", "severity", "body"],
            },
        },
    },
]

2. Loop review

Loop agent dengan batas langkah

Loop agent berbasis tool yang standar: terus memanggil sampai model mengembalikan teks tanpa tool call (atau kita mencapai batas langkah untuk membatasi biaya).

# review_bot.py — main agentic loop
import os
from openai import OpenAI
from tools import TOOLS
from impl import read_file_impl, post_review_comment_impl

client = OpenAI(
    base_url="https://api.anoman.io/v1",
    api_key=os.environ["ANOMAN_API_KEY"],
)

SYSTEM = """
You are a careful code reviewer. Given a PR diff, you can:
  - Read full file contents via the read_file tool
  - Post inline review comments via the post_review_comment tool

Rules:
  - Read the surrounding code before commenting on a change
  - Use severity "nit" for style, "concern" for likely bugs,
    "blocker" for security or correctness issues
  - Don't comment on whitespace or auto-formatter output
  - At most 10 comments per PR
"""

def review_pr(pr_diff: str, pr_metadata: dict) -> dict:
    """Run the review loop. Returns counts of comments posted by severity."""
    messages = [
        {"role": "system", "content": SYSTEM},
        {"role": "user",   "content": f"PR diff:\n\n{pr_diff}"},
    ]
    posted = {"nit": 0, "concern": 0, "blocker": 0}

    for step in range(15):  # cap loop length
        response = client.chat.completions.create(
            model="claude-sonnet-4-6",
            messages=messages,
            tools=TOOLS,
            tool_choice="auto",
            max_tokens=2000,
            metadata={"pr_id": pr_metadata["pr_id"]},
        )

        msg = response.choices[0].message

        # Bot replied with text but no tool — wrap up
        if not msg.tool_calls:
            print(msg.content)
            break

        # Execute every tool the model called this turn
        messages.append({"role": "assistant", "content": msg.content, "tool_calls": msg.tool_calls})
        for call in msg.tool_calls:
            name = call.function.name
            args = json.loads(call.function.arguments)
            if name == "read_file":
                result = read_file_impl(pr_metadata["repo"], args)
            elif name == "post_review_comment":
                result = post_review_comment_impl(pr_metadata, args)
                posted[args["severity"]] += 1
            else:
                result = {"error": f"unknown tool {name}"}

            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": json.dumps(result),
            })

    return posted

3. Entrypoint webhook GitHub

Jalankan bot dari webhook

# webhook.py — fire the bot when a PR is opened
from fastapi import FastAPI, Request, HTTPException, BackgroundTasks
import hmac, hashlib, os, json

from review_bot import review_pr
from impl import fetch_pr_diff

app = FastAPI()
GITHUB_SECRET = os.environ["GITHUB_WEBHOOK_SECRET"].encode()

@app.post("/webhook/github")
async def github_webhook(request: Request, bg: BackgroundTasks):
    raw = await request.body()
    sig = request.headers.get("x-hub-signature-256", "")
    expected = "sha256=" + hmac.new(GITHUB_SECRET, raw, hashlib.sha256).hexdigest()
    if not hmac.compare_digest(sig, expected):
        raise HTTPException(401, "invalid signature")

    event = request.headers.get("x-github-event")
    payload = json.loads(raw)

    if event == "pull_request" and payload["action"] in {"opened", "synchronize"}:
        pr = payload["pull_request"]
        bg.add_task(
            run_review,
            pr_diff=fetch_pr_diff(pr),
            pr_metadata={
                "pr_id":   pr["id"],
                "repo":    pr["base"]["repo"]["full_name"],
                "number":  pr["number"],
                "head_sha": pr["head"]["sha"],
            },
        )

    return {"ok": True}

def run_review(pr_diff, pr_metadata):
    try:
        counts = review_pr(pr_diff, pr_metadata)
        print(f"[review] PR #{pr_metadata['number']}: {counts}")
    except Exception as e:
        # Anoman returned 403 / 429 / 500. Don't crash the worker.
        print(f"[review] failed: {e}")

Pola signature yang sama dengan webhook milik Anoman sendiri — lihat /docs/webhooks untuk resep verifikasinya.

4. Pengerasan untuk produksi

Tangani blokir guardrail + rate limit

Jangan retry pada blokir guardrail (akan gagal lagi dengan cara yang sama). Retry pada 503/504 dengan backoff eksponensial. Berhenti cepat saat anggaran terlampaui.

# Wrap each completion in retry-with-backoff that respects
# Anoman-specific error codes.
from openai import APIStatusError, RateLimitError

def safe_completion(client, **kwargs):
    for attempt in range(5):
        try:
            return client.chat.completions.create(**kwargs)
        except RateLimitError as e:
            wait = float(e.response.headers.get("retry-after", 2 ** attempt))
            time.sleep(wait)
        except APIStatusError as e:
            err = e.body.get("error", {}) if isinstance(e.body, dict) else {}
            code = err.get("code", "")

            if code in {"prompt_injection", "tool_denied", "content_violation"}:
                # Guardrail block — don't retry. Log the trace + skip this PR.
                print(f"[guardrails] {code}: {err.get('message')}")
                raise

            if code == "budget_exceeded":
                # Out of money. Don't keep firing.
                print("[budget] monthly budget exhausted")
                raise

            # Provider error — retryable
            if e.status_code in {503, 504}:
                time.sleep(2 ** attempt)
                continue

            raise

    raise RuntimeError("max retries")

Deploy produksi

Tips dunia nyata

  • Anggaran per key — tetapkan batas USD bulanan pada key Anoman bot-mu. PR yang lepas kendali kena 402 alih-alih menguras dompetmu.
  • Policy group untuk tool — buat policy group khusus dengan read_file + post_review_comment di allowlist. Kini jika Anthropic sewaktu-waktu mengembalikan tool tak terduga, tool itu diblokir.
  • Tag Agent ID — kirim header anoman-agent-id: code-reviewer agar dashboard mengelompokkan semua panggilan di bawah satu agent.
  • Vision untuk screenshot — perluas untuk memanggil gpt-4o pada PR yang berisi lampiran gambar — berguna untuk PR design-system.

Resep lain untuk pola agentic.

Chat streaming, analisis dokumen batch, RAG dengan caching.