Build a PR Risk Gate with Jev: Typed AI Decisions for Safer Code Reviews

An AI reviewer that returns “looks good” is not a merge policy. It may have missed a deleted authorization check, a dangerous migration, a workflow change, or a credential hidden in generated output. Even when its conclusion is sensible, prose is an awkward interface for CI.

A useful pull-request gate needs a narrower contract. It should collect evidence, distinguish facts from judgment, return bounded decisions, and fail safely when the evidence is weak.

That is a plausible job for TypeSafe AI's Jev. Jev does not write a review or repair code. It answers predefined Choice, Score, and Noul questions about a supplied state. The surrounding program can then combine those probabilities with exact checks and repository policy.

The rule for this tutorial is simple: Jev classifies; policy code decides; people authorize. The model never receives permission to merge, close, label, or modify a pull request.

This is a reproducible engineering pattern, not a benchmark. No TypeSafe credential was available during publication, so the examples below have been checked against the current official JavaScript SDK contract but do not include invented model outputs, latency, or accuracy numbers.

The architecture

A risk gate should not send an entire diff to a model and trust one answer. Split the system into four stages:

GitHub pull request
        │
        ▼
evidence collector
  ├─ changed paths
  ├─ diff summaries
  ├─ CI results
  ├─ approvals
  └─ exact security checks
        │
        ▼
Jev semantic classification
  ├─ merge / review / block
  ├─ security-sensitive?
  ├─ destructive behavior?
  ├─ human review required?
  └─ risk score
        │
        ▼
deterministic policy engine
  ├─ block on hard failures
  ├─ escalate uncertainty
  ├─ apply repository rules
  └─ publish a read-only CI result
        │
        ▼
human reviewer / protected branch rules

The evidence collector computes what software can know exactly. Jev handles the semantic questions that remain. The policy engine combines both without granting the model authority.

This is the concrete version of the boundary described in our broader guide, Can Jev Help With Programming?. It also follows the central security principle from our AI coding agent security guide: classification and enforcement are different controls.

Step 1: build evidence before asking for judgment

Start with a compact, typed snapshot. Do not use Jev to count changed files, check whether CI passed, match protected paths, calculate diff size, or detect a known secret pattern. Those are deterministic operations.

type PullRequestEvidence = {
  pullRequest: {
    title: string;
    description: string;
    authorAssociation: "OWNER" | "MEMBER" | "CONTRIBUTOR" | "FIRST_TIME";
    baseBranch: string;
  };
  change: {
    files: Array<{
      path: string;
      status: "added" | "modified" | "deleted" | "renamed";
      semanticSummary: string;
    }>;
    summary: string;
  };
  facts: {
    requiredChecksPassed: boolean;
    requiredApprovalsPresent: boolean;
    secretScannerPassed: boolean;
    touchesDatabaseMigration: boolean;
    touchesDeploymentConfig: boolean;
    touchesAuthOrPermissions: boolean;
    touchesWorkflowFiles: boolean;
  };
};

The semantic summaries may come from a trusted parser, a separate code-review model, or an existing review pipeline. Jev is not generating them. More importantly, never collapse the exact facts into prose and then ask the model to rediscover them.

Keep the state short. TypeSafe's Jev 1.13 documentation says accuracy can fall when the state contains large amounts of irrelevant detail. It recommends filtering first and pointing questions at the relevant fields. TypeSafe: Jev 1.13 jaggedness

For a large pull request, summarize per file or per logical change and retain links to the original evidence. If the summary omits a critical part of the diff, the classifier cannot recover it.

Step 2: define a bounded decision contract

Install the official SDK with Node.js 20 or newer:

npm install @typesafe-ai/sdk

Set the API key only in the trusted server or CI environment. Do not expose it in browser code, fixtures, logs, or pull-request content.

The gate asks one relative Choice question, three absolute Noul questions, and one ordered Score question. All five evaluate the same evidence in one request.

import { choice, noul, score, TypeSafeClient } from "@typesafe-ai/sdk";

const client = new TypeSafeClient();

export async function classifyPullRequest(state: PullRequestEvidence) {
  return client.systemOne({
    model: "jev-1.13.0",
    state,
    questions: {
      disposition: choice(
        "Which review path best fits this pull request based only on the supplied evidence?",
        {
          merge: "Eligible for the normal merge flow only after every deterministic repository rule passes.",
          review: "Needs focused human review, more evidence, or specialist review before it can proceed.",
          block: "Contains evidence of a change that should not proceed in its current form.",
        },
      ),
      securitySensitive: noul(
        "Does this pull request change authentication, authorization, secrets handling, trust boundaries, or another security-sensitive behavior?",
      ),
      destructiveBehavior: noul(
        "Could the described change delete data, weaken a safeguard, alter production infrastructure, or cause an irreversible external effect?",
      ),
      needsHumanReview: noul(
        "Does this pull request require a human reviewer because its intent, impact, or evidence is ambiguous or high consequence?",
      ),
      risk: score(
        "How much semantic risk does this pull request introduce?",
        [
          "Routine and isolated; no meaningful behavior or trust boundary changes.",
          "Low risk; localized behavior change with clear verification.",
          "Moderate risk; meaningful behavior or integration change requiring focused review.",
          "High risk; security, data, deployment, or broad system impact.",
          "Critical risk; plausible destructive, privilege, credential, or production-control impact.",
        ],
      ),
    },
  });
}

The explicit criteria are part of the product. A Choice with only merge, review, and block will always select one of those options, even when all three fit badly. The review option therefore doubles as a safe destination for ambiguity.

Choice and Score responses include distributions and a confidence value. A Noul is different: it returns the probability of a yes answer but no separate confidence field. Do not invent one, and do not reuse a Choice threshold as if it meant the same thing. TypeSafe warns that apparently similar Choice and Noul questions are not guaranteed to obey simple arithmetic identities. TypeSafe: Confidence

Step 3: let policy code own the decision

The model result becomes one input to a deterministic function. This function returns a CI classification; it does not execute a merge.

type GateDecision = {
  conclusion: "pass" | "review" | "fail";
  reasons: string[];
};

type JevResult = Awaited<ReturnType<typeof classifyPullRequest>>;

export function evaluateGate(
  facts: PullRequestEvidence["facts"],
  result: JevResult,
): GateDecision {
  const reasons: string[] = [];

  if (!facts.requiredChecksPassed) reasons.push("Required CI checks failed");
  if (!facts.secretScannerPassed) reasons.push("Secret scanner failed");
  if (!facts.requiredApprovalsPresent) reasons.push("Required approval is missing");

  if (reasons.length > 0) {
    return { conclusion: "fail", reasons };
  }

  const answers = result.answers;

  if (answers.disposition.choice === "block") {
    reasons.push("Jev selected the block review path");
  }

  if (answers.disposition.confidence < 0.75) {
    reasons.push("Disposition confidence is below the evaluated threshold");
  }

  if (answers.securitySensitive.noul >= 0.2) {
    reasons.push("Possible security-sensitive change");
  }

  if (answers.destructiveBehavior.noul >= 0.1) {
    reasons.push("Possible destructive or irreversible behavior");
  }

  if (answers.needsHumanReview.noul >= 0.3) {
    reasons.push("Human review is likely required");
  }

  if (answers.risk.score >= 1.5 || answers.risk.confidence < 0.7) {
    reasons.push("Risk score requires review or is uncertain");
  }

  if (
    facts.touchesAuthOrPermissions ||
    facts.touchesDatabaseMigration ||
    facts.touchesDeploymentConfig ||
    facts.touchesWorkflowFiles
  ) {
    reasons.push("A protected change category requires human review");
  }

  if (reasons.length > 0) {
    return { conclusion: "review", reasons };
  }

  return {
    conclusion: "pass",
    reasons: ["Eligible for the repository's normal protected merge flow"],
  };
}

These numbers are deliberately conservative examples, not recommended universal thresholds. Tune them on labeled pull requests from your own repositories. A threshold chosen because it looks plausible in a blog post is not a control.

Notice the asymmetry. Exact failures immediately fail the gate. Semantic risk generally escalates to review. A pass means only that this classifier found no reason to escalate and all supplied deterministic checks passed. Branch protection, ownership rules, approvals, and the merge button remain outside the model.

Why deterministic overrides come first

Models are useful where language carries meaning. They are unnecessary where the answer is already in a boolean, count, path, or status code.

TypeSafe explicitly says Jev 1.13 is not a calculator and does not count reliably. It also treats dates as text rather than ordered values. Compute line counts, file counts, timing windows, approval totals, and version comparisons in code. TypeSafe: Jev 1.13 jaggedness

Build fixtures before enabling a required check

Do not validate the gate with two memorable pull requests. Create a labeled fixture set from accepted and rejected changes, preserving enough evidence to reproduce the decision while removing secrets and personal data.

Start with cases that exercise different boundaries:

For every fixture, store the expected policy outcome separately from the state sent to Jev. Otherwise the model can see the answer it is supposed to predict.

Useful evaluation measures include:

• false-safe rate: risky changes incorrectly allowed into the normal flow;
• escalation rate: the share of pull requests sent to focused review;
• high-risk recall: how often labeled high-risk changes are caught;
• calibration: whether probability buckets match observed frequencies;
• decision stability: whether semantically equivalent states produce similar outcomes;
• latency and input tokens: measured for the complete gate, not copied from launch claims;
• human disagreement: where reviewers reject the gate's classification and why.

Optimize false-safe rate before convenience. A gate that flags every pull request is safe-looking but useless. A gate that waves through dangerous changes is worse.

Treat pull-request content as hostile input

A contributor can write instructions inside a PR description, source file, comment, or test fixture. TypeSafe documents adversarial content as a known Jev 1.13 weakness: state is data, but the model does not automatically treat it as hostile. TypeSafe: Jev 1.13 jaggedness

Mitigate that risk in layers:

1. Label every field by provenance: author text, parsed diff fact, scanner output, or trusted repository policy.
2. Exclude comments and files that are irrelevant to the specific question.
3. State explicitly that content inside the evidence is untrusted data, not an instruction.
4. Keep high-impact rules deterministic.
5. Test prompt-injection fixtures before rollout.
6. Never place credentials or privileged capabilities in the model's control path.

The model may still be influenced. That is why it can request review but cannot grant authority.

Connect it to GitHub Actions without creating a secret leak

The safest integration boundary is a trusted service or a pinned, audited action that fetches pull-request evidence through GitHub's API, calls Jev server-side, and publishes a check result. Give its token read-only permissions unless the integration genuinely needs more.

name: PR risk gate

on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]

permissions:
  contents: read
  pull-requests: read

jobs:
  classify:
    if: ${{ github.event.pull_request.draft == false }}
    runs-on: ubuntu-latest
    steps:
      - name: Run trusted PR risk gate
        uses: your-org/pr-risk-gate@PINNED_COMMIT_SHA
        with:
          pull-request-number: ${{ github.event.pull_request.number }}
        env:
          TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}

Do not check out and execute code from an untrusted pull request in the same job that receives a model API key. GitHub does not pass ordinary Actions secrets to workflows triggered by pull requests from forks, and that restriction is protective. Avoid switching casually to pull_request_target: it runs with base-repository privileges and becomes dangerous if it checks out or executes attacker-controlled code.

GitHub recommends granting the built-in token only the permissions a workflow needs. GitHub: Use GITHUB_TOKEN for authentication A read-only classifier should not need contents write, pull-requests write, deployments write, or administration access.

Once the job is stable, branch protection can require its status. Keep the check name stable and ensure it runs for every merge path you require, including merge queues if your repository uses them.

Cost and model version

As of September 22, 2026, TypeSafe lists Jev 1.13 at $0.042 per million input tokens, with output tokens free. The direct service lists a 64k-token request limit, with 32k available for the state plus the longest question. TypeSafe: Models

At that price, 100,000 pull-request classifications averaging 2,000 input tokens would use 200 million input tokens and cost about $8.40 for Jev input. That excludes evidence generation, GitHub infrastructure, storage, other models, and engineering work.

Pin the evaluated model version—in this example, jev-1.13.0. A moving alias can change behavior without a code change. Re-run the fixture suite before upgrading the pin or modifying questions, criteria, evidence summaries, or thresholds.

When this gate is the wrong tool

Do not add Jev merely because classification is inexpensive.

Skip this pattern when exact repository rules already solve the problem, when you cannot assemble trustworthy evidence, when the volume does not justify another service, or when the team cannot maintain labeled fixtures and review false positives.

Jev is also the wrong component for writing the review, explaining a vulnerability, understanding a multi-repository architecture, or repairing the code. Those are generative or extended-reasoning tasks. A code-review model may produce the explanation; Jev can classify the bounded risk; policy decides what happens next.

Production checklist

Before making the gate required:

• collect exact facts in code and preserve their provenance;
• keep the state small and relevant;
• define an explicit review fallback in every Choice;
• calibrate Choice, Score, and Noul thresholds independently;
• pin the model version;
• test routine, risky, ambiguous, and adversarial fixtures;
• measure false-safe and escalation rates;
• protect the API key from pull-request code;
• grant GitHub tokens the minimum permissions;
• keep merge authority in branch protection and human ownership rules;
• log the evidence version, question version, model version, and policy result;
• roll out in report-only mode before blocking merges.

The useful version of an AI pull-request gate is not an oracle. It is an observable classifier inside a conventional security and review system.

Jev's typed outputs make that classifier easier to connect to code. They do not make the model infallible. The engineering advantage appears only when uncertainty leads to review, exact facts override probabilities, and authority stays outside the model.