Accelerating Promotion Boards with Agents

In the Canadian military, we convene promotion boards every year to decide who moves up.

Two years ago, I volunteered for one. Yes: an engineering officer from the Navy (with no policing background whatsoever, mind you) helped review over 400 military police files for promotion candidacy.

The importance of the task is not lost on me. But indulge me for a second: it was a little excruciating, the administrative equivalent of moving sandbags. So here’s something nobody asked for:

This is how I’d apply artificial intelligence to speed up promotion boards in the CAF.

I propose LLM-piloted agents for three tasks:

1. Data Extraction: converting unstructured data (PARs and MPRRs) into structured data;
2. Validation: checking that data against an existing specification, the SCRIT; and
3. Adversarial Review: simulating the board format before hand-off to human reviewers, who either concur or thug it out manually.

Now, don’t get twisted: promotion boards should remain, in my opinion, a human endeavour. The consequences of taking humans out of the loop are too severe.

So why bother? Read on.

---

Here Be Dragons: this thought-piece was written by a serving member. In a heroic act of empathy, an acronyms section follows.

Acronyms

CAF

Canadian Armed Forces

Capt

Captain

DNMLE

Did Not Meet Level Expected (PAR rating)

ELE

Exceeded Level Expected (PAR rating)

FELE

Far Exceeded Level Expected (PAR rating)

LCdr

Lieutenant-Commander

LLM

Large language model

Maj

Major

MITE

Military Individual Training and Education

MLE

Met Level Expected (PAR rating)

MPRR

Member's Personnel Record Résumé

MS ENG

Marine Systems Engineering

MWO

Master Warrant Officer

NAM

Non-affiliated member

NCS ENG

Naval Combat Systems Engineering

OCR

Optical character recognition

PAR

Performance Appraisal Report

PMLE

Partially Met Level Expected (PAR rating)

PY

Promotion year

SCRIT

Selection criteria (board scoring specification)

Sgt

Sergeant

How boards work

For each trade, a board is convened: a chair, three members, and a non-affiliated member (NAM), a board member from outside the trade who is present to mitigate groupthink.

1. 01: Files

   Board members are presented all eligible files for the trade, plus the SCRIT.

2. 02: Scoring

   Every board member scores each file

3. x̄
   03: Compare

   Scores are discussed, and a file's final result is the arithmetic mean.

4. 04: Top K

   Files are ranked. The top K — a forecasted number — are offered promotion.

Each eligible file is checked against a SCRIT: a tight specification that leaves, desirably, little room for subjectivity. Members score each file in silence, comparing answers at intervals. There’s room for debate, but the final tally is the arithmetic mean of all scores. This runs for about five calendar days, with breaks for meals and sanity.

I am not sure if it’s a rule per se, but there are also no windows.

Room for improvement

The first issue is time. Mapping a file onto each criterion, while trivial, is tedious and slow. The review (the discussion and consensus between members) is not (…he hints, foreshadowingly).

The second issue is, in a way, also time. Board members are volunteers whose day jobs languish in the interim.

The third is financial commentary. Foot-stomping the fact that this process is absolutely necessary, let’s look at some figures. My board (Cpl → Sgt) consisted of one Maj, one Capt (yours truly), and three MWO.

Cost of one promotion board

Rank	Members	Annual	Daily	5-day cost
Maj GSO · PI 1	1	$143,796	$553	$2,765
Capt GSO · PI 1	1	$106,332	$409	$2,045
MWO Standard · PI 1	3	$116,016	$446	$6,693
Total	5		$2,301	$11,503

Maj $2,765

Capt $2,045

3 × MWO $6,693

That’s about $CAD 11K in pay-hours¹ to fully process one trade’s junior ranks. Boards for superior ranks take less than the full week, since personnel dwindle as you climb the pyramid, so treat 11K as a floor for a trade, and one that says nothing of the Doctors, Lawyers, and Pilots who also convene these boards.

There are over 100 trades in the CAF, and this happens annually. Financially, this adds up.

So, how could we improve?

Rationale

Why a language model?

Well, for starters, the inputs are documents written for people. A PAR has narratives written for a human reader, in whatever words the supervisor chose. The MPRR is a bit tidier (a printout of postings, courses, and qualifications), but “geographic move, in rank” still has to be worked out from a list of dates and unit names.

Every fact the SCRIT asks about is in one or the other; there is no field called leadership_at_sea: true.

Language models were trained on exactly this: read a passage, answer a question about it. “Does this narrative describe a head-of-department posting at sea?” and “how many times has this member moved since promotion?” are both reading comprehension, one from the PAR and one from the MPRR.

Even better, nothing upstream needs to change. Supervisors write PARs as they always have, the SCRIT stays a spreadsheet, and nobody fills out a new form.

Extraction

Below, I define a Qualification, and an agent extracts a list of them from arbitrary text (here, an email). The class is the contract; the model’s only job is to fill it.

from dataclasses import dataclass

from pydantic_ai import Agent


@dataclass
class Qualification:
    mite_code: str
    description: str


example_email = """
Good morning Captain,

Congratulations. You've been granted two quals; the Doofus qualification
(MITE Code: DOOF), and the Nerd qual (code: NERD).

Later, dweeb."""

agent = Agent(
    "cohere:command-a-plus-05-2025",
    output_type=list[Qualification],
    instructions="Extract all qualifications from the text.",
)

print(agent.run_sync(example_email).output)
#> [Qualification(mite_code='DOOF', description='Doofus qualification'),
#>  Qualification(mite_code='NERD', description='Nerd qual')]

Point this at an MPRR instead of a snarky email and you have structured personnel data. And since current models accept PDFs directly, the PARs need no conversion either: no OCR pipeline, no exquisite software with recurring licenses.

Validation

A SCRIT is a specification, issued annually, that happens to live in a spreadsheet. Here is a condensed one from my own trade, Naval Engineering (Board #17, promotions to LCdr):

Every line has rules attached. An ELE PAR is worth 13 to 15 points, never 16. PARs from a lower rank count for exactly half. These are the rules a board member holds in their head for a week; a model can hold them in its context for a few seconds per file.

Now, this is a toy example, but we don’t need to translate the SCRIT into code. The agent reads the same spreadsheet the board reads, takes the PAR PDFs as-is.

from pathlib import Path
from typing import Literal

import pandas
from pydantic import BaseModel
from pydantic_ai import Agent, BinaryContent, ModelRetry, RunContext


class ParScore(BaseModel):
    rating: Literal['FELE', 'ELE', 'MLE', 'PMLE', 'DNMLE']
    points: int
    evidence: str


class ScoredFile(BaseModel):
    service_number: str
    pars: list[ParScore]


scorer = Agent(
    'cohere:command-a-plus-05-2026',
    deps_type=str,  # the PARs' full text, for the check below
    output_type=ScoredFile,
    instructions='Score the member against the SCRIT. Quote evidence word-for-word.',
)

@scorer.tool
def read_scrit() -> str:
    """Retrieve the scoring spec."""
    return pandas.read_excel('SCRIT_Board_17.xlsx').to_string()


@scorer.output_validator
def provide_evidence(ctx: RunContext[str], scored: ScoredFile) -> ScoredFile:
    for par in scored.pars:
        if par.evidence not in ctx.deps:
            raise ModelRetry('That quote is not in the file. Do not paraphrase.')
    return scored


member = Path('member_files/N12345678')
pars = [BinaryContent(data=p.read_bytes(), media_type='application/pdf')
        for p in member.glob('*.pdf')]

scored = scorer.run_sync(['Score this member.', *pars], deps=member_text(member)).output

That one check is provide_evidence: every point must be backed by a quote that actually appears in the file, or ModelRetry sends the model back to try again. A hallucinated quote does not survive a text search.

Notably, the SCRIT says performance scores “shall not be influenced by potential assessments or rankings”. A human reviewer has to try not to be influenced; the scoring agent simply never receives that data.

Board convention

Now, we can programmatically copy the board format itself.

Everyone scores the file. In our case: three scoring agents on models from three different labs, so they don’t share blind spots, each one something that can run inside the fence on hardware we control, plus a challenger whose only job is to find mistakes.

import asyncio


BOARD = [
    Agent(model,
          deps_type=str,
          output_type=ScoredFile,
          tools=[read_scrit],
          instructions='Score the member against the SCRIT. Quote evidence word-for-word.')
    for model in (
        'cohere:command-a-plus-05-2026',
        'mistral:mistral-large-latest',
        'meta:llama-4-maverick',
    )
]

challenger = Agent(
    'cohere:command-a-plus-05-2026',
    output_type=bool,  # True means "I object"
    tools=[read_scrit],
    instructions='Re-check these scorings against the SCRIT and the PARs. '
                 'Return True only if you find an error you can point to.',
)


def mean(scores: list[int]) -> float:
    return sum(scores) / len(scores)


async def convene(pars: list[BinaryContent], text: str) -> float | str:
    runs = await asyncio.gather(*(m.run(['Score this member.', *pars], deps=text) for m in BOARD))
    scores = [sum(p.points for p in r.output.pars) for r in runs]

    objection = await challenger.run([f'Scorings: {[r.output for r in runs]}', *pars])

    if max(scores) - min(scores) > 2 or objection.output:
        return 'FLAG FOR HUMAN REVIEW'

    return mean(scores)

If the three scorers land within a defined threshold of each other, the file gets the mean, exactly what it gets today. If the result gets challenged, or the scores are out-of-band, it gets no score at all. It goes to the humans with all three scorings, where they diverge, and the reasoning traces attached.

None of agents “see” one-another’s work, so there is nothing to anchor on. Even better, a systematic quirk in one vendor’s model would simply surface as a disagreement, which triggers human review, rather than becoming the consensus.

Where the humans go

Scoring a file is slow; reviewing a scored file is not. So move the humans to review.

Consensus files arrive scored, quoted, and traced: spot-check, then concur or override. Flagged files arrive as a ready-made argument: three scores, where they diverge, and the objection. Frankly, that’s the interesting part of a board anyway, and the part we currently get the least time for.

The arithmetic stands regardless: five days of five people’s time, per trade, per year, across more than 100 trades, spent mapping files onto a specification written specifically to leave little room for judgment. The machine’s bill for the same board, at list price²:

5 people, 5 days $11,503 ~20 API calls × 400 files $493

That excludes the reviewers, and the accredited infrastructure they’d need: PARs are Protected B, so the data stays in Canada. Consequently, the samples lean on a Canadian vendor— our home-grown Cohere 🇨🇦— for a reason. But it is the cost of the mapping, and the mapping is what eats the week.

Let the machines do the mapping, make them show their work, and save the humans for the judgment.