PowerShell Engineer

PS C:\> Get-SystemMap

There is one artifact.
Everything else delivers it or measures it.

The one artifact is AGENTS.md — the PowerShell Engineer Standard. The model is the demo. The benchmark is the moat, because models age out in a year and yardsticks do not.

Read the Standard Get the tools

Standard v1.1.0 · status August 2026

Every tool delivers, measures, or enforces the PowerShell Engineer Standard. AGENTS.md the PowerShell Engineer Standard Chrome extension Firefox add-on VS Code extension Open VSX Custom GPT AGENTS.md in repos Edge extension GitHub Copilot ext. M365 Copilot agent PowerShell Gallery MCP server Claude Skill PSEng (in weights) PSEval benchmark PSScriptAnalyzer GitHub Action
Delivered as Measured by Enforced by dashed = planned / building

The problem, measured

Not asserted. From peer-reviewed work on this exact task.

>60%of PowerShell from GPT-4o and o3-mini is insecure without structured guidance.
34%security compliance for GPT-4o — two thirds of its PowerShell violates a PSScriptAnalyzer rule.
~50%of the PowerShell in The Stack is bad. Models learned to write it that way.
3%GPT-4o's score at noticing a missing ShouldProcess. Near zero on pipeline support.

Models are decent at spotting what is present and terrible at noticing what is absent. A linter finds what is there. Only a standard tells a model what should have been.

Every figure above is from K. Zhang et al., Lightweight Yet Secure: Secure Scripting Language Generation via Lightweight LLMs (arXiv:2601.06419): the >60%-insecure result in the abstract, 34% compliance / two-thirds-violate in §4.2 and Table III, ~50% of The Stack in §3.1, and the 3% ShouldProcess-detection score in Table XI.

The thesis

The Standard is the product. The model is the demo.

Delivered as

  • Custom GPT
  • Chrome / Firefox / Edge
  • VS Code
  • Claude Skill
  • MCP server
  • AGENTS.md in repos
  • PSEng (in weights)

Measured by

  • PSEval, the scorer
  • The surface validator
  • The benchmark

Enforced by

  • PSScriptAnalyzer
  • Custom rules
  • GitHub Action
  • The repair loop

Three ways the Standard reaches a model

Each fixes the others' weakness, which is why all three ship.

ModeHowCost / callAlways current?Works everywhere?
AttachedStandard in the system prompt~15,000 tokensYes, instantlyNeeds a prompt field
BakedTrained into weights (PSEng)0 tokensNo, frozen at buildYes
AdoptedFiles committed to the repo0 (agent reads it)YesAny agent that reads AGENTS.md

Adopted is /adopt: it writes AGENTS.md, Cursor rules, Copilot instructions, analyzer settings, and a CI action. Cheapest to build, most durable, and requires nobody to install anything.

The four value arguments

Each is true in a different situation. Don't lead with the wrong one.

1

Quality

The strongest, least contested. Generic prompts produce Write-Host for data, no -WhatIf, no pipeline support. Two thirds of GPT-4o's PowerShell fails the analyzer. Lead with this for practitioners.

2

Context window

The sharpest technical argument, and one nobody else can make: your own Standard is what makes API use expensive. ~15,000 tokens, re-sent every call, tripling cost. PSEng carries it in the weights for free.

3

Cost

True at volume, false for casuals. Most people pay $20/mo flat, and subscriptions beat API pricing below ~5M input tokens/month. Say so before someone else does.

4

Offline

The unbeatable one, and the reason to lead for regulated buyers. No script leaves the building. Works air-gapped, on a plane, when procurement says no, when the vendor has an outage.

WorkloadOpus 4.8Gemini 3.1 ProPSEng local
600 scripts / month$63.60$27.00$0.11
With repair loops (2.5×)$159$67.50$0.28
Refactor 200 scripts × 3 passes$102$44$0.37
Team of 10$636$270$0.11

The deeper point is not the bill. At seventeen cents a pass you ration. At zero you re-run the whole repo three times and compare. Being able to be wasteful is a capability.

The per-workload dollar figures are illustrative, computed from published list API pricing — not from a study. The peer-reviewed anchor: on the same PowerShell task, a fine-tuned lightweight model cost about 1% of GPT-4o's $50.80 in API charges (≈100× cheaper), per Zhang et al., arXiv:2601.06419, §6.2.

Generation and repair are different products

Repair is constrained; the answer is mostly in the input. Small models excel.

GPT-4oSmall tuned model
Generation (functional)42%24%
Repair (fix success)54.8%87.8%
Measured on this project — the strongest result so far
First attemptAfter one repair
Arm A (no Standard)17.226.9
Arm B (Standard in context)45.654.2

Adding PSScriptAnalyzer to GPT-4o took fix success from 54.8% to 93.5%. The repair pass is worth ~8.6 points; the fine-tune is being asked to clear 5. If a loop is worth more than the training it would replace, that is a product finding on its own — and it is exactly the thesis the VS Code extension is built on.

GPT-4o and small-model figures from Zhang et al., arXiv:2601.06419: 42% generation and 54.8% unguided repair for GPT-4o, ~24% generation for a lightweight base model, 87.8% for the fine-tuned model (Tables III, VI, VII), and 54.8%→93.5% once PSScriptAnalyzer feedback is added (Table VII). The Arm A / Arm B rows are this project's own measurements, not from the study.

The measurement spine & the routing architecture

One scorer, four consumers. The version that wins never returns the code.

                scoring/
  parse gate · surface validator · analyzer
  AST checks · voice lint · style checks
                   │
  ┌────────┬───────┴───────┬───────────┐
  │        │               │           │
dataset  VS Code        PSEval      MCP server
filter   repair loop    benchmark   any agent
any agent → MCP: generate_powershell(task)
        │
        ├─ PSEng generates      0 tokens, offline
        ├─ scoring/ verifies    local, static, safe
        ├─ repair pass          local
        │
        ├─ clean  → return "47 lines, 0 violations"  ~20 tok
        └─ failed → escalate with analyzer output    $$$

The surface validator is the novel piece. Walk the AST, resolve every command and parameter against the user's real installed modules, reject anything invented. Shipped standalone it becomes Test-PSESurface. That changes the claim from "our model hallucinates less" to "our toolchain makes hallucination detectable" — which covers every model, frontier ones included.

87% of PowerShell generation never touched the API.

Get the tools

Same Standard, wherever you work.

Chrome extensionChrome Web Store Firefox add-onMozilla Add-ons VS Code extensionMarketplace Open VSXCursor & VSCodium Custom GPT#2 PowerShell GPT · 5,000+ conversations Edge extensioncoming to Edge Add-ons The Standardread AGENTS.md

Status, honestly

What is actually true today.

The Standard (AGENTS.md)Shipped v1.1.0, MIT
Custom GPTShipped #2 PowerShell GPT, 5,000+ conversations
VS Code extensionShipped
Chrome extensionShipped
Firefox extensionShipping
Scorer + surface validatorBuilt 259 tests, version-aware, live Windows manifests
Arm A / Arm B baselineMeasured Arm B = 45.6 first attempt, 54.2 after repair
working-with-ai.md · ManifestoWritten
PSEval benchmarkHeld-out set frozen not published; field survey unrun
PSEngCorpus generation does not exist yet
Analyzer custom rules · GitHub Action · MCP server · Gallery module · Domain packsPlanned

Gate: Arm C must beat re-baselined Arm B by 5 points with a paired bootstrap 95% CI excluding zero (currently 50.6). The margin is the registration, not the absolute number. Product decision, recorded before results exist: a parity-C ships — a PSEng that only ties Arm B still wins on cost, context, and offline operation.

Five defects the project found in its own Standard by training against it
  1. Interpolation trap. "$Path: $_" fails to parse; needs ${Path}. Section 15.1's own guidance walks models into it. 13% of all candidates.
  2. The examples incoherence. §8 exempts .EXAMPLE from splatting; §17.1 still enforces a line limit — and splatting was the mechanism for shortening lines.
  3. Type-producing vs item-consuming ambiguity in §5's pipeline rule.
  4. ~120 characters. The tilde cannot be mechanized.
  5. The canonical function caught a type its own validator could not resolve.

These are findings the project produces whether or not a model is ever trained — the clearest demonstration of the feedback loop working.

What is durable and what is not

Disposable

PSEng. Qwen3 will be superseded and the model ages out with it. That is fine. It was always the demo.

Durable

The Standard, PSEval, and the surface validator. Those get cited, forked, and built on. A model exists for a year. A yardstick is a thing other people are measured by.

The single most valuable unbuilt artifact is the benchmark run across the whole field — six community models, four frontier models, and PSEng on one table. A benchmark carrying only your own model looks self-serving. One with eleven entrants is a field survey, and the person who ran it owns the yardstick regardless of where their own model places.