PowerShell Engineer

PS C:\> Get-SystemMap

PowerShell that has

The one artifact is AGENTS.md — the PowerShell Engineer Standard. The model is the demo. The benchmark is the moat, because models age out in a year and yardsticks do not.

Read the Standard Get the tools

Standard v1.1.0 · status August 2026

Every tool delivers, measures, or enforces the PowerShell Engineer Standard. AGENTS.md the PowerShell Engineer Standard Chrome extension Firefox add-on VS Code extension Open VSX Custom GPT AGENTS.md in repos Edge extension GitHub Copilot ext. M365 Copilot agent PowerShell Gallery MCP server Claude Skill PSEng (in weights) PSEval benchmark PSScriptAnalyzer GitHub Action
Delivered as Measured by Enforced by dashed = planned / building

The problem, measured

Not asserted. From peer-reviewed work on this exact task.

>60%of PowerShell from GPT-4o and o3-mini is insecure without structured guidance.
34%security compliance for GPT-4o — two thirds of its PowerShell violates a PSScriptAnalyzer rule.
~50%of the PowerShell in The Stack is bad. Models learned to write it that way.
3%GPT-4o's score at noticing a missing ShouldProcess. Near zero on pipeline support.

Models are decent at spotting what is present and terrible at noticing what is absent. A linter finds what is there. Only a standard tells a model what should have been.

Every figure above is from K. Zhang et al., Lightweight Yet Secure: Secure Scripting Language Generation via Lightweight LLMs (arXiv:2601.06419): the >60%-insecure result in the abstract, 34% compliance / two-thirds-violate in §4.2 and Table III, ~50% of The Stack in §3.1, and the 3% ShouldProcess-detection score in Table XI.

The thesis

The Standard is the product. The model is the demo.

Delivered as

  • Custom GPT
  • Chrome / Firefox / Edge
  • VS Code
  • Claude Skill
  • MCP server
  • AGENTS.md in repos

Measured by

  • PSEval, the scorer
  • The surface validator
  • The benchmark

Enforced by

  • PSScriptAnalyzer
  • Custom rules
  • GitHub Action
  • The repair loop

Two ways the Standard reaches a model

Each fixes the other's weakness, which is why both ship.

ModeHowCost / callAlways current?Works everywhere?
AttachedStandard in the system prompt~15,000 tokensYes, instantlyNeeds a prompt field
AdoptedFiles committed to the repo0 (agent reads it)YesAny agent that reads AGENTS.md

Adopted is /adopt: it writes AGENTS.md, Cursor rules, Copilot instructions, analyzer settings, and a CI action. Cheapest to build, most durable, and requires nobody to install anything.

The four value arguments

Each is true in a different situation. Don't lead with the wrong one.

1

Quality

The strongest, least contested. Generic prompts produce Write-Host for data, no -WhatIf, no pipeline support. Two thirds of GPT-4o's PowerShell fails the analyzer. Lead with this for practitioners.

2

Context window

The sharpest technical argument, and one nobody else can make: your own Standard is what makes API use expensive. ~15,000 tokens, re-sent every call, tripling cost. Commit it to the repo instead and agents read it for free.

3

Cost

True at volume, false for casuals. Most people pay $20/mo flat, and subscriptions beat API pricing below ~5M input tokens/month. Say so before someone else does.

4

Offline

The unbeatable one, and the reason to lead for regulated buyers. No script leaves the building. Works air-gapped, on a plane, when procurement says no, when the vendor has an outage.

Generation and repair are different products

Repair is constrained; the answer is mostly in the input. Small models excel.

GPT-4oSmall tuned model
Generation (functional)42%24%
Repair (fix success)54.8%87.8%
Measured on this project — the strongest result so far
First attemptAfter one repair
Arm A (no Standard)17.226.9
Arm B (Standard in context)45.654.2

Adding PSScriptAnalyzer to GPT-4o took fix success from 54.8% to 93.5%. The repair pass alone is worth ~8.6 points. If a loop is worth more than the training it would replace, that is a product finding on its own — and it is exactly the thesis the VS Code extension is built on.

GPT-4o and small-model figures from Zhang et al., arXiv:2601.06419: 42% generation and 54.8% unguided repair for GPT-4o, ~24% generation for a lightweight base model, 87.8% for the fine-tuned model (Tables III, VI, VII), and 54.8%→93.5% once PSScriptAnalyzer feedback is added (Table VII). The Arm A / Arm B rows are this project's own measurements, not from the study.

The measurement spine & the routing architecture

One scorer, four consumers. The version that wins never returns the code.

                scoring/
  parse gate · surface validator · analyzer
  AST checks · voice lint · style checks
                   │
  ┌────────┬───────┴───────┬───────────┐
  │        │               │           │
dataset  VS Code        PSEval      MCP server
filter   repair loop    benchmark   any agent
any agent → MCP: generate_powershell(task)
        │
        ├─ generate locally     0 tokens, offline
        ├─ scoring/ verifies    local, static, safe
        ├─ repair pass          local
        │
        ├─ clean  → return "47 lines, 0 violations"  ~20 tok
        └─ failed → escalate with analyzer output    $$$

The surface validator is the novel piece. Walk the AST, resolve every command and parameter against the user's real installed modules, reject anything invented. Shipped standalone it becomes Test-PSESurface. That changes the claim from "our model hallucinates less" to "our toolchain makes hallucination detectable" — which covers every model, frontier ones included.

Get the tools

Same Standard, wherever you work.

Chrome extensionChrome Web Store Firefox add-onMozilla Add-ons VS Code extensionMarketplace Open VSXCursor & VSCodium Custom GPT#2 PowerShell GPT · 5,000+ conversations Edge extensioncoming to Edge Add-ons The Standardread AGENTS.md

Status, honestly

What is actually true today.

The Standard (AGENTS.md)Shipped v1.1.0, MIT
Custom GPTShipped #2 PowerShell GPT, 5,000+ conversations
VS Code extensionShipped
Chrome extensionShipped
Firefox extensionShipping
Scorer + surface validatorBuilt 259 tests, version-aware, live Windows manifests
Arm A / Arm B baselineMeasured Arm B = 45.6 first attempt, 54.2 after repair
working-with-ai.md · ManifestoWritten
PSEval benchmarkHeld-out set frozen not published; field survey unrun
Analyzer custom rules · GitHub Action · MCP server · Gallery module · Domain packsPlanned
Five defects the project found in its own Standard
  1. Interpolation trap. "$Path: $_" fails to parse; needs ${Path}. Section 15.1's own guidance walks models into it. 13% of all candidates.
  2. The examples incoherence. §8 exempts .EXAMPLE from splatting; §17.1 still enforces a line limit — and splatting was the mechanism for shortening lines.
  3. Type-producing vs item-consuming ambiguity in §5's pipeline rule.
  4. ~120 characters. The tilde cannot be mechanized.
  5. The canonical function caught a type its own validator could not resolve.

These are findings the scorer surfaces on its own — the clearest demonstration of the feedback loop working.

What is durable

Durable

The Standard, PSEval, and the surface validator. Those get cited, forked, and built on. A model exists for a year. A yardstick is a thing other people are measured by.

The single most valuable unbuilt artifact is the benchmark run across the whole field — every serious community and frontier PowerShell model on one table. A benchmark carrying only your own model looks self-serving; a field survey does not, and the person who ran it owns the yardstick regardless of where any single model places.