Extending CodeLoop with the Plugin SDK: Bring Your Own Runner
Extending CodeLoop with the Plugin SDK
CodeLoop's verify loop is opinionated about *evidence* (typed JSON, gate-able confidence, lineage) but unopinionated about *tooling*. If your stack is Django + pytest, Rails + RSpec, Go + go-test, or something more exotic, the plugin SDK lets you wire it into the loop without touching CodeLoop itself. This post walks through the layout, the contract, and two complete examples that ship with the repo.
The layout
A plugin lives in your project at .codeloop/plugins.json. The orchestrator auto-loads it on every codeloop_verify call and merges its runners into the built-in runner set. Plugin runners participate in gates, diagnose, and the dashboard exactly the same way the built-in ones do — they are not second-class citizens.
A plugin file is a single object:
{
"name": "django-plugin",
"version": "1.0.0",
"runners": [
{
"id": "django_pytest",
"kind": "test",
"detect_file": "manage.py",
"command": "pytest --json-report --json-report-file=.codeloop/runs/
"result_file": ".codeloop/runs/
"result_format": "pytest-json-report"
}
]
}
That is it — one runner, one detect rule, one command, one parser. The SDK supplies the framework: lineage IDs, working directory, environment variables, exit-code handling, and timeouts.
The runner contract
Every runner plugs into one of three kinds: build, test, or lint. The kind decides which gate the runner contributes to and how its output is rendered in the dashboard. Inside that, a runner needs four things:
id — a stable identifier the orchestrator uses across runs.detect_file — a path that, if present, signals the runner is applicable. Django plugins detect manage.py; Rails plugins detect Gemfile; Go plugins detect go.mod.command — the shell invocation. result_format — one of junit-xml, pytest-json-report, rspec-json, or generic-json. The SDK parses the file into a typed result; the orchestrator does the rest.There is no JS or Python entry point to write. The plugin is pure config. That is deliberate: the long-tail problem with custom runners is keeping them updated as the host product evolves, and a config file ages better than a hand-written hook.
Example: Django
The repo ships examples/plugins/django with three runners — django_pytest, django_lint (flake8), and django_migrations (manage.py migrate --check). Drop the file into any Django project at .codeloop/plugins.json, run codeloop init, and the verify loop now exercises pytest, flake8, and migration drift on every iteration. Failures appear in codeloop_diagnose with structured stack traces; passes contribute to the gate's confidence score.
The example's README walks through the Python-version requirement, the venv hint, the pytest-json-report dependency, and a slow-test skip flag we found useful in CI.
Example: Rails
examples/plugins/rails ships rails_rspec (bundle exec rspec --format json), rails_rubocop, and rails_db_migrate. The README covers the bundler hint, the json formatter gem, and a parallelisation note for projects with thousands of specs.
Both examples share the exact same shape — they are essentially a different command and result_format per runner. The SDK does the heavy lifting.
Building your own
The fastest path to a new plugin is:
result_format (or use generic-json if your tool emits a custom shape).codeloop verify. The dashboard now shows your runner.If your tool does not emit JSON or JUnit XML, the smallest viable adapter is a one-line wrapper that pipes its output through a JSON converter. The SDK explicitly accommodates this — command can be a shell pipeline, not just a single binary.
Why no JS hooks
We considered shipping a TypeScript SDK with onPreVerify / onPostVerify hooks, the way many CI tools do. We chose not to: hooks make plugins harder to share, harder to upgrade, and easier to break. The current shape — a config file plus a parser registry — keeps the plugin surface intentionally small. If you find a runner shape the parser registry cannot express, open an issue with a sample output and we will add it; that is a faster path than every plugin author writing their own parser by hand.
Try it
The two bundled examples are deliberately scaffolded as drop-in templates. Clone the repo, copy examples/plugins/django/ or examples/plugins/rails/ into your project, point detect_file at your real entry point, and you have a verify loop that understands your stack. From there the dashboard and the gate carry the rest.