Build a development bot that proposes reviewable fixes

Capture the failing input, expected result and current result before asking for a patch. For example: an import accepts an empty reference, but the agreed rule requires a non-empty reference before any record is written. Include the relevant error contract and a fixture with no real customer data.

A task contract names allowed directories, forbidden changes, the target commit and commands that may run. Keep acceptance criteria outside the agent’s editable workspace. If the requirement is ambiguous, the output should be a question for the owner, not a guessed implementation.

• Define one behaviour change and a maximum diff scope.
• Exclude schema migrations, dependency upgrades and credential changes unless separately approved.
• Provide a stop path when reproducing the defect fails or requires unavailable access.

1. 01Create an ephemeral checkout at a recorded commit and inject only approved fixtures.
2. 02Run the reproduction and existing checks to establish the starting state.
3. 03Let the worker inspect approved files and propose a patch within resource and path limits.
4. 04Run checks from a protected test source, then inspect the final diff for scope violations.
5. 05Have a separate publisher create a draft PR/MR with the evidence and limitations. The repository owner reviews and merges it.

The worker should not hold production credentials or a token that can merge. Deny network access by default and allow only required destinations. Repository scripts can execute arbitrary code, so a sandbox must constrain the process itself; a textual instruction to “be safe” does not create isolation.

Timeout, cancellation and cleanup are part of the system. A job killed during generation must not leave a half-published branch that looks completed. Deduplicate events and name the source ticket and commit in every output.

The proposed change should carry a short explanation of the defect, why the patch addresses it, commands run and their actual exit status, and checks that could not run. Do not accept a generated narrative as test evidence: collect the runner’s results directly.

A small diff can still be wrong. Review error compatibility, race conditions and downstream callers. If the fix changes a user-facing error, update its expected contract with the product owner; do not widen scope merely because the bot discovers nearby cleanup work.

The agent can generate additional tests, but an independent test should fail on the original defect and pass on the proposed patch. Keep that fixture under a separate review boundary. A bot that changes both the implementation and the sole expected result can manufacture a green run.

Check deliberate variants: remove the new validation, invert the branch, or allow whitespace-only input. If the new test still passes, it is not protecting the intended behaviour. Include existing error, authorisation and retry cases where the patch touches those boundaries.

Use the release checks for AI-generated code for the wider verification pass. The bot’s own completion message never substitutes for a release checklist.

Select a sample of resolved low-risk tickets that was not used to tune the prompts. Compare proposed fixes with the original requirements, using reviewers who can judge compatibility. Count acceptable proposals, wrong fixes, abstentions and proposals needing substantial rewrite. Measure elapsed review and correction time alongside execution time.

Include incomplete reproductions, unavailable dependencies and a malicious repository instruction. The worker should stop or stay inside its boundary. Test publication retries too: two deliveries of an event should produce one draft change, not two competing fixes.

A bot that produces more patches but increases reviewer workload has not accelerated delivery. Record the task mix and observation window when reporting improvement. Do not extrapolate a pilot on validation bugs to architectural work.

Budget model usage, sandbox runtime, dependency fetching, storage and human review. Retry limits should apply to the whole job, not just each model request. Long-running loops need a total time and spend limit with a clear stopped status.

Retain minimal operational records for investigation: task identity, commit, configuration version, checks and reviewer disposition. Redact sensitive logs and decide how long temporary workspaces persist. Re-evaluate after changing model, permissions or the build system.

A maintained integration belongs in AI software-delivery scope. Bring a reproducible example to discuss a pilot. Current permission references, checked 11 October 2026: Anthropic: Claude Code security and GitHub: secure use of Actions.