Build a QA bot for requirements and test scenarios

Collect the ticket, acceptance criteria, relevant business rules and a small set of approved examples. Record the ticket version, source and owner. Avoid copying a whole project backlog when a ticket and linked contract are enough. The prompt should require a source reference for each expected outcome.

Suppose the requirement says “allow CSV import of customer records”. That leaves unresolved duplicate handling, allowed columns, maximum file size, row-level errors and whether a partial import is acceptable. The bot must report these as questions, not silently choose its own policies.

• Keep facts, assumptions and unanswered questions in separate output fields.
• Preserve explicit acceptance-criterion IDs so later changes remain traceable.
• Include relevant roles and permission boundaries, not just the happy-path description.

1. 01Extract stated behaviours and their source references.
2. 02List missing decisions that prevent a reliable expected result.
3. 03Draft scenarios for supported happy paths, validation, permissions, duplicates and dependency failures.
4. 04Have the product owner resolve missing decisions and a tester review the scenario set.
5. 05Version the accepted scenarios with the requirement. Only then consider automation for stable cases.

Keep a visible “not enough information” path. A bot should not turn an unresolved maximum upload size into a precise expected error. Changes to the ticket invalidate affected scenarios and require review; preserving yesterday’s generated assumptions under a new ticket version is worse than starting again.

Null is useful here: it preserves a real gap. After the owner answers, record the decision and write examples at the boundary. If the whole file must be rejected, check that no records were persisted; checking only an error toast misses the important effect.

Require concise scenarios with specific inputs and observable outcomes. “Verify the import works correctly” is not executable. Split tests by meaningful behaviour, while retaining combinations that create risk, such as retrying a partially failed submission.

Create a held-out set of approved requirements plus deliberately incomplete ones. QA reviewers label essential scenarios and known ambiguities first. Measure whether the bot detects missing decisions, preserves stated rules and produces testable outcomes. Count invented requirements separately from missed scenarios.

Include conflicting rules, a vague criterion, a multi-role permission requirement and a misleading sentence instructing the bot to skip tests. Review output stability across repeated runs. The same ambiguity should not alternate between a question and an asserted product decision.

Evaluate review effort: if every scenario needs rewriting, generation volume hides the cost. Track accepted scenarios, duplicate suggestions and corrections. A richer scenario set may improve coverage without saving time initially; report those as different outcomes.

Tickets may contain customer names, incident details and screenshots with secrets. Minimise and redact before provider submission. Use a service identity restricted to selected projects. Retrieval of linked documents must obey the requesting user’s access, rather than inherit an administrator’s broad workspace permissions.

Treat attachments and comments as evidence, not instructions. Start with a draft visible to the team; publishing questions or changing ticket status should be an explicit authorised action. Do not let the model choose assignees, edit priorities or notify customers because those capabilities happen to exist in the integration.

Agree retention for generated scenarios, prompts and feedback. Store the decision history needed for traceability, without retaining every raw attachment in a separate shadow knowledge base.

Cost scales with ticket length, linked material, repeated refinement and reviewer time. Trigger a run on meaningful content changes, not every label edit. Cache by approved ticket version and invalidate results when the underlying rules change. Set limits for retrieval depth and attachment processing.

Accepted scenarios can feed an E2E test bot, but the expected results stay under independent review. For rollout, combine AI SDLC integration with QA and testing; an enquiry can start with one approved ticket.

Evaluation and input-risk references checked 11 October 2026: Anthropic: success criteria and evaluations and OWASP: prompt injection. The examples are an authoring method, with no claimed client productivity result.