An AI feature can stay online while becoming less useful. Documents go stale, access changes, model behaviour shifts and upstream tools evolve. Maintenance has to monitor task quality and permission boundaries as well as availability. Assign owners before launch rather than assuming the model provider will maintain your integration.

Name the owners of knowledge, integration and risk

A knowledge owner decides which sources are current and resolves conflicting policy. An engineering owner maintains retrieval, tool connections and deployment. A business or product owner approves task scope and acceptable risk. One person may hold several roles, but none of those responsibilities should be implicit.

  • Maintain a source inventory with audience, owner, version and deletion rules.
  • Document credentials, permitted actions and how to revoke them.
  • Keep evaluation cases and configuration versions with a reproducible release process.
  • Record the manual fallback and who handles escalated tasks.

Treat changes as small releases

  1. 01Identify what changed: model, prompt, source, retrieval, tool API or permissions.
  2. 02Run the affected deterministic checks and held-out task evaluations.
  3. 03Inspect difficult cases, supported claims, abstention and unauthorised requests.
  4. 04Review latency, usage and operational limits on representative requests.
  5. 05Have the owner approve the change and retain a rollback configuration.
  6. 06Monitor the changed scope after rollout and document unexpected results.

A provider upgrade that improves general capability may still alter the behaviour your feature depends on. A changed policy document can make yesterday’s correct answer wrong. Re-evaluate both cases; do not reserve release checks only for application-code changes.

Monitor failures without collecting unnecessary private data

Record useful categories: retrieval failed, no supporting source, provider timeout, malformed output, permission denial, tool failure and human escalation. Track volume, latency and cost. Sample content only under an approved policy with access and retention appropriate to the data.

A support assistant needs feedback that identifies unsupported or stale answers. A workflow bot needs records of proposed actions, approvals and actual execution results. Monitoring should make an investigation possible while avoiding a second uncontrolled store of customer documents and conversations.

Design an operational stop and fallback

Have a way to disable tool execution independently of ordinary read or search functions. On a permission incident, stop the affected action, preserve necessary evidence and follow the organisation’s incident process. Revoke credentials where appropriate. A prompt edit alone is not enough if the integration has excessive authority.

For provider outages or limits, show an honest failure state and preserve the manual route. Do not retry indefinitely or silently label an unprocessed task completed. Test rollback, credential expiry and duplicate-event handling before those situations occur under pressure.

Include quality review in the recurring budget

The operating plan includes model and tool usage, retrieval/storage, hosting, evaluations, source maintenance and review or exception effort. Watch request size and retries as well as volume. A small change to retrieved context can materially change usage even when the number of users is stable.

Use AI cost estimation to document assumptions. Set a review point for both benefit and cost; a feature that no longer earns its maintenance should be narrowed, revised or retired rather than kept because the initial implementation was expensive.

Distinguish launch support from maintenance

The delivery proposal should name the included support window, covered defects, reporting route and responsibilities. New knowledge sources, wider use cases, model migrations and ongoing monitoring may be separately scoped maintenance. No standard support duration or service-level response promise is asserted here without an agreed contract.

Handover should include source and account access, configuration, evaluation fixtures, deployment steps, incident guidance and external dependencies. A team choosing to operate independently needs enough information to change or disable the feature without relying on the original supplier.

For a product feature, see product AI integration and the support-assistant guide. For internal work, see business-process enablement. Describe the ownership and support you need to include it in the scope.