Teach the AI with feedback

• Flag an AI reply in the chat. For a thumbs-down, describe what went wrong — AI assist drafts a clean correction you confirm.
• Triage it in the AI Feedback queue with the full conversation and model context: edit, reject, or set it aside.
• Train the agent on a batch of flagged items — the AI rewrites its system prompt, you review the diff and accept.
• Lock wins in as test cases so future changes never undo what already works.

Three stages, two surfaces. Frontline staff capture signal in the Inbox. An admin triages it in one place. And when enough has built up, training turns that pile of corrections into a sharper set of instructions for the agent.

The crucial thing to understand up front: training doesn’t fine-tune a model. It rewrites the agent’s system prompt — the plain-English instructions that govern how it behaves. That’s why it’s fast, transparent, and completely reversible: you’re always reviewing a diff of text before anything changes.

On any message sent by an AI agent, a verdict control lets anyone working the Inbox react: Mark as good when it nailed the answer, or Flag when it didn’t.

A thumbs-up is one tap — optionally jot why it’s a good example. A thumbs-down opens a guided drawer. You don’t fill in a rigid form — you just describe what went wrong in plain language. AI assist reads it (and may ask a single follow-up if it’s unsure), then drafts a structured correction:

The category is one of Hallucination, Wrong tone, Missed context, or Other. The most valuable field is what it should have said — that’s the line the rewrite leans on hardest, so make it concrete.

Everything captured collects in the admin AI Feedback queue, grouped into three tabs by status: Pending, Trained, and Rejected. New items land in Pending. You can filter by category and search the corrections and rationales.

Open any item to judge it on its merits. The detail view has three tabs:

• Feedback — the flagged message, the issue category, the suggested correction, and the reviewer’s rationale.
• Conversation context — the messages on either side of the flagged turn, so you can see what the customer actually asked.
• LLM call — the model used, token usage, and the exact prompt and raw response behind the reply. (Shown when it was captured.)

From the Feedback tab, you have four moves on a single item:

This is where feedback becomes change. On the Pending tab, items are grouped by agent with a Train [agent] on N items button. Click it and the AI reads the agent’s current system prompt plus every flagged correction in the batch, then proposes a single rewritten prompt that addresses them all — surgical edits over wholesale rewrites.

You don’t accept blind. You get a side-by-side diff of the current prompt versus the refined one, with the changes highlighted:

Read it, tick which items are worth keeping as regression tests, and hit Accept & update draft. Three things then happen:

• The refined prompt is written to the agent’s draft — not the live agent. You publish it from the agent itself once you’ve had a final look.
• Every item in the batch moves from Pending to Trained, tagged with the refinement run it belonged to.
• The items you ticked become test cases on the agent (see Step 4).

A great answer is fragile — the next prompt edit could quietly undo it. Promoting a piece of feedback to a test case freezes that situation into a check the agent must keep passing.

The test uses the conversation up to the flagged turn as the input and your “what it should have said” as the expected response. From then on, an AI judge checks the agent against it whenever you change the prompt — so a fix for one problem can’t silently break another.

• “Train” rewrites the prompt — it doesn’t fine-tune a model. There’s no data-science step and no model weights change. The AI rereads your agent’s instructions and proposes a sharper version. That’s why it’s fast, reversible, and yours to edit.
• Training updates a draft, not the live agent. Accepting writes the refined prompt to the agent’s draft. Go to the agent to review and publish it. If the agent already has unsaved draft edits, you’ll be asked to Overwrite draft or cancel.
• Only flagged (bad) items train. A refinement run bundles Pending items with a thumbs-down verdict, for a single agent. Good replies don’t feed the rewrite — their job is to become regression tests.
• A correction teaches more than a flag. The “what it should have said” line is what the AI leans on most. The more concrete it is, the better the rewrite — and the more likely it’s worth a test.
• Every item carries its context. Open one and you get three tabs: Feedback, the full Conversation context around the flagged turn, and the LLM call — the model, token usage, and the exact prompt the agent saw. Judge it fairly before you decide.
• Reject keeps a record; Delete doesn’t. Prefer Reject — it stays searchable with its reason. Delete is permanent and only offered while an item is untouched (never trained or promoted).
• Only AI-agent replies can be flagged. Messages sent by an AI agent in the Inbox carry the verdict control. AI compute nodes inside a workflow never send a message of their own, so there’s nothing to flag there.