Teach the AI with feedback

• Flag a bad AI reply: point at it and click the thumbs-down. Say what went wrong in your own words; AI assist drafts the correction and you submit it.
• Triage in AI Feedback. Each flag is shown with its conversation. Keep it, edit it, or dismiss it.
• Review & improve: four screens (keep or dismiss, see why, review the fix, choose tests). You approve a before-and-after diff before anything is saved.
• Run the tests it added, so the fix stays fixed.

“Teaching the AI” is three different jobs, and picking the right one saves an afternoon. Ask what the reply got wrong:

Four moments, read left to right. Each is a different screen and usually a different person: whoever is working the Inbox flags, and an admin does the rest.

• A flag belongs to one reply and one person. It stores the reply, the conversation around it, what the reply should have said and a category.
• A fix is a reviewed edit, not a retrain. The same copilot you use in the agent editor writes it, in a cautious mode that prefers small surgical changes. Nothing is saved until you press Apply the fix.
• The diagnosis comes first. Before you see a fix, each kept flag is traced to the part of the setup that caused it, so you judge whether the fix removes the cause.
• A test guards the fix. Any kept flag can become a test that replays the conversation and checks the new reply.

Anyone reading the conversation can react to an AI agent’s reply. Only replies sent by an AI agent have the thumbs; messages from people, workflows and broadcasts do not.

Every flag lands in the AI Feedback page of the Admin Dashboard. Getting there takes two clicks.

This is where flags change the agent. It is one full-screen flow with four steps, on top of the agent’s editor, so the sidebar is gone on purpose. A step rail across the top shows where you are, and Leave is always top right. It works on the agent’s Pending thumbs-down items only.

A fix that is never re-checked quietly comes undone the next time someone edits the instructions for another reason. A test freezes the situation: it replays the conversation up to the flagged reply (with up to five messages before it) and an AI judge checks the agent’s new reply against what it should have said.

• From the review flow, ticked items become tests when you apply the fix.
• From one queue item, click Promote to test case in its drawer. It opens the new test in the Tests tab, and the button becomes View test case. The caret beside it offers Test as a simulated contact, which reads the whole conversation into a persona the agent must handle live on every run; it takes a few seconds longer.
• Good replies can be promoted too. The note you left is what the test checks against, so write one.

• Nothing is fine-tuned. The fix is a reviewed edit to the agent’s own setup: its instructions and, where the cause is there, its actions, arguments, exit conditions, required resources or output data. It is fast, visible, and yours to edit.
• Apply the fix saves the agent straight away. There is no separate draft step, and the fix does not touch the Draft / Published switch. A Published agent uses the new setup on its next reply, so Step 3 is your only review.
• Only Pending thumbs-down items feed a fix. Good examples never do; their job is to become tests. Dismissed and already-trained items are left out too.
• Concrete beats vague. What the AI should have said is required to submit, and it is what the fix and the test lean on hardest. “Tone was off” teaches little; the exact sentence you wanted teaches a lot.
• Dismissals save as you go; the fix does not. The moment you move to Step 2, your dismissals are saved. Leave after that and the diagnosis and the written fix are discarded, the agent stays as it was, and dismissed flags stay dismissed.
• Every item carries its context. Open one and you get Feedback, Conversation context and LLM call (tokens in and out, the exact prompt sent, the raw response). The LLM call tab is greyed out when nothing was captured.
• Reject keeps a record; Delete does not. Rejected items stay on the Rejected tab with their reason. Delete is permanent and offered only while an item is untouched: not trained, not used in a fix, not promoted to a test.
• Only an AI agent’s replies can be flagged. An AI Model step in a workflow computes things for other steps and never sends a message, so there is nothing to flag.
• Library-managed agents can’t be refined this way. Their instructions belong to the library; the service refuses with “Managed agents cannot be refined directly - edit the agent configuration instead”.
• Tests do not run themselves. A test is checked when you click Run All or run it from its row. Run them after every fix.