Skip to main content

How to test an AI Agent before it goes live

  • Nov 17, 2026 5:24 PM UTC
  • 0 replies
  • 1 view
vinicius.pereira
Community Manager

👤 For pipe admins who configure and review AI Agents
🔐 Available on all plans, subject to access to AI features and pipe permissions
🎯 To compare scenarios, refine instructions, and decide what needs validation before a behavior is used

 

A reimbursement request includes a receipt and an amount, yet the team still needs to decide whether the information is sufficient for review. When delegating that check to AI, consider what should happen when something is missing. A convincing answer for one complete request does not establish how the agent will handle everyday exceptions.

By the end of this article, you will be able to prepare scenarios with expected outcomes, compare responses using Test Behavior, and refine instructions before operational use. Our reimbursement example asks the agent to identify missing information and recommend a next step, while a person retains approval.

 

What you will learn:

 

Define what the behavior must demonstrate

Choose one behavior and describe an outcome the process owner can verify. In this hypothetical example, it should check whether a request contains the information required by the policy, explain what is missing, and indicate whether it is ready for human review. The test must distinguish that recommendation from final approval.

Record the inputs, the reference rule, and the decisions the agent must leave to a person. If the policy does not explain how to handle a missing receipt, resolve that question with its owner before assessing the AI response.

Check that inputs reference actual fields and sources. Typing a field name into the instructions does not replace selecting its dynamic value. The guide to effective instructions for AI Agents explains how to provide context and specify outputs.

 

A useful acceptance criterion: the owner can explain why each scenario should produce its expected outcome before reading the agent's response. This avoids accepting an answer simply because it is well written.

 

Choose scenarios that reflect everyday work

Prepare a small set of cases that varies the relevant conditions. For reimbursements, consider these teaching examples and adapt the criteria to company policy:

  • Complete request consistent with the policy: indicate readiness for human review and explain the reference used.
  • Missing receipt: identify the missing information without assuming what the document contains.
  • Reported amount differs from the receipt: flag the discrepancy for review rather than silently choosing a value.
  • Situation outside the available policy: indicate that review is needed rather than inventing a rule.

Write down the expected outcome for each case and retain these examples for future revisions. A set containing only complete requests covers part of the work. You also need to check whether explanations and proposed next steps remain appropriate when the inputs change.

If the behavior includes arithmetic, prepare the numerical result separately. Calculations and business rules with AI Agents explains boundaries and exceptions while separating calculation from approval.

 

Compare the simulation with the expected outcome

Test Behavior simulates an individual behavior in the AI Agent builder using a reference card, without changing production data. It helps you examine what that behavior would do before operational use. The behavior testing update, in Portuguese, introduces the feature.

 

Test one behavior at a time and compare the simulation with the outcome prepared for that scenario. Screenshot from the official feature update.

Test one behavior at a time and compare the simulation with the outcome prepared for that scenario. Screenshot from the official feature update.

 

Compare the conclusion, explanation, and proposed action. For a missing receipt, a response that recognizes the omission but still recommends approval does not meet the stated criterion. A response requesting review should explain which information is missing.

Record the scenario, instruction version, and discrepancy. The simulation assesses the selected behavior; it does not establish that every stage and connection in the complete process has been validated. The Pipefy Updates Execution journey connects testing with agent use and monitoring.

 

Refine instructions and repeat the same scenarios

Turn the discrepancy into a specific change. If the instructions omit missing receipts, state that condition and its expected handling. If the conclusion was correct but the destination was not, review the associated actions and fields rather than rewriting the entire instruction.

Improve instructions suggests a clearer, more structured version of the text. The suggestion remains editable: check that it preserves the policy, exceptions, and human responsibility before accepting it. See the Improve instructions update, in Portuguese, for more context.

After a change, rerun the scenario that prompted it and the cases that previously passed. Compare with the original reference instead of changing the criterion to accommodate a different response. Keep a record of the change and the checks so another person can resume the review.

 

Better wording does not establish validated behavior. Choosing to use the revised version depends on the scenario results and confirmation from the process rule owner.

 

Agree on rollout and monitoring

Before expanding use, agree with the admin on the actions the behavior will perform, who will review initial results, and how exceptions will be handled. Check how human approval is represented in the process. The configuration must reflect that responsibility as well as the instructions.

For initial live runs, compare the card result with the expectation and consult Logs when you need to understand the path. Continue with How to investigate an AI Agent execution. For transitions between phases, see Why a card did not move as expected.

 

To explore the relationship between context, instructions, and agents, the Pipefy Artificial Intelligence Specialist Certification provides further practical learning. The course is in Portuguese, and accessing lessons requires signing in or creating an Academy account.

 

Before moving on, confirm that you understand:

☐ Expected outcomes need to be defined before simulation.

☐ Representative tests cover complete requests, missing information, and exceptions.

☐ You can refine a discrepancy, rerun the cases, and agree on who will monitor operational use.

 

Ready to follow a request through its phases? Continue with Why a card did not move as expected.