This guide explains how to monitor your agents' performance in PromptOwlarrow-up-right , collect and analyze feedback, run systematic evaluations, and use AI to continuously improve your prompts.
Table of Contents
The Improvement Lifecycle
Continuous improvement follows a cyclical process:
Copy ┌─────────────────────────────────────────────────────────────┐
│ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ DEPLOY │────▶│ MONITOR │────▶│ COLLECT │ │
│ │ Agent │ │ Usage │ │ Feedback │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ ▲ │ │
│ │ ▼ │
│ ┌──────────┐ ┌──────────┐ ┌──────────┐ │
│ │ PUBLISH │◀────│ IMPROVE │◀────│ EVALUATE │ │
│ │ Update │ │ Prompt │ │ Quality │ │
│ └──────────┘ └──────────┘ └──────────┘ │
│ │
└─────────────────────────────────────────────────────────────┘ Publish your agent to production
Track usage, tokens, conversations
Understand real-world behavior
Gather annotations and feedback
Identify improvement areas
Test systematically with eval sets
Measure quality objectively
Collecting Quality Feedback
Types of Feedback
PromptOwl supports two levels of feedback:
1. Message-Level Annotations
Feedback on individual AI responses:
When to use : Rating specific answers
Components : Sentiment (thumbs up/down) + detailed text
Best for : Identifying specific issues
2. Conversation-Level Annotations
Feedback on entire conversations:
When to use : Rating overall experience
Components : Sentiment + summary feedback
Best for : Holistic assessment
Encouraging Quality Feedback
For Internal Teams
Train your team to provide actionable annotations:
Good annotation:
"The response was accurate but too technical for our customer audience. Should use simpler language and avoid jargon like 'API endpoint'."
Poor annotation:
"Bad response"
Annotation Guidelines
Be specific about what's wrong
Use vague terms like "bad" or "wrong"
Just criticize without direction
Include context if relevant
Assume reader knows the situation
Screenshot: Annotation Modal Sentiment Best Practices
Use sentiment consistently:
Response was helpful, accurate, appropriate
Response was wrong, unhelpful, inappropriate
Response was acceptable but could be better
Accessing the Monitor
Open your prompt from the Dashboard
Click Monitor in the top navigation
View the conversation history and analytics
Screenshot: Monitor Navigation Monitor Interface Overview
The Monitor has two main tabs:
Shows all conversations with your agent:
All Annotations Tab
Aggregates all feedback:
Screenshot: All Annotations Tab Key Metrics to Track
Conversation Metrics
Total Conversations : Overall usage volume
Average Duration : How long users engage
Messages per Conversation : Conversation depth
Quality Metrics
Satisfaction Score : Average sentiment (0-100)
Positive Rate : Percentage of thumbs up
Annotation Volume : How much feedback collected
Total Tokens Used : API consumption
Model Distribution : Which models are used
Peak Usage Times : When most active
Filtering and Search
Find specific conversations:
Search : Type to filter by user, topic, or content
Date Range : Focus on specific time periods
Eval Filter : Show only test conversations
Screenshot: Monitor Filters Viewing Conversation Details
Click any conversation to see:
User questions and AI responses
Block outputs (for sequential/supervisor)
Screenshot: Conversation Detail Creating Evaluation Sets
Evaluation sets (Eval Sets) are collections of test cases used to systematically measure prompt quality.
What's in an Eval Set?
Each eval set contains:
Component
Description
Required
Desired response (annotation)
Method 1: Create from Annotations
Convert quality feedback into test cases:
Go to Monitor → All Annotations tab
Check boxes next to high-quality annotations
Screenshot: Save to Eval Set Best annotations for eval sets:
Clear positive examples (what good looks like)
Clear negative examples (what to avoid)
Edge cases users actually encountered
Method 2: Upload CSV
Import test cases from a spreadsheet:
Go to Eval tab on your prompt
Select your file with columns:
question or input - The test query
result or annotation - Expected response
CSV Format Example:
Method 3: Manual Entry
Add test cases one at a time:
Enter the expected output
Organizing Eval Sets
Create multiple eval sets for different purposes:
Basic functionality testing
Unusual or difficult queries
Validate new capabilities
Running Evaluations
The Evaluation Process
Running an Evaluation
Go to your prompt's Eval tab
Select an Eval Set from the dropdown
Select the Version to test
Wait for all test cases to complete
Review results in the table
Screenshot: Run Eval Button Understanding Results
After running, you'll see:
What the AI actually said
Comparing Versions
Test multiple versions to find the best:
Make prompt improvements → Create Version 2
Compare results in the Runs tab
The Runs tab shows historical comparisons:
Eval Set
Version
Date
Count
Avg Score
Pass Rate
![Screenshot: Runs Comparison]
AI Judge automatically evaluates response quality using another AI model.
What AI Judge Does
Instead of manually reviewing every response, AI Judge:
Reads the expected output
Reads the actual response
Scores quality (1-5 scale)
Judge Evaluation Criteria
The AI Judge evaluates based on:
Does the response match expected output?
Does it cover all necessary points?
Is it clear, helpful, and well-structured?
Acceptable, room for improvement
Significantly wrong or unhelpful
Running Judge Evaluation
First, run a regular evaluation
Wait for judge to score all results
Review scores and reasoning
![Screenshot: Run Judge Button]
Understanding Judge Output
After judging, you'll see:
AI's reasoning for the score
Example Judge Output:
Aggregate Metrics
Judge evaluation calculates:
Average Score : Mean of all test case scores
Pass Rate : Percentage scoring 3 or above
These appear in the Runs tab for easy comparison across versions.
AI-Assisted Improvement
PromptOwl can use AI to suggest improvements based on real conversations and feedback.
Accessing Improve with AI
Open your prompt in edit mode
Click Improve with AI button
The improvement dialog opens
![Screenshot: Improve Button]
Improvement Sources
The AI can analyze:
1. Example Conversations
Paste or auto-load real conversations:
2. User Feedback
Your specific improvement requests:
Auto-loaded feedback from production:
Message-level annotations
How Improvement Works
Prompt-Type Specific Improvements
AI understands your prompt type and optimizes accordingly:
Focus areas:
Response format consistency
Sequential Prompts
Focus areas:
Block efficiency and necessity
Output quality at each stage
Supervisor Prompts
Focus areas:
Agent specialization clarity
Applying Improvements
Improvements are applied non-destructively :
Block Names : Updated if AI suggests better names
Content : Saved as new variations (original preserved)
Testing : A/B test original vs improved
![Screenshot: Improvement Results]
Improvement-to-Evaluation Loop
After applying improvements:
New variations are created
Run evaluation against the new version
Compare scores with previous version
If improved, publish; if not, iterate
Annotation Best Practices
Collecting Effective Feedback
Vague feedback isn't actionable
Same issue multiple times = priority
Building Annotation Culture
Train team on annotation guidelines
Celebrate improvements from feedback
Share "before/after" examples
Make annotation part of workflow
Monitoring Best Practices
What to Monitor Daily
Conversation volume trends
Negative sentiment spikes
What to Review Weekly
Aggregate satisfaction scores
Version performance comparison
Setting Up Alerts
Consider monitoring for:
Satisfaction score drops below threshold
High negative annotation rates
Evaluation Best Practices
Building Quality Eval Sets
70% should be typical questions
20% should test boundaries
10% should test error handling
Add new cases from production
Running Effective Evaluations
Baseline first : Eval current version before changes
One variable : Change one thing, then eval
Statistical significance : Use enough test cases (25+)
Judge consistently : Use same judge prompt
Interpreting Results
Metric
Target
Action if Below
Improvement Best Practices
When to Use AI Improvement
After identifying patterns in annotations
When manual tweaking isn't working
To get fresh perspective on prompt
When scaling improvement efforts
What to Include in Feedback
Helpful feedback:
Not helpful:
Validating Improvements
Don't trust blindly : AI suggestions need human review
Test systematically : Run eval before publishing
Start with one change : Don't apply all suggestions at once
Monitor after publish : Watch for regressions
Continuous Improvement Workflow
Troubleshooting
Annotations not appearing
Check annotation feature is enabled (Enterprise Settings)
Verify user has permission to annotate
Check correct prompt is selected
Evaluation failing
Verify eval set has test cases
Check prompt version exists
Ensure API keys are configured
Try running single test case first
Judge scores seem wrong
Review judge prompt (uses production version)
Check expected outputs are realistic
Verify input/output alignment in eval set
Consider adjusting pass threshold
AI improvement suggestions unhelpful
Provide more specific feedback
Include actual conversation examples
Describe what good looks like
Try with different conversation samples
Metrics not updating
Verify conversations are completed
Allow time for aggregation
Quick Reference
Keyboard Shortcuts
Key Metrics Glossary
Positive / (Positive + Negative)
Eval Set Size Guidelines
