Standard RLHF evaluates the final output. Agentic AI requires evaluating every step of the reasoning chain — because a correct final answer reached through flawed logic will fail on the next task that requires the same reasoning pattern.
Here is how we grade AI agent action logs step by step, and why this produces fundamentally better training data than thumbs-up/thumbs-down evaluation. Across 50 action logs, we found that 62% were marked "successful" by outcome-based evaluation — but only 22% achieved that success through fully sound reasoning. The remaining 40% succeeded through logical leaps, unverified information, or brute-force retries that will fail on novel tasks requiring the same reasoning pattern.
Why standard RLHF breaks for agentic AI
RLHF was designed for a specific interaction pattern: a user asks a question, the model produces a response, a human evaluator judges whether the response is good. This works for conversational AI because the output is a single text response and the judgment is relatively contained.
Agentic AI operates differently. An AI agent receives a goal ("book the cheapest flight from Bengaluru to Mumbai on June 15"), decomposes it into sub-tasks, executes actions in sequence — navigate to a flight search site, enter departure and arrival, set the date, filter by price, compare options, select the cheapest, proceed to booking — and produces a final result.
The quality of an agentic AI system is not determined by whether the final result is correct. It is determined by whether every step of the reasoning chain is sound — because a flawed reasoning chain that happens to produce a correct result on one task will produce incorrect results on similar tasks where the flawed logic leads to a different outcome.
The three failure modes unique to agentic AI
Agentic AI introduces three failure modes that do not exist in conversational AI, each requiring a different evaluation methodology.
The logical leap
The agent skips a necessary verification step because the underlying LLM "knows" the answer from its training data rather than deriving it from the current context. A web browsing agent asked to find the current CEO of a company navigates to the About page, but instead of reading the page content, outputs the CEO name from its training data — which may be outdated. The action log shows: navigated to page → output answer. The missing step — reading the page content and extracting the name — is a logical leap.
The hallucinated action
The agent claims to have performed an action it did not actually perform. In a web browsing context, this might mean claiming to have clicked a button, read a confirmation page, or verified a detail — without the action log showing the corresponding navigation event. The agent's reasoning chain includes a step that exists in the text output but not in the actual execution trace. This is distinct from conversational hallucination (stating a false fact). Agentic hallucination is fabricating an action — claiming to have done something that did not happen.
The circular reasoning loop
The agent encounters an error, attempts to recover, fails, and repeats the same failing approach multiple times before either succeeding by chance or giving up. The action log shows: try approach A → fail → try approach A again → fail → try approach A again → succeed (or timeout). A competent agent would try approach A → fail → diagnose the failure → try approach B.
The approach — dataset construction
We constructed a dataset of 50 AI-generated action logs from an autonomous web browsing agent tasked with realistic information retrieval and task completion goals. The action logs were generated using an open-source web browsing agent framework operating on live websites.
Each action log recorded: the goal, every navigation action (URL visited, element clicked, text entered), the agent's internal reasoning at each step (chain-of-thought), the final output, and whether the task was marked as completed or failed by the agent.
The evaluation framework — grading every step, not just the result
This is the core methodological contribution. We designed a 5-dimension evaluation framework that annotators applied to every step of every action log — not just the final outcome.
Goal decomposition quality
Does the agent correctly break the high-level goal into appropriate sub-tasks? Are the sub-tasks in a logical order? Are any necessary sub-tasks missing? Are there unnecessary sub-tasks that waste actions?
Score 1–5 per action log · 1 = incoherent · 5 = optimalAction justification
For each action the agent takes, is there a clear logical reason for that specific action at that point in the sequence? Does the agent's chain-of-thought reasoning actually justify the action, or is the reasoning post-hoc rationalisation for an action taken based on pattern matching?
Fully justified · Partially justified · Unjustified — per individual actionInformation verification
When the agent extracts information from a web page, does it actually read and correctly interpret the page content? Or does it substitute information from its training data, ignore contradictory evidence on the page, or misread the page structure? Each extraction step is verified against the actual page content captured as a screenshot at the time of the action.
Verified · Unverified · Contradicted — per extraction stepError recovery quality
When the agent encounters an error (page not found, element not clickable, unexpected content), does it diagnose the error and adjust its approach, or retry the same failing action?
Intelligent recovery · Brute force · No recovery · N/ALogical coherence across the full chain
Does the entire sequence of actions tell a coherent story from goal to completion? Are there logical inconsistencies where the agent's behaviour in step 8 contradicts its reasoning in step 3? Does the agent maintain context across a long action sequence, or does it lose track of what it has already done?
Score 1–5 per action log · 1 = internally contradictory · 5 = fully coherentAnnotator selection and calibration
Multi-step reasoning evaluation requires a fundamentally different annotator profile than standard RLHF preference annotation. We selected annotators with software engineering background (ability to read action logs and trace execution flows), experience with web technologies, and logical reasoning ability tested via a logic assessment during screening.
All annotators evaluated the same 5 action logs independently using the 5-dimension framework before live annotation began.
Goal decomposition 0.76 Action justification 0.71 (lowest — "fully vs. partially justified" is subjective) Information verification 0.83 (highest — either matches page content or it does not) Error recovery quality 0.79 Overall logical coherence 0.74 ──────────────────────── Mean across dimensions 0.77 (above 0.70 threshold for production-grade annotation)
The results — what the evaluation found
The most important finding: 31 of 50 action logs were marked "successful" by outcome-based evaluation. But only 11 of those 31 achieved success through fully sound reasoning. Standard RLHF would have marked all 31 as positive training signal. Multi-step evaluation identifies only 11 as genuinely positive signal and flags the other 20 for specific reasoning corrections.
Detailed breakdown — logical leaps
Logical leaps were the most common failure mode, appearing in 68% of action logs. The agent is asked to find the current price of a specific laptop on an e-commerce site.
Corrective training signal: The annotator rewrote the reasoning chain for Step 4 to include the missing verification step. Multiplied across hundreds of similar examples, this trains the agent to always verify against the current page rather than substituting from memory.
Detailed breakdown — hallucinated actions
Hallucinated actions appeared in 24% of logs — the agent claimed to have performed an action that the execution trace does not support. The agent is asked to compare prices across three different websites.
Detailed breakdown — correct result, flawed reasoning
This is the finding that demonstrates why outcome-based evaluation is insufficient for agentic AI. 38% of action logs that produced a correct final result did so through flawed reasoning. In every case, standard RLHF would have marked the log as a positive training example — reinforcing the flawed reasoning pattern.
The training signal comparison
The multi-step evaluation produces 20 corrected training examples that outcome-based evaluation would have treated as positive signal without correction. These 20 corrections — showing the flawed reasoning chain alongside the corrected version — are the highest-value training data for improving an agent's reasoning capabilities. The model learns not just what the right answer is, but what the right reasoning process looks like.