Paper

SteerDistill: Reviving the Judgment of Quantized Agents through Teacher Steering

Agents call tools multiple times to finish a task, so inference cost is deployment cost. Low-bit quantization reduces that cost, but a quantized agent produces different environment transitions from a successful full-precision run, which can lower the task success rate. SteerDistill has the teacher point out this mismatch, lets the student fix it itself, and then instills only the verified corrections into the student through distillation.

To summarize the results, on three models and five agent benchmarks the method scores 1.67–2.18 points higher on average than the strongest quantized baseline at W4A16, and narrows the gap to the BF16 reference to within 1 point. On τ²-Bench it achieves a 1.84%p higher success rate than QeRL with 67% less training GPU time (Figure 3a). The figure below shows the overall flow and the key performance at a glance.

SteerDistill overview: a teacher points out where the run went off track, the student fixes it itself, and the correction is internalized through distillation, together with a summary of average scores across five benchmarks and performance versus training GPU time

Figure 1 of the original paper (original PDF p.1) reads in two parts. The left side depicts a closed loop in which, in a failure context, the full-precision teacher spells out the requirement the student missed so that the student can correct its own judgment, and then refines the guidance using execution feedback. The right panel (a) shows the average score across five benchmarks approaching BF16 when Qwen3-8B is quantized to W4A16, and (b) shows a higher τ²-Bench success rate than QeRL with 67% less training GPU time. The project page and code are available at https://air-embodied-brain.github.io/SteerDistill and https://github.com/air-embodied-brain/SteerDistill , respectively.

1. Quantization throws off an agent’s judgment

Quantization is a technique that stores a model’s weights (and, depending on the setting, activations) in fewer bits to reduce memory usage and inference cost. This paper’s notation W4A16 denotes a setting with 4-bit weights and 16-bit (BF16) activations. However, an agent is evaluated not on a single answer but on the entire multi-turn rollout (the record an agent leaves while performing a task), so what quantization can damage is not only sentence generation but also decision making.

The problem is a phenomenon the authors call “behavioral divergence.” A quantized agent misses a task requirement or an environment constraint and produces different environment transitions (how the environment state changes) from a successful full-precision reference (§1, Figure 1). Two concrete examples are reported. One is a case where a read tool can only read files in the current directory, yet the agent tries to read files inside a directory before entering it (§1, Figure 11). The other is a case where the agent creates all the requested files but leaves the contents of some of them empty (Figure 6).

Why this divergence is hard to fix has two branches. Training on offline data tends to miss the context in which the student actually fails (§2), and letting the student retry on its own rarely produces a correct response in that context. In the authors’ BFCL V4 multi-turn diagnostic, when the student retried four times without guidance on divergent turns, the fraction of turns that produced at least one correct action (turn-level Pass@4 — the fraction of turns where at least one of four attempts produced a correct action, with each attempt starting from the same pre-turn history and environment state) was only 9.6% (§1, footnote 1).

2. A small example: create the file first, then write the sentence

Let us follow the Figure 8 case from the paper to see where a requirement gets missed. The task is “create 1.txt, and append from dev_summary.txt the second full sentence containing ‘server error’”; the first matching sentence is “No server error occurred during the initial phase.”, and the second is “However, a server error was detected in the final testing phase.” The table below is an illustrative example, adapted from the actual case in Figure 8 of the original paper (original PDF p.21), and is not a measurement result.

StepOriginal rollout (no guidance)After teacher steering
InputSame task + initial state with no filesSame task + same initial state
First actionAttempts to write directly to the nonexistent 1.txtCreates an empty file as guided
Tool returnRejection: cannot write to '1.txt': No such fileSuccess: file created
Remaining stateInitial state unchanged, file still absentA new empty 1.txt exists
VerificationNoneNew empty file matches the reference state → local verification passes
Next actionAfter creating the file, records a sentence fragment (“resolving the server error.”) rather than the matching sentenceRecords the second full sentence verbatim without guidance
OutputA file that does not contain the required sentenceA file that contains the second full sentence

The original rollout took the order “write rejection → file creation → sentence fragment recording,” and after teacher steering it was corrected to “file creation → local verification → full sentence recording.” What changed is that the precondition “create the file first” was stated explicitly in the guidance, and once the student fixes that condition through its own action, it continues even without guidance. This structure — “fixed with guidance, then maintained without it” — runs through the entire paper.

Behavior change before and after teacher steering in a file-writing task: originally the agent creates the file after a write failure and records a sentence fragment that misses the requirement, but after teacher steering it creates the file first, passes verification, and then records the full sentence

3. Method: fix divergent turns with a closed loop, then instill them through distillation

SteerDistill’s training repeats two stages every cycle. First, teacher-steered trajectory revision repairs the failed rollout, and then internalization of the steered behavior etches that correction into the student’s weights (§3, Figure 2).

The setup is as follows. A frozen full-precision teacher $\pi_T$ and a trainable quantized student $\pi_\theta$ start from the same checkpoint. Here $\theta$ is the BF16 master weights maintained during training, and the quantization operator $Q$ is fixed. Every cycle $k$, the frozen snapshot $\pi_{\theta_k}$ generates and revises rollouts, and distillation updates the learner $\pi_\theta$ to produce the next cycle’s snapshot (§3). At inference time only the quantized student is used, and neither the teacher nor the guidance prompt intervenes (§4.1).

3-1. Teacher-steered trajectory revision

1. Find the divergent turn. Prepare a successful teacher reference trajectory from the same task and initial state, and align the student and the reference by environment state transitions. Because alignment is by “how the environment changed after execution” rather than by action strings, superficially different actions that produce the same transition are treated as identical. Intervention happens at the student turn $t^\star$ where the transitions first diverge.

2. The teacher builds guidance and the student tries to fix it. The teacher compares the failed response with the successful reference and builds a guidance prompt $G$ that points out the requirement or environment constraint the student missed. $G$ is divided into what went wrong (the mismatch), the condition under which the guidance applies, and the correction rule. Conditioned on this $G$, the frozen student generates a revised response and actually executes it.

3. Judge by local verification. The environment state and return information produced by the revised response are compared with the reference. If they match, it is accepted; otherwise, the execution feedback is reflected in the guidance and the attempt is retried from the same pre-error state. The attempt budget per intervention is $R=4$ (Appendix B).

4. Keep only verified corrections. If the revised response passes, that turn is replaced and the student continues without guidance thereafter. If it still fails, realign with the reference and find the next divergence point. Even if the task ultimately fails, verified corrections are kept for distillation (§3.1).

Summarizing what is stored and what changes in this loop gives the following.

Iteration stepInputRemaining stateNext action
Locate intervention pointFailed rollout + fixed successful referenceThe turn $t^\star$ where the two trajectories first diverge, plus the history and environment state before itRewind to the state before $t^\star$
Generate and execute guidanceFailed response + reference + previously approved guidance memoryThe environment transition executed by the student conditioned on the teacher-built guidance $G$Judge by local verification
Verification failureExecution result + verification report $f$The failed attempt and the report accumulate in the feedback history $F$The teacher summarizes $F$, refines $G$, and retries from the same state (budget $R=4$)
Verification passThe passing revised responseThe divergent turn is replaced, and the suffix after it is discardedThe student continues anew without guidance
Termination(success / intervention budget exhausted / no further divergence to align)The current trajectory, the verified corrections, and their guidanceHanded to distillation. Corrections are preserved even if the task failed

What local verification compares differs by benchmark. In AgentWorldModel, the database state before and after execution must match the reference, responses that cause environment errors are rejected, and read actions must return the same information as the reference. For termination responses, the reference must also terminate and the candidate must pass the task verifier (Appendix A). In ALFWorld, WebShop, and Search QA, a candidate is accepted if the post-execution state matches the reference or both executions reach task success. In other words, the core of the check is “is the transition the same?”

The name “closed loop” comes from the fact that the teacher does not give guidance once and stop, but looks at the student’s execution results and repeatedly refines the guidance. In the same diagnostic, this steering raised turn-level Pass@4 on divergent turns from 9.6% to 82.3% (§1).

3-2. Distillation that makes it work without steering

Building the corrected trajectory alone is not enough. At inference time there must be neither a teacher nor a guidance prompt, so the student must have “the correct behavior when guided” etched into its weights so that it can reproduce it without guidance (§3.2).

To this end, the source of the supervision signal is split by turn. On revised turns, the distribution the learner predicts without $G$ is made to follow the distribution the steered snapshot predicts when it sees $G$. On the remaining turns (including subsequent turns), the learner follows the full-precision teacher’s distribution. In both cases the learner is not given the guidance prompt. The objectives for revised and other turns can be written as follows (§3.2).

Revised turns (forward KL, where $c$ is the dialogue context and response prefix without the guidance):

$$D_{\mathrm{KL}}\left(\pi_{\theta_k}(\cdot\mid c,G)\ \|\ \pi_\theta(\cdot\mid c)\right)$$

Other turns (reverse KL):

$$D_{\mathrm{KL}}\left(\pi_\theta(\cdot\mid c)\ \|\ \pi_T(\cdot\mid c)\right)$$

The paper only states that the rationale for choosing the two KL directions follows prior work [26] and does not explain the detailed reason (§3.2). Therefore, what effect each direction had cannot be isolated from this paper’s description alone, and the Q5 ablation later only shows that the routing combination affects the score. The loss is assigned per turn over all retained response tokens and averaged (§3.2).

One training cycle of SteerDistill: a closed-loop teacher steering repairs divergent turns, and the two stages internalize them by distilling revised turns from the steered snapshot and other turns from the full-precision teacher

Figure 2 of the original paper (original PDF p.4) shows one full cycle. (i) is the loop of teacher steering → execution → local verification → refining the guidance with feedback and retrying on failure, keeping only verified corrections and then continuing without guidance. (ii) is the distillation stage, which targets the steered snapshot’s predictions for revised turns and the full-precision teacher’s predictions for other turns. The key point of this figure is that the learner receives no guidance prompt in either case.

4. Results

4-1. Accuracy: recovering almost all of the lost performance

When the same Qwen3-8B is quantized to W4A16, how much does SteerDistill recover relative to the best baseline? The table below summarizes the scores on five benchmarks and their average, showing full-precision BF16 alongside the strongest quantized baseline (QeRL) (reconstructed from the Qwen3-8B W4A16 row of Table 1).

MethodBFCL V4τ²-BenchWebShopALFWorldSearch QAAverage
BF16 (reference)46.8932.6468.5879.9239.6553.54
QeRL (best baseline)44.9129.8465.2977.0437.9451.00
SteerDistill46.8431.7968.4179.7539.1253.18

SteerDistill’s average of 53.18 is 2.18 points above QeRL and only 0.36 points from the BF16 reference of 53.54, recovering almost all of it. It also leads every quantized baseline on each of the five benchmarks (§4.2). The advantage holds across models: 1.71 points above QeRL on Qwen3.5-4B and 1.67 points on Gemma 4 12B-it, with all three models within 1 point of BF16 (§4.2, Table 1). At lower precision the gap widens: 1.93 points above QeRL at W4A8 and 3.39 points above On-Policy QAD at W2A16. The only exception is a single W2A16 Search QA metric, where it trails BitDistiller (34.22) by 0.17 points (§4.2).

Secondary metrics keep the same ordering. τ²-Bench Pass@4 is 53.05 (QeRL 50.10), WebShop Task Score is 78.57 (76.08), and Search QA average EM is 39.12 (37.94); BF16 is 54.20, 78.69, and 39.65, respectively. On Search QA it leads the quantized baseline not only in-domain (NQ, HotpotQA) but also on the other five out-of-domain datasets (Table 6, §C.2).

4-2. Efficiency: training time and output tokens

If the accuracy gain came from training longer or talking more, its significance would be halved. The figure below separates training cost from inference cost.

Efficiency analysis on Qwen3-8B W4A16: (a) τ²-Bench success rate versus training GPU time, (b) success rate versus average output tokens per successful task

Figure 3 of the original paper (original PDF p.7) has two panels. In panel (a), the x-axis is training GPU time (based on 8 A100 80GB GPUs, including teacher steering and verification time) and the y-axis is τ²-Bench success rate. SteerDistill reaches 31.68% at 32 GPU hours, higher than On-Policy QAD (28.75%), which converged at about 40 GPU hours, and achieves a 1.84%p higher success rate than QeRL (29.84%), which converged at 96 GPU hours, with 67% less GPU time (§4.3). The paper writes the difference versus On-Policy QAD as 2.91%p, but the arithmetic difference between the 31.68% and 28.75% shown in the figure is 2.93%p. In panel (b), the x-axis is average output tokens per successful task and the y-axis is success rate. SteerDistill uses 4.0K tokens and achieves a higher success rate than quantized baselines using 4.1K–5.2K, close to BF16’s 3.9K (§4.3). In other words, the trajectories of successful tasks are shorter.

Averaging over successful tasks only, this difference shows up across several metrics. Note that each number is an average over only the tasks that method succeeded on, so different task sets are being compared (Table 7).

MethodInput tokensOutput tokensInteraction stepsTool calls
BF16185K3.9K11.44.2
GPTQ270K5.2K13.85.25
On-Policy QAD214K4.2K12.34.55
QeRL206K4.1K12.14.43
SteerDistill192K4.0K11.64.24

SteerDistill uses fewer input tokens, interaction steps, and tool calls than every quantized baseline in the table. Training that precisely fixes the divergence points may have reduced wasted retries, but this is a consistent direction in the numbers, not a causal interpretation. The same pattern repeats at W4A8 and W2A16 (Figure 4, §C.3).

4-3. How much do the design choices contribute?

How much each design choice actually contributes is checked by an ablation on BFCL V4 (Qwen3-8B, W4A16). Below is a reconstruction of Table 2, and every difference is the value subtracted from the full SteerDistill score of 46.84.

GroupVariantScoreDifference
(reference)BF16 / On-Policy QAD / full SteerDistill46.89 / 44.43 / 46.84—
Q1 intervention locationRandom turn43.52−3.32
First action mismatch45.01−1.83
Q2 intervention modeStudent retry without guidance44.46−2.38
Teacher takeover45.06−1.78
Q3 guidance format/sourceTeacher / free-form44.78−2.06
Student / structured41.74−5.10
GPT-5.6-Sol / structured45.27−1.57
Q4 retention criterionNo local verification42.18−4.66
Require final success only46.30−0.54
Q5 distillation routingAll reverse KL45.29−1.55
All forward KL44.58−2.26
Reversed routing (RKL/FKL)44.28−2.56

The reading is as follows. Intervening at the divergence point in environment state rather than at the first action mismatch is 1.83 points higher, and intervening at an arbitrary turn (43.52) is even lower than On-Policy QAD (44.43). Steering the student instead of having the teacher inject the answer directly is 2.38 points higher than retrying and 1.78 points higher than teacher takeover. Structured teacher guidance is 2.06 points higher than free-form, 5.10 points higher than student-generated guidance, and 1.57 points higher than structured guidance produced by GPT-5.6-Sol. Removing local verification drops the score sharply from 46.84 to 42.18, while a filter that requires final success after verification actually lowers it slightly to 46.30. For the distillation direction, the combination of forward KL on revised turns and reverse KL on the rest is 1.55 points higher than all-reverse and 2.26 points higher than all-forward, and reversing the routing lowers it by 2.56 points (§4.4).

The effect of guidance format shows up as a state change in the ticket-creation case (Figure 12). The task is to create a priority-3 ticket, and the transaction session was authenticated while the ticket session was not.

Guidance format/sourceAction the student continued withRemaining state
Teacher / structuredLog in, then create ticket1 ticket, no credentials in the description
Teacher / free-formClaims completion after logging in, without creatingTicket queue is empty
Student / structuredLog in, then create ticket1 ticket, no credentials in the description
GPT-5.6-Sol / structuredLog in, then create ticket1 ticket, login credentials included in the description
No guidance (original rollout)Retries after a login error and succeeds1 ticket, credentials included in the description

GPT-5.6-Sol’s structured guidance got the login order right, but the student put the credentials into the ticket description and only partially followed the instruction. The free-form guidance ended the attempt with “the ticket has already been created,” leaving no actual ticket, which is a failure that local verification must catch. In this case, then, the score difference shows up not as an abstract metric but as the state “does a ticket remain?”

5. Limitations and when to apply it

To judge applicability, the constraints must be separated. The limitations the authors state are collected in Appendix E. First, teacher-steered trajectory revision requires a successful teacher reference trajectory. Without a reference, a failed rollout remains only as distillation data and receives no correction. Second, the dialogue history and environment state before the error must be reversible. Third, there must be reliable local verification to judge the effect of the revised response. The authors state that application in environments where state restoration or local verification is difficult has not been validated. Fourth, local verification only checks the effect of the revised response in the current context; it neither guarantees final success nor that every reasoning step in the response is correct.

Beyond that, there are a few conditions a reviewer would point out (not stated by the authors). Training requires a full-precision teacher, so if a BF16 checkpoint cannot be obtained, applying it as-is is difficult. It is also designed for tool-use benchmarks where environment state can be read and rewound, so it does not carry over unchanged to open-ended dialogue where “state” is not well defined. The experiments start from checkpoints that have undergone per-task GRPO post-training. In the appendix experiment that starts without GRPO, the relative ordering is maintained, but the absolute scores of all methods drop (Table 5). The reported absolute performance therefore depends on this initial checkpoint.

Method (Qwen3-8B, W4A16)BFCL V4τ²-Bench (Pass@1)
BF1640.4030.00
BitDistiller35.3424.29
On-Policy QAD36.1624.58
QeRL38.8528.92
SteerDistill39.6729.05

In other words, the method recovers a large share of the capability lost to quantization, but that share is measured on top of an initial checkpoint that has gone through GRPO. Overall, SteerDistill is a method with clear conditions for low-bit deployment of tool-use agents that have “a full-precision teacher, a reversible environment, and reliable verification.” If those conditions break down, the benefit may shrink or application itself may be difficult, and this remains an unvalidated area.

Tables from the paper

Tables converted mechanically from the arXiv e-print LaTeX source. The numbers are the paper’s own and did not pass through a model.

Table 1. Agentic benchmark results across models and quantization settings. Scores are percentages; higher is better. Avg. is the unweighted mean of the five displayed benchmark scores. Bold marks the best quantized result for each model and precision.

ModelPrecisionMethodBFCL V4 Overall$\tau^2$-Bench Pass@1WebShop SRALFWorld SRSearch QA EMAvg.
Qwen3 8BBF16–46.8932.6468.5879.9239.6553.54
W4A16GPTQ40.7326.8559.2471.3233.6446.36
EfficientQAT41.7728.0061.3473.1835.0847.87
BitDistiller42.4628.3162.0773.9135.6948.49
On-Policy QAD44.4328.7764.7276.5337.4750.38
QeRL44.9129.8465.2977.0437.9451.00
Ours46.8431.7968.4179.7539.1253.18
W4A8GPTQ35.9022.9054.7066.8030.2042.10
EfficientQAT40.2027.1060.0071.8034.0046.62
BitDistiller41.2027.8060.8072.9034.8047.50
On-Policy QAD43.1527.8563.3575.1036.2549.14
QeRL42.8529.0564.1075.5536.9049.69
Ours45.2030.1066.7078.2537.8551.62
W2A16GPTQ13.846.7230.2124.6318.9118.86
EfficientQAT30.9218.4449.1353.8731.4636.76
BitDistiller33.7119.8651.8457.3634.2239.40
On-Policy QAD32.8820.5753.0858.4132.9639.58
QeRL31.6420.9150.7455.6231.5838.10
Ours38.1223.6855.7163.2734.0542.97
Qwen3.5 4BBF16–50.4062.8069.8078.6046.8061.68
W4A16GPTQ43.9156.1860.6569.7840.5854.22
EfficientQAT45.3657.6862.7671.8442.0855.94
BitDistiller46.1758.3363.6172.5442.7756.68
On-Policy QAD48.0259.4866.0374.9244.4958.59
QeRL48.5360.3166.7375.6345.0159.24
Ours49.8261.9169.0877.8246.1160.95
Gemma 4 12B-itBF16–64.1861.4672.4184.6342.9665.13
W4A16GPTQ57.8254.3663.4875.7236.9157.66
EfficientQAT59.4656.1265.7277.6838.2459.44
BitDistiller60.0856.7466.4378.3138.7960.07
On-Policy QAD61.8758.3668.6180.7640.5262.02
QeRL62.4159.0269.3381.3541.0762.64
Ours63.5460.7271.3883.6242.3164.31

Table 2. Ablation study on BFCL V4 using Qwen3-8B under W4A16. BF16 and On-Policy QAD provide reference scores. Each Q1–Q5 group lists alternatives to its stated default; all defaults together form the full-method row. Bold marks the best quantized result in each column. The Final success only variant adds terminal-success filtering after local verification. app:steering_case shows a case study for Q3.

ConfigurationOverallNon-LiveLiveMulti- TurnWeb SearchMemoryIrrel.
BF1646.8987.1481.7648.5318.5719.3278.63
On-Policy QAD44.4385.7280.1945.1416.8617.1674.88
Full SteerDistill\xspace (Ours)46.8488.2981.7148.4718.4418.7978.56
Q1: Intervention localization (default: Behavioral divergence)
Random eligible turn43.5285.2079.7244.0016.1016.3073.48
First action mismatch45.0186.2080.5545.9517.2017.4576.20
Q2: Intervention strategy (default: Teacher steering)
Unguided student retry44.4685.8180.0845.2616.9317.0574.97
Teacher takeover45.0686.3180.6946.0217.3117.5575.81
Q3: Steering format and source (default: Teacher / Structured)
Teacher / Free-form44.7887.1080.8545.3517.1017.0575.50
Student / Structured41.7483.7279.0241.4814.6614.9171.08
GPT-5.6-Sol / Structured45.2787.0580.9046.1517.5517.6575.90
Q4: Revision retention (default: Local verification)
No local verification42.1884.1078.9542.3014.9515.2071.55
Final success only46.3087.9181.0947.7218.1618.2778.02
Q5: Distillation routing (default: FKL at corrected turns; RKL elsewhere)
Reverse KL throughout45.2986.4580.9346.3317.4817.7876.02
Forward KL throughout44.5885.9880.4745.2217.2816.8975.32
Reversed routing (RKL / FKL)44.2885.6480.2844.7516.9616.8974.93

Table 3. Key notation for SteerDistill\xspace. The revision algorithm handles one trajectory; training adds index $i$.

SymbolMeaning
Policies and conditioning
$\pi_T$Frozen full-precision teacher.
$\theta$, $\pi_\theta$Student master weights and trainable quantized policy.
$k$, $\theta_k$, $\pi_{\theta_k}$Training cycle, frozen student weights, and corresponding policy.
$i$, $t$, $j$Trajectory, turn, and response-token indices.
$\tau$, $\tau^T$Student trajectory and its fixed successful teacher reference.
$h_t$, $y_t$History before turn $t$ and the response generated at that turn.
$c_{i,t,j}$Token context $(h_{i,t},y_{i,t,\lt j})$, with all steering prompts removed.
Revision state and retained supervision
$t^\star$Intervention turn selected by alignment at the current revision step.
$\rho$, $\rho^T$Candidate transition and its aligned reference transition.
$R$, $v$, $f$, $F$Attempt budget, acceptance flag, verification report, and accumulated feedback.
$G$, $G_{i,t}$Current steering prompt and the prompt retained for turn $t$ in trajectory $i$.
$\mathcal M$Memory holding the latest accepted steering prompt within the current trajectory.
$\mathcal G$Stored prompt map: $\mathcal G[t]=G_{i,t}$ for trajectory $i$.
$\mathcal I_i$Retained corrected turns in trajectory $i$; these receive FKL supervision.
Training objective
$\mathcal A_k$Retained assistant-token positions $(i,t,j)$ in cycle $k$.
$\ell_{\mathrm{FKL}}$, $\ell_{\mathrm{RKL}}$Per-token losses at corrected and other turns (eq:forward_kleq:reverse_kl).
$\mathcal L^{(k)}(\theta)$Mean token loss over $\mathcal A_k$, assigned by turn (eq:steerdistill_objective).

Table 4. Hyperparameters for the low-bit training stage. Batch size counts complete trajectories. For EfficientQAT, $1+1$ denotes one epoch of Block-AP followed by one epoch of E2E-QP.

MethodEpochsLearning RateBatch Size (trajectories)
EfficientQAT$1+1$$8\times10^{-6}$16
BitDistiller2$8\times10^{-6}$16
On-Policy QAD2$8\times10^{-6}$16
QeRL2$8\times10^{-6}$16
Ours2$8\times10^{-6}$16

Table 5. Results without initial GRPO post-training on Qwen3-8B. Quantized methods use W4A16. Scores are percentages; bold marks the best quantized result.

ModelPrecisionMethodBFCL V4 Overall$\tau^{2}$-Bench Pass@1
Qwen3 8BBF16–40.4030.00
W4A16BitDistiller35.3424.29
On-Policy QAD36.1624.58
QeRL38.8528.92
Ours39.6729.05

Table 6. Additional evaluation metrics for Qwen3-8B. Quantized methods use W4A16, with BF16 as the full-precision reference. On $\tau^{2}$-Bench, Pass@4 is the fraction of tasks solved at least once in four runs. We report Task Score for WebShop and per-dataset Exact Match (EM) for Search QA. $\dagger$ and $\star$ denote in-domain and out-of-domain Search QA datasets, respectively. Higher is better for all metrics; bold marks the best quantized result in each column.

Method$\tau^{2}$-Bench Pass@4WebShop ScoreSearch QA (EM) NQ$^{\dagger}$Search QA (EM) TriviaQA$^{\star}$Search QA (EM) PopQA$^{\star}$Search QA (EM) HotpotQA$^{\dagger}$Search QA (EM) 2Wiki$^{\star}$Search QA (EM) MuSiQue$^{\star}$Search QA (EM) Bamboogle$^{\star}$
BF1654.2078.6943.3662.4241.9636.5835.1215.9242.18
GPTQ45.5070.8637.6256.0436.0530.4428.3912.2334.71
EfficientQAT47.1072.7339.0857.6937.4832.0230.1712.8836.25
BitDistiller47.5573.4139.7158.3138.1132.7130.8613.2136.92
On-Policy QAD48.2075.6141.1860.2239.6934.4732.8614.4639.44
QeRL50.1076.0841.6460.7340.1335.0333.3614.7939.92
Ours53.0578.5742.8461.8841.4136.0134.5515.5341.61

Table 7. Detailed trajectory and inference statistics for Qwen3-8B on $\tau^{2}$-Bench. Quantized methods use W4A16, with BF16 as the full-precision reference. All entries are averages over each method’s own successful tasks. Cached-input token counts are reported descriptively; K denotes thousands of tokens.

MethodInput TokensCached InputOutput TokensInteraction StepsTool Calls
BF16185K148K3.9K11.44.2
GPTQ270K223K5.2K13.85.25
EfficientQAT243K199K4.7K13.14.92
BitDistiller235K192K4.5K12.94.78
On-Policy QAD214K173K4.2K12.34.55
QeRL206K166K4.1K12.14.43
Ours192K153K4.0K11.64.24

Figures in this post are taken from the original arXiv:2611.13410 (CC BY 4.0). Only size and format were changed.

License

Author: Jaehun Ryu

Link: https://jaehun.me/en/posts/paper-2611-13410/

License: CC BY 4.0

This work is licensed under the Creative Commons Attribution 4.0 International License. You are free to use it for any purpose, including commercial use, as long as you provide proper attribution.

Comments