LangWatch
Sign Up

OpenAI Decisions API beats Jev on real-world checks

LangWatch LLM-as-a-Judge Bench evaluates model capabilities in doing evals, agent simulations and trace search, our specialty. Core and long conversations are real-world based, hard cases synthetic; none is public, to avoid benchmaxing and measure true capability for our customers.

This benchmark was created using LangWatch. Sign up to create your own · Read the method

LangWatch Instant Evals runs on Jev. Check it out

Task
Set

All three jobs, core

Accuracy, % · Higher is better

OpenAI Decisions API is clearly ahead of Jev by 9.7 points, even across 30 comparisons with Jev: the 95% interval of the gap runs from 6.4 to 13.1 points.

  1. OpenAI Decisions API73.0
    , interval 70.0 to 76.2, ahead alone
  2. Mercury Decide68.0
    , interval 64.4 to 71.5, tier 2
  3. Jev63.3
    , interval 59.9 to 66.7, tier 3
  4. deck-31B63.1
    , interval 59.9 to 66.3, tier 3
  5. Kev-27B62.8
    , interval 59.4 to 66.5, tier 3
  6. Clef62.2
    , interval 58.9 to 65.6, tier 3
  7. Decision 2.0 Vega 27B61.0
    , interval 57.4 to 64.9, tier 3
  8. pplx-decider v1.1 27B60.7
    , interval 56.8 to 64.4, tier 3

Ahead aloneTied for the leadThe rest95% interval

One job at a time

All three jobs, core: Accuracy, higher is better

  1. OpenAI Decisions API73.0%, interval 73.0% [70.0, 76.2], tier 1
  2. Mercury Decide68.0%, interval 68.0% [64.4, 71.5], tier 2
  3. Jev63.3%, interval 63.3% [59.9, 66.7], tier 3
  4. deck-31B63.1%, interval 63.1% [59.9, 66.3], tier 3
  5. Kev-27B62.8%, interval 62.8% [59.4, 66.5], tier 3
  6. Clef62.2%, interval 62.2% [58.9, 65.6], tier 3
  7. Decision 2.0 Vega 27B61.0%, interval 61.0% [57.4, 64.9], tier 3
  8. pplx-decider v1.1 27B60.7%, interval 60.7% [56.8, 64.4], tier 3
  9. GLiDE58.6%, interval 58.6% [55.0, 62.3], tier 3
  10. Quyet-1.0-Large58.4%, interval 58.4% [54.9, 62.1], tier 3
  11. Rene-1 31B FP858.4%, interval 58.4% [54.4, 62.3], tier 3
  12. H2O-Lightning-4B v1.2.257.4%, interval 57.4% [53.3, 61.5], tier 4
  13. AutoJev-27B55.6%, interval 55.6% [51.9, 59.4], tier 4
  14. Winnow55.3%, interval 55.3% [51.7, 59.1], tier 4
  15. Jev-Omni52.6%, interval 52.6% [48.6, 56.7], tier 4
  16. decisio v0.8.051.9%, interval 51.9% [48.1, 56.0], tier 4
  17. Decision 2.0 Nox 4B51.7%, interval 51.7% [47.6, 56.0], tier 4
  18. Cygnet51.1%, interval 51.1% [47.2, 55.2], tier 4
  19. decider-4b v250.0%, interval 50.0% [45.8, 54.3], tier 5
  20. JevK548.9%, interval 48.9% [45.1, 52.7], tier 5
  21. Julia-148.8%, interval 48.8% [45.5, 52.1], tier 5
  22. Microsoft Decision 148.8%, interval 48.8% [44.6, 53.2], tier 5
  23. DiffusionGemma-Jev48.7%, interval 48.7% [44.5, 53.0], tier 5
  24. Clef Omni48.4%, interval 48.4% [44.5, 52.5], tier 5
  25. Hopper 12B48.3%, interval 48.3% [44.5, 52.5], tier 5
  26. Decision 2.0 Eos 0.8B46.3%, interval 46.3% [42.1, 50.5], tier 5
  27. Decision 2.0 Lux 9B45.4%, interval 45.4% [41.5, 49.6], tier 5
  28. Clef Flash45.1%, interval 45.1% [40.8, 49.3], tier 5
  29. Decision 2.0 Sol 2B45.1%, interval 45.1% [40.8, 49.5], tier 5
  30. Decision 2.0 Kai 0.6B43.4%, interval 43.4% [39.1, 47.5], tier 5
  31. Lev 4B42.1%, interval 42.1% [38.1, 46.2], tier 6
Dot: accuracy. Line: its 95% interval. Higher is better. Orange: ahead by the tie test; green: tied for the lead. Axis starts at 30%, not 0.
Plot

All three jobs, core: accuracy (higher is better) vs cost

Orange: most accurate decision model; blue: LLMs for reference

↖ Better
  • Jev
  • OpenAI Decisions API
  • Mercury Decide
  • Opus
  • Sol
  • Luna
  • Flash
← Cost per 1,000 decisions, USD (log scale). Lower is betterHigher is better ↑

Blue LLMs are not ranked. Opus and Sol (hollow): an estimate at API list price. LLM cost is one figure over all three sets.

Dashed line: the Pareto frontier (no other model is both better and cheaper). Hollow mark: an estimate (~).

Jev alternatives, every job

Every visible model on every task, sorted by All three jobs, core. Value with half its interval; bands are tie tiers.
Model
Tier 1 on All three jobs, core
OpenAI Decisions API73.0±3.1 · T173.0% [70.0, 76.2], tier 171.7±5.6 · T171.7% [66.1, 77.2], tier 170.4±4.4 · T170.4% [66.0, 74.8], tier 183.3±6.9 · T183.3% [76.1, 89.9], tier 170.8±4.6 · T270.8% [66.2, 75.3], tier 271.9±8.7 · T171.9% [62.8, 80.2], tier 178.1±6.9 · T178.1% [71.3, 85.1], tier 162.5±7.6 · T262.5% [55.1, 70.2], tier 275.0±3.4 · T275.0% [71.7, 78.5], tier 277.0±5.5 · T177.0% [71.3, 82.2], tier 174.5±5.9 · T274.5% [68.4, 80.2], tier 273.5±6.4 · T273.5% [67.0, 79.9], tier 2
Tier 2
Mercury Decide68.0±3.6 · T268.0% [64.4, 71.5], tier 261.7±6.2 · T261.7% [55.6, 67.9], tier 266.3±4.7 · T166.3% [61.5, 71.0], tier 183.3±7.4 · T183.3% [75.5, 90.3], tier 178.1±4.4 · T178.1% [73.6, 82.4], tier 177.1±7.8 · T177.1% [69.2, 84.7], tier 180.2±8.1 · T180.2% [71.7, 88.0], tier 177.1±6.9 · T177.1% [70.2, 84.0], tier 178.7±3.4 · T278.7% [75.3, 82.1], tier 279.0±6.5 · T179.0% [72.2, 85.2], tier 176.0±6.0 · T176.0% [69.8, 81.8], tier 181.0±5.4 · T281.0% [75.7, 86.5], tier 2
Tier 3
Jev63.3±3.4 · T363.3% [59.9, 66.7], tier 361.3±6.6 · T261.3% [54.8, 67.9], tier 263.3±4.9 · T263.3% [58.2, 68.1], tier 266.7±9.0 · T266.7% [57.3, 75.3], tier 272.6±4.7 · T272.6% [67.9, 77.3], tier 271.9±7.9 · T171.9% [63.8, 79.6], tier 172.9±9.0 · T172.9% [63.7, 81.6], tier 172.9±7.9 · T172.9% [65.0, 80.8], tier 172.7±3.5 · T372.7% [69.2, 76.3], tier 372.5±5.8 · T272.5% [66.7, 78.3], tier 269.0±6.6 · T269.0% [62.3, 75.5], tier 276.5±5.8 · T276.5% [70.4, 82.1], tier 2
deck-31B63.1±3.2 · T363.1% [59.9, 66.3], tier 359.6±5.6 · T259.6% [54.3, 65.6], tier 258.9±4.6 · T258.9% [54.3, 63.5], tier 281.9±6.7 · T181.9% [74.8, 88.1], tier 168.8±5.0 · T268.8% [63.8, 73.8], tier 268.8±7.9 · T168.8% [60.8, 76.6], tier 161.5±9.4 · T261.5% [52.0, 70.8], tier 276.0±7.6 · T176.0% [68.1, 83.3], tier 181.5±3.2 · T181.5% [78.4, 84.8], tier 180.0±6.4 · T180.0% [73.4, 86.2], tier 177.5±6.3 · T177.5% [71.0, 83.7], tier 187.0±4.5 · T187.0% [82.5, 91.5], tier 1
Kev-27B62.8±3.6 · T362.8% [59.4, 66.5], tier 359.1±7.0 · T259.1% [52.4, 66.3], tier 261.1±4.7 · T261.1% [56.4, 65.7], tier 274.3±7.8 · T174.3% [66.4, 82.0], tier 175.3±4.7 · T175.3% [70.8, 80.1], tier 171.9±7.8 · T171.9% [64.3, 79.8], tier 177.1±8.5 · T177.1% [68.4, 85.4], tier 177.1±7.7 · T177.1% [69.0, 84.4], tier 172.0±3.4 · T472.0% [68.5, 75.4], tier 472.0±6.4 · T272.0% [65.6, 78.4], tier 266.0±6.0 · T266.0% [59.7, 71.8], tier 278.0±6.2 · T278.0% [71.6, 84.0], tier 2
Clef62.2±3.4 · T362.2% [58.9, 65.6], tier 357.8±6.3 · T257.8% [51.6, 64.1], tier 260.2±4.6 · T260.2% [55.7, 64.9], tier 275.7±7.9 · T175.7% [67.4, 83.1], tier 176.0±4.7 · T176.0% [71.5, 80.8], tier 171.9±7.8 · T171.9% [64.0, 79.6], tier 184.4±6.9 · T184.4% [77.1, 90.8], tier 171.9±8.0 · T171.9% [63.8, 79.8], tier 174.0±3.5 · T374.0% [70.5, 77.5], tier 376.5±6.5 · T176.5% [69.9, 83.0], tier 166.5±5.7 · T266.5% [60.8, 72.2], tier 279.0±5.3 · T279.0% [73.6, 84.3], tier 2
Decision 2.0 Vega 27B61.0±3.7 · T361.0% [57.4, 64.9], tier 360.4±6.3 · T260.4% [54.3, 67.0], tier 258.3±5.3 · T258.3% [53.1, 63.7], tier 270.8±8.1 · T270.8% [62.4, 78.7], tier 280.2±4.3 · T180.2% [76.0, 84.6], tier 176.0±8.2 · T176.0% [67.7, 84.0], tier 184.4±6.4 · T184.4% [77.8, 90.6], tier 180.2±6.7 · T180.2% [73.4, 86.7], tier 175.8±3.5 · T275.8% [72.5, 79.4], tier 275.5±5.7 · T275.5% [69.8, 81.3], tier 271.0±6.2 · T271.0% [64.5, 77.0], tier 281.0±5.3 · T281.0% [75.5, 86.1], tier 2
pplx-decider v1.1 27B60.7±3.8 · T360.7% [56.8, 64.4], tier 361.3±5.7 · T261.3% [55.8, 67.1], tier 255.0±5.4 · T355.0% [49.5, 60.2], tier 377.8±7.6 · T177.8% [69.8, 84.9], tier 172.6±4.6 · T272.6% [68.2, 77.3], tier 274.0±8.0 · T174.0% [66.0, 81.9], tier 172.9±7.6 · T272.9% [65.2, 80.4], tier 270.8±8.0 · T170.8% [62.7, 78.7], tier 183.7±2.9 · T183.7% [80.8, 86.6], tier 184.0±5.0 · T184.0% [79.0, 88.9], tier 182.5±5.5 · T182.5% [77.0, 87.9], tier 184.5±4.9 · T184.5% [79.6, 89.4], tier 1
GLiDE58.6±3.6 · T358.6% [55.0, 62.3], tier 355.2±6.8 · T255.2% [48.6, 62.2], tier 263.9±4.4 · T263.9% [59.5, 68.3], tier 247.2±10.4 · T347.2% [36.9, 57.7], tier 379.5±4.9 · T179.5% [74.5, 84.4], tier 176.0±9.1 · T176.0% [66.7, 84.8], tier 185.4±6.3 · T185.4% [79.0, 91.7], tier 177.1±8.9 · T177.1% [67.7, 85.4], tier 174.3±3.5 · T374.3% [70.9, 78.0], tier 372.0±6.2 · T272.0% [65.7, 78.1], tier 271.5±6.7 · T271.5% [64.4, 77.8], tier 279.5±5.1 · T279.5% [74.4, 84.7], tier 2
Quyet-1.0-Large58.4±3.6 · T358.4% [54.9, 62.1], tier 361.3±5.8 · T261.3% [55.8, 67.3], tier 250.4±5.0 · T350.4% [45.3, 55.4], tier 379.2±6.9 · T179.2% [72.0, 85.9], tier 167.0±4.6 · T267.0% [62.6, 71.8], tier 266.7±8.0 · T266.7% [58.5, 74.5], tier 266.7±7.9 · T266.7% [58.7, 74.5], tier 267.7±6.9 · T167.7% [60.6, 74.5], tier 181.3±3.4 · T181.3% [78.0, 84.9], tier 181.0±5.8 · T181.0% [75.0, 86.6], tier 177.5±6.5 · T177.5% [70.6, 83.6], tier 185.5±5.6 · T185.5% [79.7, 91.0], tier 1
Rene-1 31B FP858.4±4.0 · T358.4% [54.4, 62.3], tier 355.2±6.7 · T255.2% [48.8, 62.2], tier 254.8±5.5 · T354.8% [48.9, 60.0], tier 375.0±8.0 · T175.0% [66.9, 82.9], tier 177.4±4.6 · T177.4% [72.9, 82.2], tier 177.1±8.7 · T177.1% [68.1, 85.4], tier 179.2±6.8 · T179.2% [72.2, 85.9], tier 176.0±7.7 · T176.0% [68.3, 83.7], tier 176.3±3.5 · T276.3% [72.9, 79.9], tier 279.5±6.0 · T179.5% [73.3, 85.3], tier 164.5±6.2 · T364.5% [58.2, 70.6], tier 385.0±5.0 · T185.0% [80.0, 90.0], tier 1
Tier 4
H2O-Lightning-4B v1.2.257.4±4.1 · T457.4% [53.3, 61.5], tier 457.4±6.5 · T257.4% [51.1, 64.1], tier 255.0±6.2 · T355.0% [48.7, 61.1], tier 365.3±8.8 · T265.3% [56.3, 73.8], tier 264.9±4.8 · T364.9% [60.3, 69.9], tier 369.8±8.5 · T169.8% [61.2, 78.1], tier 161.5±8.4 · T261.5% [53.1, 70.0], tier 263.5±7.5 · T263.5% [56.3, 71.3], tier 268.5±3.8 · T468.5% [64.8, 72.5], tier 467.5±6.8 · T267.5% [60.5, 74.0], tier 263.5±6.0 · T363.5% [57.4, 69.3], tier 374.5±6.3 · T274.5% [68.3, 80.9], tier 2
AutoJev-27B55.6±3.8 · T455.6% [51.9, 59.4], tier 452.6±6.7 · T252.6% [46.1, 59.6], tier 252.2±5.2 · T352.2% [47.0, 57.5], tier 371.5±8.2 · T271.5% [62.9, 79.4], tier 269.8±4.8 · T269.8% [65.0, 74.7], tier 266.7±8.3 · T166.7% [58.3, 75.0], tier 170.8±8.6 · T270.8% [62.2, 79.3], tier 271.9±6.9 · T171.9% [65.0, 78.7], tier 172.0±3.5 · T472.0% [68.7, 75.7], tier 471.5±6.6 · T271.5% [64.7, 78.0], tier 269.0±6.4 · T269.0% [62.2, 75.0], tier 275.5±5.6 · T275.5% [70.2, 81.3], tier 2
Winnow55.3±3.7 · T455.3% [51.7, 59.1], tier 456.5±6.0 · T256.5% [50.6, 62.7], tier 249.4±5.6 · T349.4% [43.9, 55.1], tier 372.2±8.3 · T172.2% [63.5, 80.1], tier 169.8±4.9 · T269.8% [64.9, 74.7], tier 268.8±8.9 · T168.8% [59.8, 77.7], tier 164.6±8.5 · T264.6% [56.3, 73.3], tier 276.0±6.9 · T176.0% [69.2, 83.0], tier 173.0±3.8 · T373.0% [69.4, 76.9], tier 370.5±6.2 · T270.5% [64.0, 76.3], tier 269.5±6.2 · T269.5% [63.2, 75.6], tier 279.0±5.9 · T279.0% [73.1, 85.0], tier 2
Jev-Omni52.6±4.0 · T452.6% [48.6, 56.7], tier 460.9±7.0 · T260.9% [54.1, 68.1], tier 243.3±5.3 · T443.3% [37.9, 48.5], tier 469.4±8.3 · T269.4% [60.7, 77.3], tier 267.7±4.5 · T267.7% [63.4, 72.4], tier 264.6±8.3 · T264.6% [56.3, 72.8], tier 267.7±7.6 · T267.7% [60.4, 75.5], tier 270.8±6.5 · T170.8% [64.6, 77.7], tier 173.3±3.8 · T373.3% [69.7, 77.2], tier 372.0±6.5 · T272.0% [65.5, 78.5], tier 272.5±6.5 · T272.5% [65.7, 78.7], tier 275.5±6.2 · T275.5% [69.3, 81.7], tier 2
decisio v0.8.051.9±4.0 · T451.9% [48.1, 56.0], tier 449.1±7.9 · T249.1% [41.4, 57.3], tier 253.0±5.1 · T353.0% [47.9, 58.0], tier 352.8±10.7 · T352.8% [42.1, 63.5], tier 370.5±4.7 · T270.5% [65.8, 75.2], tier 265.6±8.8 · T265.6% [56.9, 74.5], tier 270.8±7.2 · T270.8% [63.7, 78.1], tier 275.0±7.1 · T175.0% [68.0, 82.3], tier 181.2±3.3 · T181.2% [77.8, 84.5], tier 179.5±6.3 · T179.5% [73.0, 85.6], tier 178.5±6.2 · T178.5% [71.9, 84.3], tier 185.5±4.5 · T185.5% [81.1, 90.1], tier 1
Decision 2.0 Nox 4B51.7±4.2 · T451.7% [47.6, 56.0], tier 461.3±6.2 · T261.3% [55.2, 67.7], tier 244.8±5.5 · T444.8% [39.3, 50.2], tier 458.3±10.2 · T358.3% [47.9, 68.3], tier 360.4±4.5 · T360.4% [56.2, 65.3], tier 360.4±8.3 · T260.4% [52.2, 68.8], tier 261.5±8.3 · T261.5% [53.3, 69.8], tier 259.4±6.8 · T259.4% [53.0, 66.7], tier 256.3±3.8 · T756.3% [52.6, 60.3], tier 758.5±6.8 · T358.5% [51.7, 65.4], tier 354.5±6.3 · T454.5% [48.3, 61.0], tier 456.0±6.4 · T556.0% [49.5, 62.4], tier 5
Cygnet51.1±4.0 · T451.1% [47.2, 55.2], tier 452.2±6.4 · T252.2% [45.9, 58.7], tier 243.5±5.4 · T443.5% [38.1, 48.9], tier 473.6±8.2 · T173.6% [64.9, 81.4], tier 167.7±4.3 · T267.7% [63.4, 72.0], tier 266.7±7.6 · T266.7% [58.7, 74.0], tier 264.6±7.2 · T264.6% [57.5, 71.9], tier 271.9±6.7 · T171.9% [65.3, 78.7], tier 165.8±3.9 · T565.8% [62.0, 69.8], tier 560.0±6.9 · T360.0% [52.8, 66.7], tier 362.5±6.0 · T362.5% [56.5, 68.6], tier 375.0±6.5 · T275.0% [68.5, 81.3], tier 2
Tier 5
decider-4b v250.0±4.3 · T550.0% [45.8, 54.3], tier 553.0±6.9 · T253.0% [46.2, 60.1], tier 245.0±5.9 · T445.0% [39.2, 51.0], tier 461.1±9.7 · T361.1% [51.3, 70.6], tier 358.7±4.5 · T358.7% [54.2, 63.3], tier 359.4±9.0 · T259.4% [50.0, 68.1], tier 254.2±5.5 · T354.2% [48.9, 60.0], tier 362.5±8.3 · T262.5% [54.2, 70.8], tier 258.3±3.8 · T658.3% [54.5, 62.0], tier 659.5±6.9 · T359.5% [52.7, 66.5], tier 354.5±6.5 · T454.5% [48.0, 60.9], tier 461.0±6.1 · T461.0% [54.7, 67.0], tier 4
JevK548.9±3.8 · T548.9% [45.1, 52.7], tier 556.5±6.2 · T256.5% [50.6, 63.1], tier 243.0±5.5 · T443.0% [37.6, 48.5], tier 455.6±9.2 · T355.6% [46.2, 64.6], tier 355.9±4.4 · T455.9% [51.4, 60.2], tier 458.3±8.0 · T258.3% [50.0, 66.0], tier 253.1±8.3 · T353.1% [45.1, 61.8], tier 356.3±6.4 · T256.3% [50.0, 62.8], tier 257.7±3.6 · T657.7% [54.1, 61.4], tier 655.0±7.0 · T455.0% [47.7, 61.8], tier 454.5±5.9 · T454.5% [48.5, 60.2], tier 463.5±6.0 · T363.5% [57.6, 69.7], tier 3
Julia-148.8±3.3 · T548.8% [45.5, 52.1], tier 549.6±6.2 · T249.6% [43.2, 55.6], tier 248.9±4.1 · T348.9% [44.8, 53.0], tier 347.2±8.5 · T347.2% [38.4, 55.5], tier 346.9±3.2 · T546.9% [43.6, 50.0], tier 545.8±4.7 · T345.8% [40.6, 50.0], tier 347.9±7.2 · T347.9% [40.8, 55.2], tier 346.9±4.3 · T346.9% [42.5, 51.1], tier 350.0±3.6 · T850.0% [46.4, 53.5], tier 853.0±5.6 · T453.0% [47.7, 58.8], tier 449.5±6.3 · T449.5% [43.4, 56.1], tier 447.5±6.9 · T647.5% [40.5, 54.3], tier 6
Microsoft Decision 148.8±4.3 · T548.8% [44.6, 53.2], tier 551.7±7.5 · T251.7% [44.6, 59.6], tier 248.5±6.2 · T348.5% [42.4, 54.8], tier 345.1±10.5 · T445.1% [34.8, 55.9], tier 478.8±4.3 · T178.8% [74.7, 83.3], tier 179.2±8.1 · T179.2% [70.2, 86.5], tier 182.3±7.3 · T182.3% [74.5, 89.1], tier 175.0±7.3 · T175.0% [67.7, 82.3], tier 170.5±4.0 · T470.5% [66.6, 74.5], tier 470.5±6.4 · T270.5% [63.8, 76.5], tier 269.5±7.3 · T269.5% [61.8, 76.3], tier 271.5±6.4 · T371.5% [65.1, 78.0], tier 3
DiffusionGemma-Jev48.7±4.3 · T548.7% [44.5, 53.0], tier 547.4±6.1 · T247.4% [41.4, 53.6], tier 243.0±6.3 · T443.0% [36.6, 49.2], tier 468.8±9.0 · T268.8% [59.1, 77.1], tier 264.2±4.6 · T364.2% [59.8, 69.0], tier 366.7±7.2 · T266.7% [59.6, 74.0], tier 263.5±7.8 · T263.5% [56.1, 71.7], tier 262.5±7.4 · T262.5% [55.3, 70.2], tier 269.5±3.8 · T469.5% [65.7, 73.3], tier 470.5±6.8 · T270.5% [63.8, 77.4], tier 263.0±6.6 · T363.0% [56.3, 69.5], tier 375.0±5.6 · T275.0% [69.5, 80.6], tier 2
Clef Omni48.4±4.0 · T548.4% [44.5, 52.5], tier 557.0±6.6 · T257.0% [50.6, 63.9], tier 239.4±5.6 · T439.4% [33.7, 45.0], tier 463.9±8.5 · T263.9% [55.2, 72.2], tier 255.9±4.7 · T455.9% [51.4, 60.8], tier 461.5±8.3 · T261.5% [53.2, 69.8], tier 246.9±7.5 · T346.9% [39.1, 54.1], tier 359.4±7.5 · T259.4% [52.1, 67.0], tier 262.8±3.7 · T562.8% [59.1, 66.5], tier 568.5±6.8 · T268.5% [61.3, 74.9], tier 253.5±5.9 · T453.5% [47.4, 59.3], tier 466.5±6.5 · T366.5% [60.0, 73.1], tier 3
Hopper 12B48.3±4.0 · T548.3% [44.5, 52.5], tier 557.0±6.5 · T257.0% [50.8, 63.8], tier 238.0±5.1 · T438.0% [32.9, 43.2], tier 467.4±8.4 · T267.4% [58.7, 75.5], tier 262.5±4.6 · T362.5% [57.9, 67.1], tier 367.7±8.8 · T167.7% [58.5, 76.1], tier 157.3±7.7 · T357.3% [50.0, 65.3], tier 362.5±7.5 · T262.5% [55.2, 70.2], tier 265.8±3.9 · T565.8% [62.0, 69.8], tier 565.0±6.9 · T365.0% [57.8, 71.6], tier 357.0±6.3 · T357.0% [50.5, 63.2], tier 375.5±6.0 · T275.5% [69.5, 81.5], tier 2
Decision 2.0 Eos 0.8B46.3±4.2 · T546.3% [42.1, 50.5], tier 561.3±6.7 · T261.3% [54.6, 68.0], tier 236.1±6.1 · T536.1% [30.0, 42.1], tier 554.9±8.4 · T354.9% [46.1, 62.8], tier 353.8±3.6 · T453.8% [50.3, 57.6], tier 458.3±7.0 · T258.3% [51.1, 65.0], tier 246.9±6.2 · T346.9% [40.8, 53.1], tier 356.3±4.7 · T256.3% [52.0, 61.5], tier 249.7±3.7 · T849.7% [45.9, 53.4], tier 851.5±6.4 · T451.5% [45.1, 57.8], tier 448.0±6.5 · T448.0% [41.5, 54.5], tier 449.5±6.1 · T649.5% [43.4, 55.6], tier 6
Decision 2.0 Lux 9B45.4±4.0 · T545.4% [41.5, 49.6], tier 552.6±6.4 · T252.6% [46.3, 59.1], tier 237.0±5.5 · T537.0% [31.3, 42.4], tier 561.1±9.3 · T361.1% [51.4, 70.1], tier 362.2±4.6 · T362.2% [57.8, 66.9], tier 365.6±8.2 · T265.6% [57.6, 74.0], tier 260.4±7.3 · T260.4% [53.1, 67.7], tier 260.4±7.4 · T260.4% [53.2, 68.1], tier 261.0±3.9 · T661.0% [57.2, 65.0], tier 659.0±6.9 · T359.0% [52.0, 65.8], tier 359.5±6.6 · T359.5% [53.1, 66.2], tier 364.5±6.8 · T364.5% [57.4, 71.0], tier 3
Clef Flash45.1±4.3 · T545.1% [40.8, 49.3], tier 563.0±6.8 · T263.0% [56.2, 69.8], tier 235.9±5.9 · T535.9% [29.9, 41.7], tier 545.8±8.0 · T445.8% [37.6, 53.5], tier 461.1±4.5 · T361.1% [56.6, 65.6], tier 367.7±8.2 · T167.7% [59.2, 75.5], tier 157.3±7.1 · T357.3% [50.0, 64.3], tier 358.3±6.7 · T258.3% [52.0, 65.3], tier 257.7±4.0 · T657.7% [53.8, 61.8], tier 658.0±7.0 · T358.0% [51.0, 65.0], tier 355.0±6.0 · T455.0% [49.0, 61.1], tier 460.0±7.1 · T460.0% [52.9, 67.0], tier 4
Decision 2.0 Sol 2B45.1±4.4 · T545.1% [40.8, 49.5], tier 561.3±6.8 · T261.3% [54.4, 67.9], tier 234.6±6.0 · T534.6% [28.6, 40.6], tier 552.8±9.7 · T352.8% [42.6, 61.9], tier 350.3±2.9 · T450.3% [47.6, 53.4], tier 453.1±5.9 · T353.1% [46.9, 58.7], tier 346.9±4.4 · T346.9% [42.4, 51.1], tier 351.0±3.6 · T351.0% [47.9, 55.1], tier 348.5±3.8 · T948.5% [44.7, 52.3], tier 951.0±6.5 · T451.0% [44.5, 57.5], tier 446.5±6.9 · T446.5% [39.8, 53.6], tier 448.0±6.4 · T648.0% [41.7, 54.5], tier 6
Decision 2.0 Kai 0.6B43.4±4.2 · T543.4% [39.1, 47.5], tier 553.9±6.6 · T253.9% [47.2, 60.4], tier 235.2±5.9 · T535.2% [29.0, 40.9], tier 552.8±7.9 · T352.8% [44.9, 60.7], tier 348.3±2.8 · T448.3% [45.4, 51.0], tier 445.8±5.7 · T345.8% [39.6, 51.1], tier 345.8±3.4 · T345.8% [42.2, 48.9], tier 353.1±4.6 · T253.1% [49.0, 58.2], tier 248.3±4.0 · T948.3% [44.3, 52.3], tier 950.0±7.0 · T450.0% [43.1, 57.1], tier 445.0±6.6 · T545.0% [38.5, 51.6], tier 550.0±7.1 · T550.0% [42.9, 57.2], tier 5
Tier 6
Lev 4B42.1±4.0 · T642.1% [38.1, 46.2], tier 655.6±6.2 · T255.6% [49.6, 61.9], tier 230.2±5.6 · T630.2% [24.5, 35.6], tier 658.3±10.0 · T358.3% [48.0, 68.0], tier 353.5±2.8 · T453.5% [50.7, 56.3], tier 453.1±4.5 · T353.1% [49.0, 58.0], tier 353.1±5.3 · T353.1% [47.9, 58.5], tier 354.2±3.7 · T254.2% [51.0, 58.5], tier 252.7±3.6 · T852.7% [49.0, 56.3], tier 850.5±6.3 · T450.5% [44.3, 56.9], tier 453.0±6.2 · T453.0% [46.9, 59.3], tier 454.5±6.4 · T554.5% [48.0, 60.9], tier 5

Scores in percent, higher is better, ± half the 95% interval. T1 is the leading tie tier: models in one tier can't be told apart by the tie test. Orange: the one model ahead; green: tied for the lead. All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).

Jev

Provider
TypeSafe
Weights
Closed (API)
Size
Not disclosed
Licence
Proprietary
TaskScore [95% interval]TierLatency p50Cost / 1k
All three jobs, core63.3% [59.9, 66.7]3394 ms (Hosted API)$0.39
LLM-as-a-judge, core61.3% [54.8, 67.9]2414 ms (Hosted API)$0.49
Search, core63.3% [58.2, 68.1]2363 ms (Hosted API)$0.32
Scenario judge, core66.7% [57.3, 75.3]2524 ms (Hosted API)$0.43
All three jobs, long conversations72.6% [67.9, 77.3]21,701 ms (Hosted API)$0.82
LLM-as-a-judge, long conversations71.9% [63.8, 79.6]1, tied1,906 ms (Hosted API)$0.85
Search, long conversations72.9% [63.7, 81.6]1, tied1,858 ms (Hosted API)$0.84
Scenario judge, long conversations72.9% [65.0, 80.8]1, tied1,334 ms (Hosted API)$0.77
All three jobs, hard cases72.7% [69.2, 76.3]3269 ms (Hosted API)$0.086
LLM-as-a-judge, hard cases72.5% [66.7, 78.3]2258 ms (Hosted API)$0.11
Search, hard cases69.0% [62.3, 75.5]2256 ms (Hosted API)$0.11
Scenario judge, hard cases76.5% [70.4, 82.1]2325 ms (Hosted API)$0.040
    How we measured

    Scores carry a 95% bootstrap interval (2,000 resamples, the checks built from one scenario resampled together). Two models are tied when the 95% interval of their paired difference on the same checks holds zero.

    Tie tiers

    Models in one tier can't be separated by the tie test (the 95% interval of the paired difference on the same cases holds zero). Tiers are computed in the export over every model; filters on this page never change them.

    Contamination

    No check was ever published, so no model can have trained on them. Core and long conversations are real-world based; the hard cases are synthetic, generated for this benchmark from made-up data, with every label computed from that data. The page shows aggregates only: no check and no check text.

    Latency and hardware

    • Hosted API: Jev, OpenAI Decisions API, Mercury Decide, Clef, Clef Flash, GLiDE, Clef Omni and Microsoft Decision 1, called as hosted APIs from a laptop, one call per check; the time includes the network
    • H100: Decision 2.0 Vega 27B, Kev-27B, pplx-decider v1.1 27B, AutoJev-27B, DiffusionGemma-Jev, Jev-Omni, Cygnet, Hopper 12B, Decision 2.0 Lux 9B, Quyet-1.0-Large, decisio v0.8.0, deck-31B, Rene-1 31B FP8 and H2O-Lightning-4B v1.2.2, open weights on one rented H100; the server’s own time per check, 30 checks one at a time
    • L4: Winnow, decider-4b v2, JevK5, Decision 2.0 Nox 4B, Lev 4B, Decision 2.0 Sol 2B, Decision 2.0 Eos 0.8B and Decision 2.0 Kai 0.6B, open weights on one rented L4; the server’s own time per check, 30 checks one at a time
    • Server CPU: Julia-1, served on the GPU box’s CPU; the server’s own time per check, 30 checks one at a time

    Latencies are only compared within one hardware tier.

    Datasets

    • All three jobs, core (834 test items): real-world based labelled checks, never published (private, aggregates only)
    • LLM-as-a-judge, core (230 test items): real-world based labelled checks, never published (private, aggregates only)
    • Search, core (460 test items): real-world based labelled checks, never published (private, aggregates only)
    • Scenario judge, core (144 test items): real-world based labelled checks, never published (private, aggregates only)
    • All three jobs, long conversations (288 test items): real-world based long sessions, never published (private, aggregates only)
    • LLM-as-a-judge, long conversations (96 test items): real-world based long sessions, never published (private, aggregates only)
    • Search, long conversations (96 test items): real-world based long sessions, never published (private, aggregates only)
    • Scenario judge, long conversations (96 test items): real-world based long sessions, never published (private, aggregates only)
    • All three jobs, hard cases (600 test items): synthetic lookup and arithmetic checks, labels computed from the fixture data (private, aggregates only)
    • LLM-as-a-judge, hard cases (200 test items): synthetic lookup and arithmetic checks, labels computed from the fixture data (private, aggregates only)
    • Search, hard cases (200 test items): synthetic lookup and arithmetic checks, labels computed from the fixture data (private, aggregates only)
    • Scenario judge, hard cases (200 test items): synthetic lookup and arithmetic checks, labels computed from the fixture data (private, aggregates only)
    • An error, a reply with no verdict, or an inconclusive Scenario verdict counts as wrong.
    • All three jobs: every check of the three jobs pooled, one accuracy per set (834 checks on core, 288 checks on long conversations and 600 checks on hard cases).
    • Scores count answers on texts cut to fit the model’s window. For 7 models some texts were still too long after the cut LangWatch uses in production, so they were cut further until they fit. Counted as errors instead, which is what production would return, their scores are lower: pplx-decider v1.1 27B, 161 core and 22 long conversation checks (60.7% becomes 49.6% and 72.6% becomes 68.1%); AutoJev-27B, 161 core and 22 long conversation checks (55.6% becomes 45.9% and 69.8% becomes 64.6%); JevK5, 155 core and 16 long conversation checks (48.9% becomes 40.1% and 55.9% becomes 53.8%); Lev 4B, 155 core and 16 long conversation checks (42.1% becomes 33.8% and 53.5% becomes 51.0%); Decision 2.0 Kai 0.6B, 158 core and 16 long conversation checks (43.4% becomes 35.7% and 48.3% becomes 45.8%); Julia-1, 169 core and 14 long conversation checks (48.8% becomes 40.9% and 46.9% becomes 44.1%); Quyet-1.0-Large, 260 core and 186 long conversation checks (58.4% becomes 40.9% and 67.0% becomes 23.6%).
    • Latency is the median time per check, compared only within one hardware tier. Hosted APIs were timed from a laptop on every check, so their time includes the network. Open-weights models were timed on the GPU box by the server itself, 30 checks one at a time, one number for every job and set.
    • The 23 open-weights models ran in bf16 on rented L4 and H100 GPUs (Winnow in Q8_0 GGUF, decisio v0.8.0 in FP8, deck-31B in FP8 and Rene-1 31B FP8 in FP8). In the timing run, Julia-1 was served on the box’s CPU.
    • AutoJev-27B: The same weights as Perplexity’s pplx-decider v1 27B, file for file, so the two are one model here.
    • Winnow: Run from its Q8_0 GGUF weights, the precision its authors recommend.
    • Cygnet: A recipe, not a trained model: a letter readout on stock Gemma 4 12B IT. Run without its calibration temperature, which rescales probabilities but cannot flip a yes or no at the 0.5 threshold.
    • decider-4b v2: Served with a smaller CUDA graph token budget than its default, a capacity setting that does not change answers.
    • JevK5: Takes an 8k token state.
    • Lev 4B: Takes an 8k token state.
    • Julia-1: Served on the box’s CPU, so its latency is CPU time.
    • Quyet-1.0-Large: Reads at most 6,000 tokens of a conversation, so most core and long conversation texts were cut to fit.
    • decisio v0.8.0: A serving recipe on stock Gemma 4 31B. Refused 146 core checks whose text carries inline image data; each counts as wrong.
    • deck-31B: A recipe, not a trained model: a letter readout on stock Gemma 4 31B, run in FP8.
    • Rene-1 31B FP8: Ran on one to three GPUs at once, so its cost per 1,000 checks is approximate.
    • H2O-Lightning-4B v1.2.2: Version 1.2.2, the release its card tells users to download; the public JevBench board ranks version 1.1, which has different weights.
    • Mercury Decide: Hard cases through Inception’s own API; core and long conversations through OpenRouter, with zero data retention.
    • Clef: Open weights, run here on Cloudflare’s hosted API, so its latency is the hosted service’s. Calls made on 10 October 2026, after Cloudflare’s faster serving.
    • Clef Flash: Open weights, run here on Cloudflare’s hosted API, so its latency is the hosted service’s. Calls made on 10 October 2026, at its new price. The hosted service reads at most 24,000 tokens of a check and cuts longer ones without an error.
    • GLiDE: A hosted “thinking” decision model: an uncertain check can take extra time. Refused 70 core checks that are over its context window; each counts as wrong.
    • Microsoft Decision 1: Called through OpenRouter, with zero data retention. Reads at most 32,768 tokens: refused 110 core checks over that limit, each counts as wrong.
    • pplx-decider v1.1 27B: hosted API price (Perplexity), we ran the open weights ourselves. $0.02 per 1M input tokens, output free (Perplexity’s API and OpenRouter), times the input tokens our server counted on each check with the model’s own tokenizer. At our GPU rent it would cost $0.51 on core, $0.51 on long conversations and $0.27 on hard cases per 1,000 checks.
    • AutoJev-27B: hosted API price (Perplexity), we ran the open weights ourselves. $0.04 per 1M input tokens, output free (Perplexity’s API and OpenRouter), times the input tokens our server counted on each check with the model’s own tokenizer. At our GPU rent it would cost $0.45 on core, $0.51 on long conversations and $0.26 on hard cases per 1,000 checks.
    • Cost per 1,000 checks: hosted APIs at the price recorded on the run, pplx-decider v1.1 27B and AutoJev-27B at Perplexity’s hosted API price. Other open-weights models at GPU rent (€2.87 and €3.31 per GPU hour, in dollars at 1.149 per euro) times the GPU time each check took on that set, shared with the other models on the same card; on hard cases, the box’s median time per check, one check at a time.

    Release jev-class-2026-10-10, integrity sha256-leaves-v1.