LangWatch
Sign Up

Amin RoudakiOpen weights model31.3B parameters

decisio v0.8.0 benchmarks

All models
  • General rank
    Too few tasks3 non-coding tasks ranked5 needed for a rank
  • Speed#3 / 14
    178 msMedian latency, p50H100 only
  • Cost
    Self-hostedRan on H100No hosted price
  • LLM-as-a-judge#2 / 31 tied
    49.1%Best task · Accuracyvs Jev-class models

Comparison summary

Strongest
Judging and evals
Leads 0 of 3 (0%)

A recipe, not a trained model: a fixed way of asking a stock open model a typed question and reading its answer, so it decides like a decision model. Strongest at judging.

Specifications

Lab
Amin Roudaki
Weights
Open weights
Parameters
31.3B
Licence
Apache-2.0
Type
Recipe on a stock model

Judging and evals

Set

Share of tasks led · #16 of 31

  1. Mercury DecideLeads 2 of 3
  2. OpenAI Decisions APILeads 4 of 6
  3. ClefLeads 1 of 3
  4. deck-31BLeads 1 of 3
  5. Kev-27BLeads 1 of 3
  6. pplx-decider v1.1 27BLeads 1 of 3
  7. Quyet-1.0-LargeLeads 1 of 3
  8. decisio v0.8.0
    Leads 0 of 3
See the 3 tasks

LLM-as-a-judge

Tier 2 of 2#2 / 31

Accuracy, % · Higher is better

  1. OpenAI Decisions API71.7, interval 66.1 to 77.2, ahead alone
  2. Clef Flash63.0, interval 56.2 to 69.8, tier 2
  3. Mercury Decide61.7, interval 55.6 to 67.9, tier 2
  4. Decision 2.0 Eos 0.8B61.3, interval 54.6 to 68.0, tier 2
  5. Decision 2.0 Nox 4B61.3, interval 55.2 to 67.7, tier 2
  6. Decision 2.0 Sol 2B61.3, interval 54.4 to 67.9, tier 2
  7. Jev61.3, interval 54.8 to 67.9, tier 2
  8. decisio v0.8.0
    49.1, interval 41.4 to 57.3, tier 2
Tested against Jev and 29 Jev-class models

Scenario judge

Tier 3 of 4#19 / 31

Accuracy, % · Higher is better

  1. Mercury Decide83.3, interval 75.5 to 90.3, tied for the lead
  2. OpenAI Decisions API83.3, interval 76.1 to 89.9, tied for the lead
  3. deck-31B81.9, interval 74.8 to 88.1, tied for the lead
  4. Quyet-1.0-Large79.2, interval 72.0 to 85.9, tied for the lead
  5. pplx-decider v1.1 27B77.8, interval 69.8 to 84.9, tied for the lead
  6. Clef75.7, interval 67.4 to 83.1, tied for the lead
  7. Rene-1 31B FP875.0, interval 66.9 to 82.9, tied for the lead
  8. decisio v0.8.0
    52.8, interval 42.1 to 63.5, tier 3
Tested against Jev and 29 Jev-class models

Search

Tier 3 of 6#9 / 31

Accuracy, % · Higher is better

  1. OpenAI Decisions API70.4, interval 66.0 to 74.8, tied for the lead
  2. Mercury Decide66.3, interval 61.5 to 71.0, tied for the lead
  3. GLiDE63.9, interval 59.5 to 68.3, tier 2
  4. Jev63.3, interval 58.2 to 68.1, tier 2
  5. Kev-27B61.1, interval 56.4 to 65.7, tier 2
  6. Clef60.2, interval 55.7 to 64.9, tier 2
  7. deck-31B58.9, interval 54.3 to 63.5, tier 2
  8. decisio v0.8.0
    53.0, interval 47.9 to 58.0, tier 3
Tested against Jev and 29 Jev-class models

Speed

Median latency, p50 · H100 only · lower is better

  1. H2O-Lightning-4B v1.2.277 ms
  2. Cygnet162 ms
  3. decisio v0.8.0
    178 ms
  4. DiffusionGemma-Jev188 ms
  5. Decision 2.0 Lux 9B213 ms
  6. AutoJev-27B245 ms
  7. Jev-Omni248 ms
  8. pplx-decider v1.1 27B256 ms
How we count

A model leads a task when it is in the task's leading tie tier. Each task counts inside its own benchmark, against that benchmark's models; no score is averaged. An area chart shows the models ranked on at least 2 of the area's tasks in a benchmark this model is in too. The overall standing counts every non-coding tasks and needs 5 for a rank. Latency is compared only on one hardware tier.

decisio v0.8.0Other modelsWhisker: 95% interval