LangWatch
Sign Up

Community (Hugging Face)Open weights model26.1B parameters

AutoJev-27B benchmarks

All models
  • General rank
    #6of 28Leads 8 of 15 (53%)non-coding tasks
  • Speed#6 / 14
    245 msMedian latency, p50H100 only
  • Cost#2 / 10
    $0.16Per 1,000 decisionsHosted price
  • Tool routing#1 / 16 tied
    80.6%Best task · Accuracyvs open models

Comparison summary

Strongest
Safety and guardrails
Leads 2 of 2 (100%)
Weakest
Judging and evals
Leads 0 of 3 (0%)

A typed decision model: it answers a typed question, such as yes or no or one of a few options, with a probability for each answer and no text. Strong at safety and general decisions, weaker at judging.

Specifications

Lab
Community (Hugging Face)
Weights
Open weights
Parameters
26.1B
Licence
Apache-2.0
Also published as
pplx-decider v1 27B (Perplexity)

Overall standing

Share of non-coding tasks led · #6 of 28

  1. Claude Opus 5.585%
  2. OpenAI Decisions API67%
  3. Jev61%
  4. Eikos-27B60%
  5. GPT-6.1 Sol54%
  6. AutoJev-27B
    53%
  7. Claude Sonnet 5.546%
  8. Shisa DE-142%

Safety and guardrails

Share of tasks led · #2 of 13

  1. Eikos-27BLeads 3 of 3
  2. AutoJev-27B
    Leads 2 of 2
  3. JevLeads 8 of 12
  4. Shisa DE-1Leads 1 of 4
  5. Kev-9BLeads 0 of 4
  6. Eikos-4BLeads 0 of 3
  7. SemIf (Qwen3.5-4B)Leads 0 of 4
  8. Kev-0.8BLeads 0 of 8
See the 4 tasks

Prompt injection

Tied for the lead#1 / 12

Catch rate at 5% false alarms, % · Higher is better

  1. Jev94.6, interval 92.2 to 96.7, tied for the lead
  2. Eikos-27B94.2, interval 90.5 to 97.1, tied for the lead
  3. Shisa DE-194.0, interval 90.3 to 96.2, tied for the lead
  4. AutoJev-27B
    92.0, interval 86.6 to 95.0, tied for the lead
  5. Kev-4B91.6, interval 88.2 to 94.2, tier 2
  6. Eikos-4B90.8, interval 88.3 to 94.0, tier 2
  7. SemIf (Qwen3.5-4B)90.6, interval 87.8 to 93.5, tier 2
  8. Kev-9B87.6, interval 82.4 to 92.5, tier 2
Tested against Jev and 14 open models

Moderation

Trained on the test data, not ranked

AUROC, % · Higher is better

  1. AutoJev-27B*
    90.6*, interval 88.4 to 92.7, trained on this data, not ranked
  2. Eikos-27B90.3, interval 87.9 to 92.4, tied for the lead
  3. Jev90.3, interval 87.9 to 92.4, tied for the lead
  4. Eikos-4B88.8, interval 86.3 to 91.3, tier 2
  5. SemIf (Qwen3.5-4B)88.7, interval 86.2 to 91.2, tier 2
  6. Kev-9B88.5, interval 86.0 to 90.9, tier 2
  7. Shisa DE-188.3, interval 85.6 to 90.8, tier 2
  8. Kev-4B82.0, interval 78.8 to 85.0, tier 3
Tested against Jev and 14 open models

Off-topic

Trained on the test data, not ranked

Balanced accuracy, % · Higher is better

  1. Jev93.4, interval 91.8 to 94.9, ahead alone
  2. AutoJev-27B*
    92.7*, interval 91.0 to 94.4, trained on this data, not ranked
  3. Eikos-27B*90.6*, interval 88.8 to 92.4, trained on this data, not ranked
  4. Kev-9B89.6, interval 87.7 to 91.5, tier 2
  5. Shisa DE-188.0, interval 86.1 to 89.9, tier 3
  6. Kev-4B84.4, interval 82.1 to 86.7, tier 4
  7. SemIf (Qwen3.5-4B)81.1, interval 79.0 to 83.4, tier 5
  8. Eikos-4B*79.6*, interval 77.1 to 82.0, trained on this data, not ranked
Tested against Jev and 14 open models

PII

Tied for the lead#1 / 13

Catch rate at 5% false alarms, % · Higher is better

  1. Eikos-27B95.2, interval 92.8 to 97.8, tied for the lead
  2. AutoJev-27B
    92.8, interval 90.2 to 95.5, tied for the lead
  3. Jev90.8, interval 88.1 to 93.8, tier 2
  4. Shisa DE-184.2, interval 77.3 to 89.1, tier 3
  5. Eikos-4B78.6, interval 74.5 to 85.3, tier 3
  6. SemIf (Qwen3.5-4B)66.1, interval 60.9 to 74.6, tier 4
  7. Kev-9B62.5, interval 0.2 to 66.4, tier 4
  8. Kev-0.8B22.8, interval 6.1 to 26.3, tier 5
Tested against Jev and 14 open models

Routing and classification

Share of tasks led · #1 of 16

  1. AutoJev-27B
    Leads 4 of 5
  2. Shisa DE-1Leads 3 of 4
  3. JevLeads 9 of 14
  4. Eikos-27BLeads 1 of 3
  5. SemIf (Qwen3.5-4B)Leads 1 of 3
  6. Kev-4BLeads 0 of 3
  7. Kev-9BLeads 0 of 2
  8. Eikos-4BLeads 0 of 2
See the 5 tasks

Routing, 20 intents

Ahead alone#1 / 7

Accuracy, % · Higher is better

  1. Kev-4B*93.0*, interval 91.4 to 94.5, trained on this data, not ranked
  2. Kev-9B*93.0*, interval 91.4 to 94.6, trained on this data, not ranked
  3. Kev-0.8B*91.3*, interval 89.5 to 93.0, trained on this data, not ranked
  4. AutoJev-27B
    89.7, interval 87.7 to 91.6, ahead alone
  5. Eikos-27B*89.6*, interval 87.7 to 91.4, trained on this data, not ranked
  6. Kev-0.6B*89.5*, interval 87.5 to 91.5, trained on this data, not ranked
  7. Jev89.1, interval 87.0 to 91.0, tier 2
  8. Shisa DE-188.1, interval 85.9 to 90.1, tier 2
Tested against Jev and 14 open models

Routing, 77 intents

Tied for the lead#1 / 5

Accuracy, % · Higher is better

  1. Kev-4B*85.0*, interval 81.8 to 88.0, trained on this data, not ranked
  2. Kev-9B*85.0*, interval 81.8 to 88.2, trained on this data, not ranked
  3. Kev-0.8B*83.0*, interval 79.6 to 86.4, trained on this data, not ranked
  4. AutoJev-27B
    80.6, interval 77.0 to 84.0, tied for the lead
  5. Jev79.6, interval 76.0 to 83.0, tied for the lead
  6. Kev-0.6B*78.2*, interval 74.4 to 81.8, trained on this data, not ranked
  7. Eikos-27B*77.8*, interval 74.0 to 81.4, trained on this data, not ranked
  8. Eikos-4B*74.0*, interval 70.0 to 77.8, trained on this data, not ranked
Tested against Jev and 14 open models

Tool routing

Tied for the lead#1 / 16

Accuracy, % · Higher is better

  1. Shisa DE-181.3, interval 78.9 to 83.5, tied for the lead
  2. AutoJev-27B
    80.6, interval 78.1 to 82.9, tied for the lead
  3. SemIf (Qwen3.5-4B)79.9, interval 77.4 to 82.2, tied for the lead
  4. Eikos-27B79.3, interval 76.8 to 81.8, tier 2
  5. Jev78.3, interval 75.7 to 80.7, tier 2
  6. Eikos-4B75.5, interval 72.8 to 78.0, tier 3
  7. Kev-9B72.4, interval 69.7 to 75.0, tier 4
  8. Kev-4B71.4, interval 68.6 to 74.0, tier 4
Tested against Jev and 14 open models

Complaint routing

Tier 2 of 8#3 / 15

Accuracy, % · Higher is better

  1. Jev78.7, interval 76.3 to 81.2, tied for the lead
  2. Shisa DE-178.0, interval 75.4 to 80.6, tied for the lead
  3. Kev-4B76.2, interval 73.6 to 79.0, tier 2
  4. AutoJev-27B
    75.9, interval 73.2 to 78.7, tier 2
  5. Eikos-27B75.6, interval 73.0 to 78.4, tier 2
  6. Kev-9B75.0, interval 72.4 to 77.7, tier 2
  7. SemIf (Qwen3.5-4B)72.9, interval 70.2 to 75.7, tier 3
  8. Kev-0.6B59.3, interval 56.3 to 62.3, tier 4
Tested against Jev and 13 open models

Typed decisions

Tied for the lead#1 / 14

Accuracy, % · Higher is better

  1. Laya-typed*77.4*, interval 75.4 to 79.4, trained on this data, not ranked
  2. Shisa DE-175.1, interval 72.8 to 77.3, tied for the lead
  3. Jev73.9, interval 71.8 to 76.0, tied for the lead
  4. AutoJev-27B
    73.8, interval 71.7 to 75.9, tied for the lead
  5. Eikos-27B73.4, interval 71.2 to 75.6, tied for the lead
  6. Kev-4B65.8, interval 63.6 to 68.0, tier 2
  7. Eikos-4B65.3, interval 62.7 to 68.0, tier 2
  8. SemIf (Qwen3.5-4B)63.0, interval 60.5 to 65.4, tier 3
Tested against Jev and 13 open models

Grounding

Share of tasks led · #3 of 15

  1. Eikos-27BLeads 2 of 2
  2. JevLeads 4 of 6
  3. AutoJev-27B
    Leads 0 of 2
  4. Shisa DE-1Leads 0 of 2
  5. Eikos-4BLeads 0 of 2
  6. Kev-4BLeads 0 of 2
  7. Kev-9BLeads 0 of 2
  8. Kev-0.8BLeads 0 of 4
See the 2 tasks

RAG faithfulness

Tier 2 of 5#3 / 13

Balanced accuracy, % · Higher is better

  1. Eikos-27B82.1, interval 79.2 to 84.9, tied for the lead
  2. Jev80.3, interval 77.5 to 83.1, tied for the lead
  3. Shisa DE-179.2, interval 76.2 to 82.2, tier 2
  4. AutoJev-27B
    79.1, interval 76.1 to 82.0, tier 2
  5. Kev-9B73.1, interval 70.0 to 76.0, tier 3
  6. Kev-4B72.5, interval 69.4 to 75.5, tier 3
  7. SemIf (Qwen3.5-4B)72.0, interval 68.9 to 75.2, tier 3
  8. Eikos-4B71.0, interval 67.4 to 74.5, tier 3
Tested against Jev and 14 open models

Search relevance

Tier 2 of 7#3 / 16

Accuracy, % · Higher is better

  1. Jev57.7, interval 54.7 to 60.8, tied for the lead
  2. Eikos-27B56.6, interval 53.4 to 59.7, tied for the lead
  3. AutoJev-27B
    52.8, interval 49.6 to 55.9, tier 2
  4. Shisa DE-150.3, interval 47.1 to 53.6, tier 2
  5. Kev-4B44.8, interval 41.7 to 48.0, tier 3
  6. Kev-9B43.4, interval 40.2 to 46.5, tier 3
  7. Eikos-4B43.3, interval 40.1 to 46.3, tier 3
  8. SemIf (Qwen3.5-4B)41.7, interval 38.6 to 44.7, tier 4
Tested against Jev and 14 open models

Agents

Leads 0 of 1, too few tasks to compare here.

See the task

Web-agent actions

Tier 2 of 10#2 / 15

Accuracy, % · Higher is better

  1. Jev70.8, interval 68.9 to 72.6, ahead alone
  2. Eikos-27B68.4, interval 66.5 to 70.4, tier 2
  3. AutoJev-27B
    68.2, interval 66.2 to 70.2, tier 2
  4. Shisa DE-165.2, interval 63.3 to 67.2, tier 3
  5. Eikos-4B62.9, interval 60.9 to 64.8, tier 4
  6. SemIf (Qwen3.5-4B)59.5, interval 57.5 to 61.5, tier 5
  7. Kev-4B58.0, interval 55.9 to 60.2, tier 5
  8. SimpleJev (Qwen3.5-0.8B)56.3, interval 54.7 to 57.8, tier 6
Tested against Jev and 13 open models

General decisions

Share of tasks led · #2 of 10

  1. JevLeads 4 of 4
  2. AutoJev-27B
    Leads 2 of 2
  3. Kev-0.6BLeads 0 of 4
  4. Kev-0.8BLeads 0 of 4
  5. LayaLeads 0 of 4
  6. Laya-typedLeads 0 of 4
  7. SimpleJev (Qwen3.5-0.8B)Leads 0 of 2
  8. Kev-4BLeads 0 of 2
See the 2 tasks

Community sets

Tied for the lead#1 / 10

Accuracy, % · Higher is better

  1. AutoJev-27B
    63.0, interval 60.2 to 65.8, tied for the lead
  2. Jev62.3, interval 59.5 to 65.1, tied for the lead
  3. Eikos-27B59.8, interval 56.9 to 62.6, tier 2
  4. Eikos-4B55.8, interval 52.9 to 58.7, tier 3
  5. Kev-4B55.8, interval 53.0 to 58.7, tier 3
  6. Laya-typed47.8, interval 44.9 to 50.5, tier 4
  7. Kev-0.8B46.4, interval 43.5 to 49.3, tier 4
  8. Laya45.9, interval 43.1 to 48.8, tier 4
Tested against Jev and 13 open models

JevBench public

Tied for the lead#1 / 12

Accuracy, % · Higher is better

  1. Eikos-27B*91.8*, interval 87.9 to 95.2, trained on this data, not ranked
  2. AutoJev-27B
    86.6, interval 81.9 to 90.9, tied for the lead
  3. Shisa DE-186.1, interval 81.2 to 90.6, tied for the lead
  4. Jev85.7, interval 81.0 to 90.1, tied for the lead
  5. Eikos-4B*85.3*, interval 80.4 to 89.8, trained on this data, not ranked
  6. SemIf (Qwen3.5-4B)80.1, interval 74.7 to 85.3, tier 2
  7. Kev-4B71.4, interval 65.2 to 77.4, tier 3
  8. Kev-0.6B60.6, interval 53.8 to 67.1, tier 4
Tested against Jev and 13 open models

Judging and evals

Set

Share of tasks led · #13 of 31

  1. Mercury DecideLeads 2 of 3
  2. OpenAI Decisions APILeads 4 of 6
  3. ClefLeads 1 of 3
  4. deck-31BLeads 1 of 3
  5. Kev-27BLeads 1 of 3
  6. pplx-decider v1.1 27BLeads 1 of 3
  7. Quyet-1.0-LargeLeads 1 of 3
  8. AutoJev-27B
    Leads 0 of 3
See the 3 tasks

LLM-as-a-judge

Tier 2 of 2#2 / 31

Accuracy, % · Higher is better

  1. OpenAI Decisions API71.7, interval 66.1 to 77.2, ahead alone
  2. Clef Flash63.0, interval 56.2 to 69.8, tier 2
  3. Mercury Decide61.7, interval 55.6 to 67.9, tier 2
  4. Decision 2.0 Eos 0.8B61.3, interval 54.6 to 68.0, tier 2
  5. Decision 2.0 Nox 4B61.3, interval 55.2 to 67.7, tier 2
  6. Decision 2.0 Sol 2B61.3, interval 54.4 to 67.9, tier 2
  7. Jev61.3, interval 54.8 to 67.9, tier 2
  8. AutoJev-27B
    52.6, interval 46.1 to 59.6, tier 2
Tested against Jev and 29 Jev-class models

Scenario judge

Tier 2 of 4#11 / 31

Accuracy, % · Higher is better

  1. Mercury Decide83.3, interval 75.5 to 90.3, tied for the lead
  2. OpenAI Decisions API83.3, interval 76.1 to 89.9, tied for the lead
  3. deck-31B81.9, interval 74.8 to 88.1, tied for the lead
  4. Quyet-1.0-Large79.2, interval 72.0 to 85.9, tied for the lead
  5. pplx-decider v1.1 27B77.8, interval 69.8 to 84.9, tied for the lead
  6. Clef75.7, interval 67.4 to 83.1, tied for the lead
  7. Rene-1 31B FP875.0, interval 66.9 to 82.9, tied for the lead
  8. AutoJev-27B
    71.5, interval 62.9 to 79.4, tier 2
Tested against Jev and 29 Jev-class models

Search

Tier 3 of 6#9 / 31

Accuracy, % · Higher is better

  1. OpenAI Decisions API70.4, interval 66.0 to 74.8, tied for the lead
  2. Mercury Decide66.3, interval 61.5 to 71.0, tied for the lead
  3. GLiDE63.9, interval 59.5 to 68.3, tier 2
  4. Jev63.3, interval 58.2 to 68.1, tier 2
  5. Kev-27B61.1, interval 56.4 to 65.7, tier 2
  6. Clef60.2, interval 55.7 to 64.9, tier 2
  7. deck-31B58.9, interval 54.3 to 63.5, tier 2
  8. AutoJev-27B
    52.2, interval 47.0 to 57.5, tier 3
Tested against Jev and 29 Jev-class models

Speed

Median latency, p50 · H100 only · lower is better

  1. H2O-Lightning-4B v1.2.277 ms
  2. Cygnet162 ms
  3. decisio v0.8.0178 ms
  4. DiffusionGemma-Jev188 ms
  5. Decision 2.0 Lux 9B213 ms
  6. AutoJev-27B
    245 ms
  7. Jev-Omni248 ms
  8. pplx-decider v1.1 27B256 ms

Cost

Per 1,000 decisions, hosted price · lower is better

  1. pplx-decider v1.1 27B$0.081
  2. AutoJev-27B
    $0.16
  3. Microsoft Decision 1$0.26
  4. Clef Flash$0.31
  5. Jev$0.43
  6. Mercury Decide$0.52
  7. OpenAI Decisions API$1.23
  8. GLiDE$1.30
How we count

A model leads a task when it is in the task's leading tie tier. Each task counts inside its own benchmark, against that benchmark's models; no score is averaged. An area chart shows the models ranked on at least 2 of the area's tasks in a benchmark this model is in too. The overall standing counts every non-coding tasks and needs 5 for a rank. Latency is compared only on one hardware tier.

AutoJev-27BOther modelsTrained on the test data*Whisker: 95% interval