LangWatch
Sign Up

Caio VicentinoOpen weights model27.8B parameters

Eikos-27B benchmarks

All models
  • General rank
    #4of 28Leads 6 of 10 (60%)non-coding tasks
  • Speed#3 / 4
    82 msMedian latency, p50H100 only
  • Cost
    Self-hostedRan on H100No hosted price
  • Search relevance#1 / 16 tied
    56.6%Best task · Accuracyvs open models

Comparison summary

Strongest
Safety and guardrails
Leads 3 of 3 (100%)
Weakest
General decisions
Leads 0 of 1 (0%)

A typed decision model: it answers a typed question, such as yes or no or one of a few options, with a probability for each answer and no text. Strong at safety and grounding, weaker at general decisions.

Specifications

Lab
Caio Vicentino
Weights
Open weights
Parameters
27.8B
Licence
MIT

Overall standing

Share of non-coding tasks led · #4 of 28

  1. Claude Opus 5.585%
  2. OpenAI Decisions API67%
  3. Jev61%
  4. Eikos-27B
    60%
  5. GPT-6.1 Sol54%
  6. AutoJev-27B53%
  7. Claude Sonnet 5.546%
  8. Shisa DE-142%

Safety and guardrails

Share of tasks led · #1 of 13

  1. Eikos-27B
    Leads 3 of 3
  2. AutoJev-27BLeads 2 of 2
  3. JevLeads 8 of 12
  4. Shisa DE-1Leads 1 of 4
  5. Kev-9BLeads 0 of 4
  6. Eikos-4BLeads 0 of 3
  7. SemIf (Qwen3.5-4B)Leads 0 of 4
  8. Kev-0.8BLeads 0 of 8
See the 4 tasks

Prompt injection

Tied for the lead#1 / 12

Catch rate at 5% false alarms, % · Higher is better

  1. Jev94.6, interval 92.2 to 96.7, tied for the lead
  2. Eikos-27B
    94.2, interval 90.5 to 97.1, tied for the lead
  3. Shisa DE-194.0, interval 90.3 to 96.2, tied for the lead
  4. AutoJev-27B92.0, interval 86.6 to 95.0, tied for the lead
  5. Kev-4B91.6, interval 88.2 to 94.2, tier 2
  6. Eikos-4B90.8, interval 88.3 to 94.0, tier 2
  7. SemIf (Qwen3.5-4B)90.6, interval 87.8 to 93.5, tier 2
  8. Kev-9B87.6, interval 82.4 to 92.5, tier 2
Tested against Jev and 14 open models

Moderation

Tied for the lead#1 / 12

AUROC, % · Higher is better

  1. AutoJev-27B*90.6*, interval 88.4 to 92.7, trained on this data, not ranked
  2. Eikos-27B
    90.3, interval 87.9 to 92.4, tied for the lead
  3. Jev90.3, interval 87.9 to 92.4, tied for the lead
  4. Eikos-4B88.8, interval 86.3 to 91.3, tier 2
  5. SemIf (Qwen3.5-4B)88.7, interval 86.2 to 91.2, tier 2
  6. Kev-9B88.5, interval 86.0 to 90.9, tier 2
  7. Shisa DE-188.3, interval 85.6 to 90.8, tier 2
  8. Kev-4B82.0, interval 78.8 to 85.0, tier 3
Tested against Jev and 14 open models

Off-topic

Trained on the test data, not ranked

Balanced accuracy, % · Higher is better

  1. Jev93.4, interval 91.8 to 94.9, ahead alone
  2. AutoJev-27B*92.7*, interval 91.0 to 94.4, trained on this data, not ranked
  3. Eikos-27B*
    90.6*, interval 88.8 to 92.4, trained on this data, not ranked
  4. Kev-9B89.6, interval 87.7 to 91.5, tier 2
  5. Shisa DE-188.0, interval 86.1 to 89.9, tier 3
  6. Kev-4B84.4, interval 82.1 to 86.7, tier 4
  7. SemIf (Qwen3.5-4B)81.1, interval 79.0 to 83.4, tier 5
  8. Eikos-4B*79.6*, interval 77.1 to 82.0, trained on this data, not ranked
Tested against Jev and 14 open models

PII

Tied for the lead#1 / 13

Catch rate at 5% false alarms, % · Higher is better

  1. Eikos-27B
    95.2, interval 92.8 to 97.8, tied for the lead
  2. AutoJev-27B92.8, interval 90.2 to 95.5, tied for the lead
  3. Jev90.8, interval 88.1 to 93.8, tier 2
  4. Shisa DE-184.2, interval 77.3 to 89.1, tier 3
  5. Eikos-4B78.6, interval 74.5 to 85.3, tier 3
  6. SemIf (Qwen3.5-4B)66.1, interval 60.9 to 74.6, tier 4
  7. Kev-9B62.5, interval 0.2 to 66.4, tier 4
  8. Kev-0.8B22.8, interval 6.1 to 26.3, tier 5
Tested against Jev and 14 open models

Routing and classification

Share of tasks led · #4 of 16

  1. AutoJev-27BLeads 4 of 5
  2. Shisa DE-1Leads 3 of 4
  3. JevLeads 9 of 14
  4. Eikos-27B
    Leads 1 of 3
  5. SemIf (Qwen3.5-4B)Leads 1 of 3
  6. Kev-4BLeads 0 of 3
  7. Kev-9BLeads 0 of 2
  8. Eikos-4BLeads 0 of 2
See the 5 tasks

Routing, 20 intents

Trained on the test data, not ranked

Accuracy, % · Higher is better

  1. Kev-4B*93.0*, interval 91.4 to 94.5, trained on this data, not ranked
  2. Kev-9B*93.0*, interval 91.4 to 94.6, trained on this data, not ranked
  3. Kev-0.8B*91.3*, interval 89.5 to 93.0, trained on this data, not ranked
  4. AutoJev-27B89.7, interval 87.7 to 91.6, ahead alone
  5. Eikos-27B*
    89.6*, interval 87.7 to 91.4, trained on this data, not ranked
  6. Kev-0.6B*89.5*, interval 87.5 to 91.5, trained on this data, not ranked
  7. Jev89.1, interval 87.0 to 91.0, tier 2
  8. Shisa DE-188.1, interval 85.9 to 90.1, tier 2
Tested against Jev and 14 open models

Routing, 77 intents

Trained on the test data, not ranked

Accuracy, % · Higher is better

  1. Kev-4B*85.0*, interval 81.8 to 88.0, trained on this data, not ranked
  2. Kev-9B*85.0*, interval 81.8 to 88.2, trained on this data, not ranked
  3. Kev-0.8B*83.0*, interval 79.6 to 86.4, trained on this data, not ranked
  4. AutoJev-27B80.6, interval 77.0 to 84.0, tied for the lead
  5. Jev79.6, interval 76.0 to 83.0, tied for the lead
  6. Kev-0.6B*78.2*, interval 74.4 to 81.8, trained on this data, not ranked
  7. Eikos-27B*
    77.8*, interval 74.0 to 81.4, trained on this data, not ranked
  8. Eikos-4B*74.0*, interval 70.0 to 77.8, trained on this data, not ranked
Tested against Jev and 14 open models

Tool routing

Tier 2 of 10#4 / 16

Accuracy, % · Higher is better

  1. Shisa DE-181.3, interval 78.9 to 83.5, tied for the lead
  2. AutoJev-27B80.6, interval 78.1 to 82.9, tied for the lead
  3. SemIf (Qwen3.5-4B)79.9, interval 77.4 to 82.2, tied for the lead
  4. Eikos-27B
    79.3, interval 76.8 to 81.8, tier 2
  5. Jev78.3, interval 75.7 to 80.7, tier 2
  6. Eikos-4B75.5, interval 72.8 to 78.0, tier 3
  7. Kev-9B72.4, interval 69.7 to 75.0, tier 4
  8. Kev-4B71.4, interval 68.6 to 74.0, tier 4
Tested against Jev and 14 open models

Complaint routing

Tier 2 of 8#3 / 15

Accuracy, % · Higher is better

  1. Jev78.7, interval 76.3 to 81.2, tied for the lead
  2. Shisa DE-178.0, interval 75.4 to 80.6, tied for the lead
  3. Kev-4B76.2, interval 73.6 to 79.0, tier 2
  4. AutoJev-27B75.9, interval 73.2 to 78.7, tier 2
  5. Eikos-27B
    75.6, interval 73.0 to 78.4, tier 2
  6. Kev-9B75.0, interval 72.4 to 77.7, tier 2
  7. SemIf (Qwen3.5-4B)72.9, interval 70.2 to 75.7, tier 3
  8. Kev-0.6B59.3, interval 56.3 to 62.3, tier 4
Tested against Jev and 13 open models

Typed decisions

Tied for the lead#1 / 14

Accuracy, % · Higher is better

  1. Laya-typed*77.4*, interval 75.4 to 79.4, trained on this data, not ranked
  2. Shisa DE-175.1, interval 72.8 to 77.3, tied for the lead
  3. Jev73.9, interval 71.8 to 76.0, tied for the lead
  4. AutoJev-27B73.8, interval 71.7 to 75.9, tied for the lead
  5. Eikos-27B
    73.4, interval 71.2 to 75.6, tied for the lead
  6. Kev-4B65.8, interval 63.6 to 68.0, tier 2
  7. Eikos-4B65.3, interval 62.7 to 68.0, tier 2
  8. SemIf (Qwen3.5-4B)63.0, interval 60.5 to 65.4, tier 3
Tested against Jev and 13 open models

Grounding

Share of tasks led · #1 of 15

  1. Eikos-27B
    Leads 2 of 2
  2. JevLeads 4 of 6
  3. AutoJev-27BLeads 0 of 2
  4. Shisa DE-1Leads 0 of 2
  5. Eikos-4BLeads 0 of 2
  6. Kev-4BLeads 0 of 2
  7. Kev-9BLeads 0 of 2
  8. Kev-0.8BLeads 0 of 4
See the 2 tasks

RAG faithfulness

Tied for the lead#1 / 13

Balanced accuracy, % · Higher is better

  1. Eikos-27B
    82.1, interval 79.2 to 84.9, tied for the lead
  2. Jev80.3, interval 77.5 to 83.1, tied for the lead
  3. Shisa DE-179.2, interval 76.2 to 82.2, tier 2
  4. AutoJev-27B79.1, interval 76.1 to 82.0, tier 2
  5. Kev-9B73.1, interval 70.0 to 76.0, tier 3
  6. Kev-4B72.5, interval 69.4 to 75.5, tier 3
  7. SemIf (Qwen3.5-4B)72.0, interval 68.9 to 75.2, tier 3
  8. Eikos-4B71.0, interval 67.4 to 74.5, tier 3
Tested against Jev and 14 open models

Search relevance

Tied for the lead#1 / 16

Accuracy, % · Higher is better

  1. Jev57.7, interval 54.7 to 60.8, tied for the lead
  2. Eikos-27B
    56.6, interval 53.4 to 59.7, tied for the lead
  3. AutoJev-27B52.8, interval 49.6 to 55.9, tier 2
  4. Shisa DE-150.3, interval 47.1 to 53.6, tier 2
  5. Kev-4B44.8, interval 41.7 to 48.0, tier 3
  6. Kev-9B43.4, interval 40.2 to 46.5, tier 3
  7. Eikos-4B43.3, interval 40.1 to 46.3, tier 3
  8. SemIf (Qwen3.5-4B)41.7, interval 38.6 to 44.7, tier 4
Tested against Jev and 14 open models

Agents

Leads 0 of 1, too few tasks to compare here.

See the task

Web-agent actions

Tier 2 of 10#2 / 15

Accuracy, % · Higher is better

  1. Jev70.8, interval 68.9 to 72.6, ahead alone
  2. Eikos-27B
    68.4, interval 66.5 to 70.4, tier 2
  3. AutoJev-27B68.2, interval 66.2 to 70.2, tier 2
  4. Shisa DE-165.2, interval 63.3 to 67.2, tier 3
  5. Eikos-4B62.9, interval 60.9 to 64.8, tier 4
  6. SemIf (Qwen3.5-4B)59.5, interval 57.5 to 61.5, tier 5
  7. Kev-4B58.0, interval 55.9 to 60.2, tier 5
  8. SimpleJev (Qwen3.5-0.8B)56.3, interval 54.7 to 57.8, tier 6
Tested against Jev and 13 open models

General decisions

Leads 0 of 1, too few tasks to compare here.

See the 2 tasks

Community sets

Tier 2 of 5#3 / 10

Accuracy, % · Higher is better

  1. AutoJev-27B63.0, interval 60.2 to 65.8, tied for the lead
  2. Jev62.3, interval 59.5 to 65.1, tied for the lead
  3. Eikos-27B
    59.8, interval 56.9 to 62.6, tier 2
  4. Eikos-4B55.8, interval 52.9 to 58.7, tier 3
  5. Kev-4B55.8, interval 53.0 to 58.7, tier 3
  6. Laya-typed47.8, interval 44.9 to 50.5, tier 4
  7. Kev-0.8B46.4, interval 43.5 to 49.3, tier 4
  8. Laya45.9, interval 43.1 to 48.8, tier 4
Tested against Jev and 13 open models

JevBench public

Trained on the test data, not ranked

Accuracy, % · Higher is better

  1. Eikos-27B*
    91.8*, interval 87.9 to 95.2, trained on this data, not ranked
  2. AutoJev-27B86.6, interval 81.9 to 90.9, tied for the lead
  3. Shisa DE-186.1, interval 81.2 to 90.6, tied for the lead
  4. Jev85.7, interval 81.0 to 90.1, tied for the lead
  5. Eikos-4B*85.3*, interval 80.4 to 89.8, trained on this data, not ranked
  6. SemIf (Qwen3.5-4B)80.1, interval 74.7 to 85.3, tier 2
  7. Kev-4B71.4, interval 65.2 to 77.4, tier 3
  8. Kev-0.6B60.6, interval 53.8 to 67.1, tier 4
Tested against Jev and 13 open models

Speed

Median latency, p50 · H100 only · lower is better

  1. Kev-9B43 ms
  2. Shisa DE-146 ms
  3. Eikos-27B
    82 ms
  4. AutoJev-27B99 ms
How we count

A model leads a task when it is in the task's leading tie tier. Each task counts inside its own benchmark, against that benchmark's models; no score is averaged. An area chart shows the models ranked on at least 2 of the area's tasks in a benchmark this model is in too. The overall standing counts every non-coding tasks and needs 5 for a rank. Latency is compared only on one hardware tier.

Eikos-27BOther modelsTrained on the test data*Whisker: 95% interval