LabZone Labs — own experiment (not a client case study)

LabZone Labs · W47 · 19 Nov 2026 · Status: Done

Decision models vs. a local LLM: who should make the choices in a quoting assistant?

Try the quoting assistant →The demo runs the local model. The catalogue, prices and test enquiries are fictional.

The problem

Our quoting assistant reads a customer enquiry with a local language model, and a deterministic engine computes the price. The model never calculates. It decides: which product, which surface, which foil, timer yes or no, which town. Those are choices from a fixed list.

A new class of models is built for exactly that. Gemini Decision 1 from Google and Jev from TypeSafe do not write text. You give them an input and questions with fixed options, and they return a probability for each option in one pass. The pitch is speed, low cost and “no hallucinations”.

The question for this week: if the model’s job is to choose, does a model built only for choosing do it more accurately and faster, and is there a local one that works in Slovak?

What we built

Six other ways to make the same decisions inside the same calculator, next to the one it uses today:

Engine Where it runs
today Qwen 3.6 35B extracting all parameters (also numbers and text) local GPU
cloud Gemini Decision 1, one request with all questions Google
A Qwen 3.6 35B answering the same choice questions as Decision local GPU
B Small local models on the same questions: Qwen 3.5 9B, Gemma 4 E2B local GPU
C Zero-shot classifier: multilingual mDeBERTa (NLI), no text generation CPU
D OpenJev-4B, an open model that copies Jev’s interface Mac (MPS)

Each engine gets the same questions, generated from the catalogue file: the product (four catalogue products, “not in the catalogue”, “unclear”) and one question per choice parameter, each with an extra option “not stated”. Answers are mapped into the shape the calculator already uses, so the pricing engine does not change.

We scored only what all engines can do. A decision model cannot return a number or the text of a sign, so dimensions and sign text are left out of the comparison.

Hypotheses (written before the run)

  1. Framing, not the model. Our local LLM, asked the same choice questions as Decision (A), is at least as accurate as today and fills in fewer unstated values.
  2. Small and fast. A small local model (B) answers in under 2 s with at most 5 more errors than A.
  3. CPU classifier. A zero-shot classifier (C) routes the product well (≥ 33/36) but fails on corrections and negations.
  4. Disagreement as a check. When two engines disagree, a human looks. That catches ≥ 80 % of errors while flagging < 10 % of enquiries.

Kill criterion: if A matches Decision, the cloud adds nothing for decisions.

The test set

How long it took

One day (19 Nov 2026), including the harder test set and vetting the open-source alternatives.

What we measured

19 Nov 2026, 36 Slovak enquiries, 152 decisions, temperature 0. Today’s engine and Gemini Decision 1 were run twice with the same result; the others once.

Engine Correct Product right Filled in, not stated Enquiries without error Median time
Gemini Decision 1 (cloud) 150 / 152 36 / 36 1 34 / 36 0.6 s
Qwen 3.6 35B, extraction (today) 149 / 152 36 / 36 2 33 / 36 8.4 s
A · Qwen 3.6 35B, choice questions 142 / 152 33 / 36 0 28 / 36 6.2 s
B · Qwen 3.5 9B, choice questions 141 / 152 30 / 36 1 28 / 36 12.7 s
D · OpenJev-4B 135 / 152 34 / 36 10 22 / 36 9.5 s
B · Gemma 4 E2B, choice questions 133 / 152 30 / 36 5 23 / 36 3.7 s
C · mDeBERTa zero-shot (CPU) 99 / 152 17 / 36 15 9 / 36 3.1 s

Times are per enquiry after warm-up. Today’s engine also reads numbers and text in those 8 s. The GPU server is shared with our other demos, and OpenJev ran on a Mac without its optimised kernels, so treat local times as rough.

Check by disagreement (a human sees every enquiry where the two engines differ):

Pair Enquiries to check Errors caught
today + Gemini Decision 1 3 / 36 4 / 4
today + A (all local) 10 / 36 13 / 13
today + B Qwen 9B (all local) 9 / 36 14 / 14
today + D OpenJev 15 / 36 19 / 19

Hypotheses, verdict

  1. Rejected. The same model asked choice questions got worse (142 vs 149). It became overly cautious: “insulated facade” turned into “surface not stated”. It never filled in an unstated value, so its errors are the safe kind, but the questions alone are not what makes Decision good.
  2. Rejected. No small model came close: Gemma was fast enough but made 19 errors; Qwen 9B was accurate-ish but slower than the big model on a shared server.
  3. Half confirmed, half worse than expected. The CPU classifier was weak everywhere, product routing included (17 / 36). It never answered “unclear”, “not in the catalogue” or “not stated”, and it defaulted to the same town.
  4. Confirmed for today + Decision. Both engines’ errors showed up as disagreements, 4 of 4, with 3 enquiries out of 36 sent to a human. All-local pairs also caught everything, but flag a quarter of the enquiries.

What we learned

About Jev and the “open alternatives”

TypeSafe’s Jev is a hosted model with no public weights. Its claims (70–500 ms, “can’t hallucinate”) are self-reported and its quality evaluation compares it against other frontier models, not against checked answers. We did not have access, so it is not in the table. The open code TypeSafe publishes is a comparison adapter that makes ordinary LLMs return probabilities; it is not Jev.

Searching for local Jev alternatives returns a mix. We checked each before installing anything:

What we recommend

Next step: a Slovak decision model of our own

The limitation is the language, so the fix is training, not prompting. A small multilingual encoder (100–300M parameters, e.g. mmBERT or XLM-R) fine-tuned on Slovak could answer these questions in well under a second on a CPU. The labels do not have to be written by hand:

  1. The big local model labels a few thousand generated enquiries, and a person checks a sample.
  2. Every quote a technologist approves or corrects is a checked example, for free.

The catch: a new product or a new allowed value means retraining, where today we only edit a YAML file. That is the next experiment.

Limits of this experiment

36 enquiries, 16 of them written by us, mostly one run per engine. Enough to see direction and failure modes, not enough to quote percentages as a guarantee. Local timings come from a shared GPU server and a Mac, not dedicated hardware.

Where this could be used

Who it’s for

Teams that already use AI to fill in structured data and want to know when to trust it, and companies deciding whether a cloud decision model is worth sending customer text out of the house.

How we deploy local AI for companies →

We'll show this on your data.

We start with a free 15-minute audit. If it makes sense, we prove local AI on a sample of your data (1–3 weeks, from €500 excl. VAT). Only then, if we agree, we build a working prototype in 2–4 weeks. The solution will run in your environment. Your data stays yours.

Book a free 15-minute auditFree · 15 min · online

Calendar not working? Email us: info@labzone.tech