← back to TRAINED

Laya Decision Model Fine-Tuning

The closed decision API Jev cannot be trained, so it cannot improve on topics it does not know. I fine-tuned Laya, an open-source decision model that runs in 2-3GB of GPU memory, and compared both on the same test sets: on bank customer questions it got 90 of 100 right versus Jev's 79, and on Korean hate-speech comments 79% versus 51% had every category right.

year
2026
tags
decision-model, laya, fine-tuning, jev, evaluation, banking77, k-mhas

What I did

The decision model Jev cannot be trained, so there is no way to make it better on topics it does not know well. I trained Laya, an open-source decision model that runs on a small GPU, to see whether it can get past that limit.

What a decision model is

Instead of writing text, it picks an answer from fixed options and returns how confident it is (0 to 1). It answers questions such as “Which type is this customer question?” or “Is this comment hate speech?” Jev, which TypeSafe released in September 2026 as “the first System One model”, is a leading example.

Why an open-source model

  • Jev is API-only. Data leaves the company, and there is no way to train it on company data.
  • Laya can be installed and trained in-house. It is Apache-2.0, and one trained model runs in 2–3GB of GPU memory. Response time is close to Jev’s (median 0.16s vs 0.19s).
  • I served it with the same request format as Jev and sent both models the same questions.

Results

  • Bank customer questions: Out of 100, Jev got 79 right and the trained Laya 90 (accuracy 79.4% → 90.4%).
  • Automation: Among confident answers (confidence 0.9 or higher), 6.4% of Jev’s were wrong and 3.3% of the trained Laya’s (precision 93.6% → 96.7%). Laya makes fewer mistakes while handling more questions automatically (coverage 81% vs Jev’s 65%). With training followed by temperature scaling, which aligns confidence with actual accuracy, ECE (the gap between confidence and accuracy, 0 is ideal) on a 400-question calibration set fell from 0.633 before training to 0.039.
  • Korean hate-speech comments: All 8 categories were right (exact match) for 51% of comments with Jev and 79% with the trained Laya. Macro F1, the mean of per-category F1 (harmonic mean of precision and recall), rose from 0.517 to 0.733. Of the comments Jev flagged as profanity, only 34% really were (precision); for Laya it was 90%.
  • A topic it was not trained on: On retail questions Jev scored 97% and Laya 80%. Training is what makes the difference.

How I trained it

  • Method: Trained for 4 epochs with RLCD, a reinforcement-learning objective based on a scoring rule (seed 42).
  • Data: Trained on public datasets, 10,000 Banking77 bank questions (CC BY 4.0) and 30,000 K-MHaS Korean news comments (CC BY-SA 4.0). Evaluation used the official test data, which was not used in training.
  • Hardware and time: On one RTX 5090, bank questions took 35 minutes and Korean comments 2 hours 42 minutes.
  • Problems solved: With default settings, the 77 options were cut off by the input length limit, so the model could not choose properly; raising the length fixed it. Error analysis also showed the question wording did not match the dataset’s categories (for example, sexual orientation belongs to “gender”), so I fixed the wording and retrained.

Limits

  • Tested on public data, not company data.
  • Each experiment was trained once, and training and test data come from the same source.
  • Jev cannot be trained, so it was compared as is.
  • Hate-speech examples are not shown.

Jev (closed API)Laya, before trainingLaya, after training

Bank customer questions

Correct out of 100

  • Jev (closed API)79.4%
  • Laya, before training36.3%
  • Laya, after training90.4%

77 question types, 3,080 test questions (Banking77)

Metric: accuracy

Errors in confident answers

Share wrong among answers with confidence 0.9 or higher (lower is better)

  • Jev (closed API)6.4%
  • Laya, before training38.6%
  • Laya, after training3.3%

Automating only these answers covers 65% of questions with Jev and 81% with the tuned Laya.

Metric: error rate at confidence ≥ 0.9 (1 − precision), read together with coverage

Korean hate-speech comments

Share with all 8 categories right (origin, looks, politics, profanity, age, gender, race, religion)

  • Jev (closed API)51.3%
  • Laya, before training36.1%
  • Laya, after training79.4%

3,000 test news comments (K-MHaS)

Metric: exact match ratio. Macro F1 on the same test (mean of per-category F1, 1 is perfect): Jev 0.517, before 0.207, after 0.733

Profanity flags

Share of comments flagged as profanity that really are

  • Jev (closed API)34.4%
  • Laya, before training39.0%
  • Laya, after training90.4%

Jev rarely misses profanity but flags many comments that are not.

Metric: precision on the profanity category. Profanity F1 (harmonic mean of precision and recall): Jev 0.483, after training 0.898

A topic Laya was not trained on

Retail customer questions, correct out of 100

  • Jev (closed API)97.2%
  • Laya, before training80.3%

On a topic it was not trained on, Laya trails Jev. Training is what closes the gap.

Metric: accuracy