Measure the rung.
Keep the evidence.
A private GPU scorer for deterministic language-model benchmarks. Same pair IDs, same tokenizer boundaries, same fp32 protocol—ready for controlled small-model comparisons.
Paste a Hugging Face model.
The RTX 3090 loads the model, scores up to 2,000 provisional Polish agreement pairs, then returns continuous pair probabilities. First run includes model download.
Make Polish model scaling predictable.
We are not starting with a giant model and hoping the measurements catch up. We are building an instrument that tells us what the next rung can teach us.
| Rung | Scale | Primary signal | Decision |
|---|---|---|---|
| 01 | 8M | BPB · morphology · agreement | Foundations |
| 02 | 16M | BPB · minimal pairs | Language structure |
| 03 | 32M | Pairs · short composition | Data mix |
| 04 | 64M | Structured language · reasoning-lite | Capability separation |
| 05 | 125M | MC correct-choice BPB | Useful comparison |
| 06 | 250M | Recall · composition · basic code | Scaling check |
| 07 | 500M | Multi-step tasks | Prediction update |
| 08–12 | 1B → 20B | Reasoning · tools · long context | Stop or scale |
Measure loss, morphology, pairs, composition and useful capability only when the scale can separate it.
Train, measure, compare, update the curve, predict the next scale, then spend the next order of magnitude of compute.
The valuable output is the curve connecting 8M to 20B: what moved it, what saturated and what was worth the compute.
Measured Polish micro
2,000 identical provisional agreement pairs. Pair probability is continuous; accuracy is shown only as a diagnostic. BPB is not included because the frozen validation corpus is not yet available.
| Model | Sentence pair probability | Critical-region probability | Time |
|---|---|---|---|
| Koliber v1.1 | 0.9083 | 0.8913 | 111.3 s |
| Pollock 1.4 | 0.5914 | 0.5718 | 100.4 s |
What this service measures
Sentence and critical-region pair probabilities, with raw log margins retained.
One worker, serialized requests, fixed protocol and no generation sampling.
Revision, parameter count, tokenizer hash, device and runtime are returned.
Loopback-only by default. Reach it through SSH forwarding or a private authenticated proxy.
The workflow
Train 32M → measure it → compare against 16M → update scaling curves → predict 64M → train 64M → check the prediction.
If a larger run misses its predicted loss or capability profile, stop and understand why. The benchmark is a measurement instrument with different sensitivity ranges, not a static collection of standard scores.
API
Run the micro battery from a model id or URL, or send your own one to 2,000 pair records. The service serializes GPU work.
curl -X POST https://eval.fabryka.ai/v1/benchmark/micro \
-H 'content-type: application/json' \
-d '{"model":"org/model","limit":2000}'
curl -X POST https://eval.fabryka.ai/v1/score/pairs \
-H 'content-type: application/json' \
--data-binary @pairs-request.json