Examples
Examples
Try the starting example
- JSON Paired per-example scores [model A,model B]
- [[1,0],[1,1],[0,1],[1,0],[1,1]]
- Bootstrap replicates (count)
- 1000
- Deterministic random seed (integer)
- 42
- Mean paired score difference
- 0.2
- Lower percentile bound
- -0.4
- Upper percentile bound
- 0.8
Try another scenario
- JSON Paired per-example scores [model A,model B]
- [[1,0],[1,0],[1,1],[0,0],[1,1]]
- Bootstrap replicates (count)
- 1000
- Deterministic random seed (integer)
- 42
- Mean paired score difference
- 0.4
- Lower percentile bound
- 0
- Upper percentile bound
- 0.8
Calculation method
Calculation method
Mean paired score difference
Mean(A−B) over supplied pairs.
Lower percentile bound
Nearest-rank 2.5th percentile of seeded paired resampled mean differences.
Upper percentile bound
Nearest-rank 97.5th percentile of seeded paired resampled mean differences.
Symbols and units
- a
- JSON Paired per-example scores [model A,model B]
- b
- Bootstrap replicates (count)
- c
- Deterministic random seed (integer)
Use the units shown. A list keeps the order in which you enter its values. The symbol ^ means “raised to the power of”.
Assumptions & limits
Assumptions & limits
- Paired percentile bootstrap over independent examples, fixed 95% nominal interval. Assumes sampling unit is independent; not a guarantee about future model quality. Python Random MT19937 is the specified seeded reference stream. Supported limits: 2–1000 pairs, 100–10000 resamples. Same ordered pairs for both models. Seed integer 0–4294967295; Python-compatible integer-seeded MT19937 and rejection-sampled randrange are specified for reproducibility, not secrecy.
Are these inputs private?
These calculations run in your browser. A shared link contains the values you choose to share. Only questions submitted to the separate AI finder are sent to OpenAI.
Further reading
Background reading; this does not imply endorsement or independent review of SumFizz.