The Book of Jev
Chapter 14 of 36

Jev: Full Uncertainty and Calibration Tables

Explain this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Apply this chapter with AI

Copy this prompt into ChatGPT, Claude, Gemini, a local model, or another AI.

Return to Jev From First Principles.

These tables preserve all rows and values from the enhanced edition. Columns are transposed into metric/value records for small screens. Chapter 10’s displayed examples use seed 0; its acceptance verdicts use medians across fit seeds. Bootstrap intervals need not contain a finite-sample point estimate.

Chapter 9, table 1

Return to Chapter 9.

Record 1: intent/test — prior/s0/w0 — 0.0064 [0.0028, 0.0103]

Field Value
Dataset/split intent/test
Provider/seed/wording prior/s0/w0
ECE10 width [95% bootstrap] 0.0064 [0.0028, 0.0103]
Classwise ECE10 0.0026
Brier sum 0.9878
Log loss 4.3820
Accuracy 0.0130
AUROC 0.5000
n 3080

Record 2: intent/test — tfidf-lr/s0/w0 — 0.0519 [0.0430, 0.0616]

Field Value
Dataset/split intent/test
Provider/seed/wording tfidf-lr/s0/w0
ECE10 width [95% bootstrap] 0.0519 [0.0430, 0.0616]
Classwise ECE10 0.0023
Brier sum 0.1835
Log loss 0.5147
Accuracy 0.8779
AUROC 0.9971
n 3080

Record 3: intent/test — fasttext-style/s0/w0 — 0.0408 [0.0343, 0.0540]

Field Value
Dataset/split intent/test
Provider/seed/wording fasttext-style/s0/w0
ECE10 width [95% bootstrap] 0.0408 [0.0343, 0.0540]
Classwise ECE10 0.0026
Brier sum 0.2251
Log loss 0.6051
Accuracy 0.8526
AUROC 0.9962
n 3080

Record 4: intent/test — embed-sim/s0/w0 — 0.5630 [0.5469, 0.5784]

Field Value
Dataset/split intent/test
Provider/seed/wording embed-sim/s0/w0
ECE10 width [95% bootstrap] 0.5630 [0.5469, 0.5784]
Classwise ECE10 0.0100
Brier sum 0.8226
Log loss 2.4848
Accuracy 0.6799
AUROC 0.9804
n 3080

Record 5: safety/test — prior/s0/w0 — 0.1484 [0.0622, 0.2432]

Field Value
Dataset/split safety/test
Provider/seed/wording prior/s0/w0
ECE10 width [95% bootstrap] 0.1484 [0.0622, 0.2432]
Classwise ECE10 0.1484
Brier sum 0.5434
Log loss 0.7380
Accuracy 0.4828
AUROC 0.5000
n 116

Record 6: safety/test — tfidf-lr/s0/w0 — 0.0593 [0.0339, 0.1279]

Field Value
Dataset/split safety/test
Provider/seed/wording tfidf-lr/s0/w0
ECE10 width [95% bootstrap] 0.0593 [0.0339, 0.1279]
Classwise ECE10 0.0859
Brier sum 0.1710
Log loss 0.3392
Accuracy 0.8966
AUROC 0.9661
n 116

Record 7: safety/test — fasttext-style/s0/w0 — 0.0642 [0.0378, 0.1212]

Field Value
Dataset/split safety/test
Provider/seed/wording fasttext-style/s0/w0
ECE10 width [95% bootstrap] 0.0642 [0.0378, 0.1212]
Classwise ECE10 0.0743
Brier sum 0.1580
Log loss 0.2471
Accuracy 0.9052
AUROC 0.9658
n 116

Record 8: safety/test — embed-sim/s0/w0 — 0.0574 [0.0181, 0.1554]

Field Value
Dataset/split safety/test
Provider/seed/wording embed-sim/s0/w0
ECE10 width [95% bootstrap] 0.0574 [0.0181, 0.1554]
Classwise ECE10 0.0685
Brier sum 0.4827
Log loss 0.6753
Accuracy 0.5345
AUROC 0.6423
n 116

Record 9: safety/shift — prior/s0/w0 — 0.1617 [0.1044, 0.2266]

Field Value
Dataset/split safety/shift
Provider/seed/wording prior/s0/w0
ECE10 width [95% bootstrap] 0.1617 [0.1044, 0.2266]
Classwise ECE10 0.1617
Brier sum 0.5504
Log loss 0.7452
Accuracy 0.4695
AUROC 0.5000
n 262

Record 10: safety/shift — tfidf-lr/s0/w0 — 0.4375 [0.3754, 0.4937]

Field Value
Dataset/split safety/shift
Provider/seed/wording tfidf-lr/s0/w0
ECE10 width [95% bootstrap] 0.4375 [0.3754, 0.4937]
Classwise ECE10 0.4396
Brier sum 0.8646
Log loss 3.4633
Accuracy 0.5496
AUROC 0.8644
n 262

Record 11: safety/shift — fasttext-style/s0/w0 — 0.4361 [0.3731, 0.4934]

Field Value
Dataset/split safety/shift
Provider/seed/wording fasttext-style/s0/w0
ECE10 width [95% bootstrap] 0.4361 [0.3731, 0.4934]
Classwise ECE10 0.4378
Brier sum 0.8633
Log loss 2.1766
Accuracy 0.5382
AUROC 0.5115
n 262

Record 12: safety/shift — embed-sim/s0/w0 — 0.1861 [0.1209, 0.2420]

Field Value
Dataset/split safety/shift
Provider/seed/wording embed-sim/s0/w0
ECE10 width [95% bootstrap] 0.1861 [0.1209, 0.2420]
Classwise ECE10 0.1861
Brier sum 0.5652
Log loss 0.7600
Accuracy 0.3740
AUROC 0.3085
n 262

Chapter 10, table 1

Return to Chapter 10.

Record 1: intent/test — embed-sim — isotonic — 1000/0/w0

Field Value
Task/split intent/test
Provider embed-sim
Method isotonic
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.0863 [0.0743, 0.1001]
Classwise ECE 0.0056
Brier sum 0.4483
Log loss 3.0628
Accuracy 0.6873
AUROC 0.9572
n 3080

Record 2: intent/test — embed-sim — sigmoid — 1000/0/w0

Field Value
Task/split intent/test
Provider embed-sim
Method sigmoid
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.2241 [0.2108, 0.2375]
Classwise ECE 0.0070
Brier sum 0.4654
Log loss 1.2237
Accuracy 0.7097
AUROC 0.9851
n 3080

Record 3: intent/test — embed-sim — temperature — 1000/0/w0

Field Value
Task/split intent/test
Provider embed-sim
Method temperature
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.0187 [0.0156, 0.0354]
Classwise ECE 0.0055
Brier sum 0.4359
Log loss 1.2117
Accuracy 0.6799
AUROC 0.9861
n 3080

Record 4: intent/test — embed-sim — uncalibrated — 1000/0/w0

Field Value
Task/split intent/test
Provider embed-sim
Method uncalibrated
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.5630 [0.5469, 0.5784]
Classwise ECE 0.0100
Brier sum 0.8226
Log loss 2.4848
Accuracy 0.6799
AUROC 0.9804
n 3080

Record 5: intent/test — embed-sim — vector — 1000/0/w0

Field Value
Task/split intent/test
Provider embed-sim
Method vector
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.0551 [0.0428, 0.0683]
Classwise ECE 0.0041
Brier sum 0.3880
Log loss 1.0291
Accuracy 0.7159
AUROC 0.9880
n 3080

Record 6: intent/test — fasttext-style — isotonic — 1000/0/w0

Field Value
Task/split intent/test
Provider fasttext-style
Method isotonic
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.0423 [0.0344, 0.0541]
Classwise ECE 0.0029
Brier sum 0.2401
Log loss 2.4683
Accuracy 0.8474
AUROC 0.9663
n 3080

Record 7: intent/test — fasttext-style — sigmoid — 1000/0/w0

Field Value
Task/split intent/test
Provider fasttext-style
Method sigmoid
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.1125 [0.1006, 0.1246]
Classwise ECE 0.0040
Brier sum 0.2614
Log loss 0.6734
Accuracy 0.8468
AUROC 0.9963
n 3080

Record 8: intent/test — fasttext-style — temperature — 1000/0/w0

Field Value
Task/split intent/test
Provider fasttext-style
Method temperature
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.0256 [0.0186, 0.0391]
Classwise ECE 0.0026
Brier sum 0.2258
Log loss 0.5705
Accuracy 0.8526
AUROC 0.9963
n 3080

Record 9: intent/test — fasttext-style — uncalibrated — 1000/0/w0

Field Value
Task/split intent/test
Provider fasttext-style
Method uncalibrated
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.0408 [0.0343, 0.0540]
Classwise ECE 0.0026
Brier sum 0.2251
Log loss 0.6051
Accuracy 0.8526
AUROC 0.9962
n 3080

Record 10: intent/test — fasttext-style — vector — 1000/0/w0

Field Value
Task/split intent/test
Provider fasttext-style
Method vector
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.0260 [0.0224, 0.0399]
Classwise ECE 0.0029
Brier sum 0.2343
Log loss 0.6835
Accuracy 0.8477
AUROC 0.9961
n 3080

Record 11: intent/test — prior — uncalibrated — 0/0/w0

Field Value
Task/split intent/test
Provider prior
Method uncalibrated
Labels/seed/wording 0/0/w0
ECE10 width [95% bootstrap] 0.0064 [0.0028, 0.0103]
Classwise ECE 0.0026
Brier sum 0.9878
Log loss 4.3820
Accuracy 0.0130
AUROC 0.5000
n 3080

Record 12: intent/test — tfidf-lr — isotonic — 1000/0/w0

Field Value
Task/split intent/test
Provider tfidf-lr
Method isotonic
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.0432 [0.0347, 0.0535]
Classwise ECE 0.0023
Brier sum 0.1919
Log loss 1.8461
Accuracy 0.8731
AUROC 0.9754
n 3080

Record 13: intent/test — tfidf-lr — sigmoid — 1000/0/w0

Field Value
Task/split intent/test
Provider tfidf-lr
Method sigmoid
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.0832 [0.0729, 0.0933]
Classwise ECE 0.0032
Brier sum 0.1974
Log loss 0.5254
Accuracy 0.8766
AUROC 0.9973
n 3080

Record 14: intent/test — tfidf-lr — temperature — 1000/0/w0

Field Value
Task/split intent/test
Provider tfidf-lr
Method temperature
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.0117 [0.0093, 0.0238]
Classwise ECE 0.0021
Brier sum 0.1768
Log loss 0.4481
Accuracy 0.8779
AUROC 0.9972
n 3080

Record 15: intent/test — tfidf-lr — uncalibrated — 1000/0/w0

Field Value
Task/split intent/test
Provider tfidf-lr
Method uncalibrated
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.0519 [0.0430, 0.0616]
Classwise ECE 0.0023
Brier sum 0.1835
Log loss 0.5147
Accuracy 0.8779
AUROC 0.9971
n 3080

Record 16: intent/test — tfidf-lr — vector — 1000/0/w0

Field Value
Task/split intent/test
Provider tfidf-lr
Method vector
Labels/seed/wording 1000/0/w0
ECE10 width [95% bootstrap] 0.0229 [0.0167, 0.0339]
Classwise ECE 0.0023
Brier sum 0.1847
Log loss 0.5610
Accuracy 0.8766
AUROC 0.9971
n 3080

Record 17: safety/shift — embed-sim — isotonic — 90/0/w0

Field Value
Task/split safety/shift
Provider embed-sim
Method isotonic
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.2592 [0.2116, 0.3295]
Classwise ECE 0.2570
Brier sum 0.7001
Log loss 1.5463
Accuracy 0.4046
AUROC 0.3386
n 262

Record 18: safety/shift — embed-sim — sigmoid — 90/0/w0

Field Value
Task/split safety/shift
Provider embed-sim
Method sigmoid
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.2893 [0.2294, 0.3491]
Classwise ECE 0.2893
Brier sum 0.6545
Log loss 0.8629
Accuracy 0.3550
AUROC 0.3085
n 262

Record 19: safety/shift — embed-sim — temperature — 90/0/w0

Field Value
Task/split safety/shift
Provider embed-sim
Method temperature
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.2447 [0.1970, 0.3030]
Classwise ECE 0.2598
Brier sum 0.6483
Log loss 0.8581
Accuracy 0.3740
AUROC 0.3085
n 262

Record 20: safety/shift — embed-sim — uncalibrated — 90/0/w0

Field Value
Task/split safety/shift
Provider embed-sim
Method uncalibrated
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.1861 [0.1209, 0.2420]
Classwise ECE 0.1861
Brier sum 0.5652
Log loss 0.7600
Accuracy 0.3740
AUROC 0.3085
n 262

Record 21: safety/shift — embed-sim — vector — 90/0/w0

Field Value
Task/split safety/shift
Provider embed-sim
Method vector
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.2660 [0.2124, 0.3302]
Classwise ECE 0.2660
Brier sum 0.6362
Log loss 0.8407
Accuracy 0.3702
AUROC 0.3085
n 262

Record 22: safety/shift — fasttext-style — isotonic — 90/0/w0

Field Value
Task/split safety/shift
Provider fasttext-style
Method isotonic
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.2796 [0.2245, 0.3391]
Classwise ECE 0.2861
Brier sum 0.6516
Log loss 0.9323
Accuracy 0.5725
AUROC 0.5409
n 262

Record 23: safety/shift — fasttext-style — sigmoid — 90/0/w0

Field Value
Task/split safety/shift
Provider fasttext-style
Method sigmoid
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.1572 [0.1129, 0.2215]
Classwise ECE 0.1699
Brier sum 0.5610
Log loss 0.8021
Accuracy 0.5878
AUROC 0.5115
n 262

Record 24: safety/shift — fasttext-style — temperature — 90/0/w0

Field Value
Task/split safety/shift
Provider fasttext-style
Method temperature
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.3095 [0.2461, 0.3675]
Classwise ECE 0.3138
Brier sum 0.6931
Log loss 1.0495
Accuracy 0.5382
AUROC 0.5115
n 262

Record 25: safety/shift — fasttext-style — uncalibrated — 90/0/w0

Field Value
Task/split safety/shift
Provider fasttext-style
Method uncalibrated
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.4361 [0.3731, 0.4934]
Classwise ECE 0.4378
Brier sum 0.8633
Log loss 2.1766
Accuracy 0.5382
AUROC 0.5115
n 262

Record 26: safety/shift — fasttext-style — vector — 90/0/w0

Field Value
Task/split safety/shift
Provider fasttext-style
Method vector
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.1747 [0.1164, 0.2344]
Classwise ECE 0.1960
Brier sum 0.5784
Log loss 0.8108
Accuracy 0.5573
AUROC 0.5115
n 262

Record 27: safety/shift — prior — uncalibrated — 0/0/w0

Field Value
Task/split safety/shift
Provider prior
Method uncalibrated
Labels/seed/wording 0/0/w0
ECE10 width [95% bootstrap] 0.1617 [0.1044, 0.2266]
Classwise ECE 0.1617
Brier sum 0.5504
Log loss 0.7452
Accuracy 0.4695
AUROC 0.5000
n 262

Record 28: safety/shift — tfidf-lr — isotonic — 90/0/w0

Field Value
Task/split safety/shift
Provider tfidf-lr
Method isotonic
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.4094 [0.3470, 0.4646]
Classwise ECE 0.4119
Brier sum 0.8023
Log loss 13.1233
Accuracy 0.6031
AUROC 0.5903
n 262

Record 29: safety/shift — tfidf-lr — sigmoid — 90/0/w0

Field Value
Task/split safety/shift
Provider tfidf-lr
Method sigmoid
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.3701 [0.3152, 0.4266]
Classwise ECE 0.3867
Brier sum 0.7205
Log loss 1.2790
Accuracy 0.5611
AUROC 0.8644
n 262

Record 30: safety/shift — tfidf-lr — temperature — 90/0/w0

Field Value
Task/split safety/shift
Provider tfidf-lr
Method temperature
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.4103 [0.3491, 0.4644]
Classwise ECE 0.4176
Brier sum 0.7953
Log loss 1.7027
Accuracy 0.5496
AUROC 0.8644
n 262

Record 31: safety/shift — tfidf-lr — uncalibrated — 90/0/w0

Field Value
Task/split safety/shift
Provider tfidf-lr
Method uncalibrated
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.4375 [0.3754, 0.4937]
Classwise ECE 0.4396
Brier sum 0.8646
Log loss 3.4633
Accuracy 0.5496
AUROC 0.8644
n 262

Record 32: safety/shift — tfidf-lr — vector — 90/0/w0

Field Value
Task/split safety/shift
Provider tfidf-lr
Method vector
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.3998 [0.3416, 0.4531]
Classwise ECE 0.4098
Brier sum 0.7813
Log loss 1.8997
Accuracy 0.5573
AUROC 0.8644
n 262

Record 33: safety/test — embed-sim — isotonic — 90/0/w0

Field Value
Task/split safety/test
Provider embed-sim
Method isotonic
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.2102 [0.1420, 0.3084]
Classwise ECE 0.2091
Brier sum 0.5611
Log loss 0.7815
Accuracy 0.5172
AUROC 0.6629
n 116

Record 34: safety/test — embed-sim — sigmoid — 90/0/w0

Field Value
Task/split safety/test
Provider embed-sim
Method sigmoid
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.1950 [0.1330, 0.2867]
Classwise ECE 0.2182
Brier sum 0.5441
Log loss 0.7443
Accuracy 0.4914
AUROC 0.6423
n 116

Record 35: safety/test — embed-sim — temperature — 90/0/w0

Field Value
Task/split safety/test
Provider embed-sim
Method temperature
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.1209 [0.0597, 0.2088]
Classwise ECE 0.1378
Brier sum 0.5008
Log loss 0.6955
Accuracy 0.5345
AUROC 0.6423
n 116

Record 36: safety/test — embed-sim — uncalibrated — 90/0/w0

Field Value
Task/split safety/test
Provider embed-sim
Method uncalibrated
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.0574 [0.0181, 0.1554]
Classwise ECE 0.0685
Brier sum 0.4827
Log loss 0.6753
Accuracy 0.5345
AUROC 0.6423
n 116

Record 37: safety/test — embed-sim — vector — 90/0/w0

Field Value
Task/split safety/test
Provider embed-sim
Method vector
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.1820 [0.1231, 0.2793]
Classwise ECE 0.2012
Brier sum 0.5397
Log loss 0.7378
Accuracy 0.5000
AUROC 0.6423
n 116

Record 38: safety/test — fasttext-style — isotonic — 90/0/w0

Field Value
Task/split safety/test
Provider fasttext-style
Method isotonic
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.0574 [0.0241, 0.1183]
Classwise ECE 0.1106
Brier sum 0.2023
Log loss 0.5848
Accuracy 0.8707
AUROC 0.9565
n 116

Record 39: safety/test — fasttext-style — sigmoid — 90/0/w0

Field Value
Task/split safety/test
Provider fasttext-style
Method sigmoid
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.0535 [0.0329, 0.1233]
Classwise ECE 0.1416
Brier sum 0.2176
Log loss 0.3420
Accuracy 0.8534
AUROC 0.9658
n 116

Record 40: safety/test — fasttext-style — temperature — 90/0/w0

Field Value
Task/split safety/test
Provider fasttext-style
Method temperature
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.0765 [0.0604, 0.1280]
Classwise ECE 0.0852
Brier sum 0.1594
Log loss 0.2710
Accuracy 0.9052
AUROC 0.9658
n 116

Record 41: safety/test — fasttext-style — uncalibrated — 90/0/w0

Field Value
Task/split safety/test
Provider fasttext-style
Method uncalibrated
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.0642 [0.0378, 0.1212]
Classwise ECE 0.0743
Brier sum 0.1580
Log loss 0.2471
Accuracy 0.9052
AUROC 0.9658
n 116

Record 42: safety/test — fasttext-style — vector — 90/0/w0

Field Value
Task/split safety/test
Provider fasttext-style
Method vector
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.0659 [0.0479, 0.1357]
Classwise ECE 0.1427
Brier sum 0.2002
Log loss 0.3328
Accuracy 0.8879
AUROC 0.9658
n 116

Record 43: safety/test — prior — uncalibrated — 0/0/w0

Field Value
Task/split safety/test
Provider prior
Method uncalibrated
Labels/seed/wording 0/0/w0
ECE10 width [95% bootstrap] 0.1484 [0.0622, 0.2432]
Classwise ECE 0.1484
Brier sum 0.5434
Log loss 0.7380
Accuracy 0.4828
AUROC 0.5000
n 116

Record 44: safety/test — tfidf-lr — isotonic — 90/0/w0

Field Value
Task/split safety/test
Provider tfidf-lr
Method isotonic
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.1045 [0.0502, 0.1647]
Classwise ECE 0.1045
Brier sum 0.2167
Log loss 0.5863
Accuracy 0.7845
AUROC 0.9604
n 116

Record 45: safety/test — tfidf-lr — sigmoid — 90/0/w0

Field Value
Task/split safety/test
Provider tfidf-lr
Method sigmoid
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.0099 [0.0223, 0.0905]
Classwise ECE 0.1076
Brier sum 0.1922
Log loss 0.3123
Accuracy 0.8707
AUROC 0.9661
n 116

Record 46: safety/test — tfidf-lr — temperature — 90/0/w0

Field Value
Task/split safety/test
Provider tfidf-lr
Method temperature
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.0144 [0.0213, 0.0852]
Classwise ECE 0.0698
Brier sum 0.1633
Log loss 0.2709
Accuracy 0.8966
AUROC 0.9661
n 116

Record 47: safety/test — tfidf-lr — uncalibrated — 90/0/w0

Field Value
Task/split safety/test
Provider tfidf-lr
Method uncalibrated
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.0593 [0.0339, 0.1279]
Classwise ECE 0.0859
Brier sum 0.1710
Log loss 0.3392
Accuracy 0.8966
AUROC 0.9661
n 116

Record 48: safety/test — tfidf-lr — vector — 90/0/w0

Field Value
Task/split safety/test
Provider tfidf-lr
Method vector
Labels/seed/wording 90/0/w0
ECE10 width [95% bootstrap] 0.0367 [0.0249, 0.1083]
Classwise ECE 0.1055
Brier sum 0.1955
Log loss 0.3107
Accuracy 0.8621
AUROC 0.9661
n 116

Chapter 10, table 2

Return to Chapter 10.

Record 1: intent — tfidf-lr — ece

Field Value
Task intent
Provider tfidf-lr
Metric ece
Difference [95% paired bootstrap] -0.0402 [-0.0447, -0.0264]
n 3080

Record 2: intent — tfidf-lr — brier

Field Value
Task intent
Provider tfidf-lr
Metric brier
Difference [95% paired bootstrap] -0.0067 [-0.0096, -0.0035]
n 3080

Record 3: intent — tfidf-lr — accuracy

Field Value
Task intent
Provider tfidf-lr
Metric accuracy
Difference [95% paired bootstrap] 0.0000 [0.0000, 0.0000]
n 3080

Record 4: intent — fasttext-style — ece

Field Value
Task intent
Provider fasttext-style
Metric ece
Difference [95% paired bootstrap] -0.0152 [-0.0305, -0.0000]
n 3080

Record 5: intent — fasttext-style — brier

Field Value
Task intent
Provider fasttext-style
Metric brier
Difference [95% paired bootstrap] 0.0007 [-0.0016, 0.0032]
n 3080

Record 6: intent — fasttext-style — accuracy

Field Value
Task intent
Provider fasttext-style
Metric accuracy
Difference [95% paired bootstrap] 0.0000 [0.0000, 0.0000]
n 3080

Record 7: intent — embed-sim — ece

Field Value
Task intent
Provider embed-sim
Metric ece
Difference [95% paired bootstrap] -0.5443 [-0.5500, -0.5223]
n 3080

Record 8: intent — embed-sim — brier

Field Value
Task intent
Provider embed-sim
Metric brier
Difference [95% paired bootstrap] -0.3867 [-0.4006, -0.3725]
n 3080

Record 9: intent — embed-sim — accuracy

Field Value
Task intent
Provider embed-sim
Metric accuracy
Difference [95% paired bootstrap] 0.0000 [0.0000, 0.0000]
n 3080

Record 10: safety — tfidf-lr — ece

Field Value
Task safety
Provider tfidf-lr
Metric ece
Difference [95% paired bootstrap] -0.0449 [-0.0703, 0.0268]
n 116

Record 11: safety — tfidf-lr — brier

Field Value
Task safety
Provider tfidf-lr
Metric brier
Difference [95% paired bootstrap] -0.0077 [-0.0315, 0.0152]
n 116

Record 12: safety — tfidf-lr — accuracy

Field Value
Task safety
Provider tfidf-lr
Metric accuracy
Difference [95% paired bootstrap] 0.0000 [0.0000, 0.0000]
n 116

Record 13: safety — fasttext-style — ece

Field Value
Task safety
Provider fasttext-style
Metric ece
Difference [95% paired bootstrap] 0.0124 [-0.0442, 0.0726]
n 116

Record 14: safety — fasttext-style — brier

Field Value
Task safety
Provider fasttext-style
Metric brier
Difference [95% paired bootstrap] 0.0014 [-0.0287, 0.0291]
n 116

Record 15: safety — fasttext-style — accuracy

Field Value
Task safety
Provider fasttext-style
Metric accuracy
Difference [95% paired bootstrap] 0.0000 [0.0000, 0.0000]
n 116

Record 16: safety — embed-sim — ece

Field Value
Task safety
Provider embed-sim
Metric ece
Difference [95% paired bootstrap] 0.0636 [-0.0053, 0.0865]
n 116

Record 17: safety — embed-sim — brier

Field Value
Task safety
Provider embed-sim
Metric brier
Difference [95% paired bootstrap] 0.0181 [-0.0124, 0.0500]
n 116

Record 18: safety — embed-sim — accuracy

Field Value
Task safety
Provider embed-sim
Metric accuracy
Difference [95% paired bootstrap] 0.0000 [0.0000, 0.0000]
n 116

Chapter 10, table 3

Return to Chapter 10.

Record 1: intent/test — tfidf-lr — temperature — 50/0/w0

Field Value
Task/split intent/test
Provider tfidf-lr
Method temperature
Labels/seed/wording 50/0/w0
ECE10 width [95% bootstrap] 0.0155 [0.0115, 0.0274]
Classwise ECE 0.0022
Brier sum 0.1768
Log loss 0.4504
Accuracy 0.8779
AUROC 0.9972
n 3080

Record 2: intent/test — tfidf-lr — isotonic — 50/0/w0

Field Value
Task/split intent/test
Provider tfidf-lr
Method isotonic
Labels/seed/wording 50/0/w0
ECE10 width [95% bootstrap] 0.1591 [0.1501, 0.1711]
Classwise ECE 0.0055
Brier sum 0.6515
Log loss 2.8632
Accuracy 0.4581
AUROC 0.8566
n 3080

Record 3: intent/test — tfidf-lr — temperature — 100/0/w0

Field Value
Task/split intent/test
Provider tfidf-lr
Method temperature
Labels/seed/wording 100/0/w0
ECE10 width [95% bootstrap] 0.0349 [0.0279, 0.0460]
Classwise ECE 0.0022
Brier sum 0.1790
Log loss 0.4722
Accuracy 0.8779
AUROC 0.9972
n 3080

Record 4: intent/test — tfidf-lr — isotonic — 100/0/w0

Field Value
Task/split intent/test
Provider tfidf-lr
Method isotonic
Labels/seed/wording 100/0/w0
ECE10 width [95% bootstrap] 0.0957 [0.0905, 0.1076]
Classwise ECE 0.0042
Brier sum 0.4564
Log loss 2.7424
Accuracy 0.6166
AUROC 0.9216
n 3080

Record 5: intent/test — fasttext-style — temperature — 50/0/w0

Field Value
Task/split intent/test
Provider fasttext-style
Method temperature
Labels/seed/wording 50/0/w0
ECE10 width [95% bootstrap] 0.0259 [0.0193, 0.0392]
Classwise ECE 0.0026
Brier sum 0.2254
Log loss 0.5707
Accuracy 0.8526
AUROC 0.9963
n 3080

Record 6: intent/test — fasttext-style — isotonic — 50/0/w0

Field Value
Task/split intent/test
Provider fasttext-style
Method isotonic
Labels/seed/wording 50/0/w0
ECE10 width [95% bootstrap] 0.1533 [0.1438, 0.1664]
Classwise ECE 0.0053
Brier sum 0.6682
Log loss 3.1343
Accuracy 0.4571
AUROC 0.8532
n 3080

Record 7: intent/test — fasttext-style — temperature — 100/0/w0

Field Value
Task/split intent/test
Provider fasttext-style
Method temperature
Labels/seed/wording 100/0/w0
ECE10 width [95% bootstrap] 0.0326 [0.0268, 0.0458]
Classwise ECE 0.0026
Brier sum 0.2243
Log loss 0.5904
Accuracy 0.8526
AUROC 0.9962
n 3080

Record 8: intent/test — fasttext-style — isotonic — 100/0/w0

Field Value
Task/split intent/test
Provider fasttext-style
Method isotonic
Labels/seed/wording 100/0/w0
ECE10 width [95% bootstrap] 0.0842 [0.0735, 0.0958]
Classwise ECE 0.0040
Brier sum 0.4888
Log loss 3.0462
Accuracy 0.6045
AUROC 0.9140
n 3080

Record 9: intent/test — embed-sim — temperature — 50/0/w0

Field Value
Task/split intent/test
Provider embed-sim
Method temperature
Labels/seed/wording 50/0/w0
ECE10 width [95% bootstrap] 0.0175 [0.0155, 0.0340]
Classwise ECE 0.0055
Brier sum 0.4358
Log loss 1.2116
Accuracy 0.6799
AUROC 0.9861
n 3080

Record 10: intent/test — embed-sim — isotonic — 50/0/w0

Field Value
Task/split intent/test
Provider embed-sim
Method isotonic
Labels/seed/wording 50/0/w0
ECE10 width [95% bootstrap] 0.0842 [0.0695, 0.0973]
Classwise ECE 0.0053
Brier sum 0.8055
Log loss 4.8522
Accuracy 0.3558
AUROC 0.7848
n 3080

Record 11: intent/test — embed-sim — temperature — 100/0/w0

Field Value
Task/split intent/test
Provider embed-sim
Method temperature
Labels/seed/wording 100/0/w0
ECE10 width [95% bootstrap] 0.0154 [0.0147, 0.0346]
Classwise ECE 0.0054
Brier sum 0.4356
Log loss 1.2117
Accuracy 0.6799
AUROC 0.9861
n 3080

Record 12: intent/test — embed-sim — isotonic — 100/0/w0

Field Value
Task/split intent/test
Provider embed-sim
Method isotonic
Labels/seed/wording 100/0/w0
ECE10 width [95% bootstrap] 0.0643 [0.0522, 0.0803]
Classwise ECE 0.0059
Brier sum 0.6993
Log loss 5.3657
Accuracy 0.4685
AUROC 0.8491
n 3080

Chapter 10, table 4

Return to Chapter 10.

Record 1: intent/test — tfidf-lr — 1000

Field Value
Task/split intent/test
Provider tfidf-lr
Labels 1000
Coverage [95% bootstrap] 0.9088 [0.8987, 0.9192]
Mean set size [95% bootstrap] 1.0955 [1.0844, 1.1062]
Empty share 0.0010
Singleton share 0.9081
n 3080

Record 2: intent/test — fasttext-style — 1000

Field Value
Task/split intent/test
Provider fasttext-style
Labels 1000
Coverage [95% bootstrap] 0.9084 [0.8987, 0.9185]
Mean set size [95% bootstrap] 1.2545 [1.2344, 1.2744]
Empty share 0.0000
Singleton share 0.7977
n 3080

Record 3: intent/test — embed-sim — 1000

Field Value
Task/split intent/test
Provider embed-sim
Labels 1000
Coverage [95% bootstrap] 0.9166 [0.9068, 0.9253]
Mean set size [95% bootstrap] 6.1019 [6.0156, 6.1848]
Empty share 0.0003
Singleton share 0.0133
n 3080

Record 4: safety/test — tfidf-lr — 90

Field Value
Task/split safety/test
Provider tfidf-lr
Labels 90
Coverage [95% bootstrap] 0.9310 [0.8793, 0.9741]
Mean set size [95% bootstrap] 1.1034 [1.0517, 1.1638]
Empty share 0.0000
Singleton share 0.8966
n 116

Record 5: safety/shift — tfidf-lr — 90

Field Value
Task/split safety/shift
Provider tfidf-lr
Labels 90
Coverage [95% bootstrap] 0.5687 [0.5076, 0.6221]
Mean set size [95% bootstrap] 1.0229 [1.0038, 1.0420]
Empty share 0.0000
Singleton share 0.9771
n 262

Record 6: safety/test — fasttext-style — 90

Field Value
Task/split safety/test
Provider fasttext-style
Labels 90
Coverage [95% bootstrap] 0.9569 [0.9224, 0.9914]
Mean set size [95% bootstrap] 1.1466 [1.0862, 1.2155]
Empty share 0.0000
Singleton share 0.8534
n 116

Record 7: safety/shift — fasttext-style — 90

Field Value
Task/split safety/shift
Provider fasttext-style
Labels 90
Coverage [95% bootstrap] 0.5649 [0.5038, 0.6183]
Mean set size [95% bootstrap] 1.0344 [1.0153, 1.0573]
Empty share 0.0000
Singleton share 0.9656
n 262

Record 8: safety/test — embed-sim — 90

Field Value
Task/split safety/test
Provider embed-sim
Labels 90
Coverage [95% bootstrap] 0.8793 [0.8190, 0.9397]
Mean set size [95% bootstrap] 1.6466 [1.5603, 1.7414]
Empty share 0.0000
Singleton share 0.3534
n 116

Record 9: safety/shift — embed-sim — 90

Field Value
Task/split safety/shift
Provider embed-sim
Labels 90
Coverage [95% bootstrap] 0.8550 [0.8092, 0.8969]
Mean set size [95% bootstrap] 1.8168 [1.7672, 1.8626]
Empty share 0.0000
Singleton share 0.1832
n 262