Labs → Classification Metrics Lab

Classification Metrics Lab

When is precision low? Why does accuracy lie on rare classes? What does overfitting look like on data the model never saw — and what actually fixes each failure? Drag three knobs and find out: every metric, both curves, the bias–variance target and a live diagnosis with its remedy.

D-IV · Model EvaluationInteractive — no login needed~10 min

healthyP 88.2%R 94.7%Acc 94.6%Gap 0.0 ptsAUC-ROC 0.989PR-AUC 0.977

Baseline — darts tight around a centered ×. Break it yourself.

Cases per set 500small data → unstable model (adds variance, not bias)Model complexity balancedtoo simpletoo complexPositive share 30%rarebalancedDecision threshold 0.500 — all flagged positive1 — none

Healthy. Dots near their ★ corners, darts tight around a centered ×. Threshold is a business choice: which costs more, FP or FN?

1 · Training set (memorized)acc 94.6%

2 · Held-out test set (all metrics use this)acc 94.6%

actually positive · actually negative · red slash = misclassified

ROC · AUC 0.989

y = recall · x = false-positive rate · closer to ★ (top-left) = better

PR · AUC 0.977

y = precision · x = recall · closer to ★ (top-right) = better

Bias–variance target · bias low · variance low

× = cluster average. bias = where × sits · variance = spread around ×

How to use the AUC curves — three moves

1 · Compare models by the AUC number (always on held-out data): ~0.5 coin flip · 0.7–0.8 okay · 0.8–0.9 good · >0.9 excellent — or suspicious, check the Gap.
2 · Pick which AUC to read by class balance: balanced → AUC-ROC; rare positives → PR-AUC. If they disagree, believe PR-AUC.
3 · Set the threshold from the curve's shape: decide which mistake costs more (FP vs FN), find that trade-off point, use its threshold. The dot moves — the AUC never does: only a better model lifts the curve.

Confusion matrix as pictures — predicted on rows, actual on columns

TP caught 142

FP false alarm 19

FN missed 8

TN cleared 331

Metrics — formula · your numbers · what it asks

Precision = TP/(TP+FP) = 142/(142+19) = 88.2%

of everything the model FLAGGED, how much was real? (reads ACROSS the predicted+ row)

Recall = TP/(TP+FN) = 142/(142+8) = 94.7%

of everything actually POSITIVE, how much did we catch? (reads DOWN the actually+ column)

F1 = 2·P·R/(P+R) = 91.3%

one number combining both — high only when BOTH are decent

Accuracy = (TP+TN)/all = (142+331)/500 = 94.6%

share of ALL cases correct — flattering when one class dominates

Gap = acc(train) − acc(test) = 94.6% − 94.6% = 0.0 pts

how much better on memorized data than new — the overfitting detector

Reference — cause · consequence · remedy

You observe	Why	Consequence	Remedy
Precision low, recall fine	Loose threshold / FP flood	False alarms erode trust	Raise threshold; rebalance
Recall low, precision fine	Strict threshold / rare positives	Real cases slip through	Lower threshold; oversample, class weights
Both low, train ≈ test	Underfit — high bias	Bad everywhere; threshold useless	More features/capacity, boosting
Train great, test poor	Overfit — high variance	Accuracy drops on real data	Regularization, bagging, more data, early stop
Both high (bias + variance)	Wrong model AND too little data	Unreliable and wrong — worst case	More data first, then grow capacity
Accuracy high, P or R low	Imbalance — negatives dominate	Metric lies; model useless at its job	Judge by F1/PR-AUC; rebalance