Is the model any good? The confusion matrix
For classification, “the model is 95% right” isn't enough. We need to know where it goes wrong. Nakhla has a model that gives every payment a fraud score from 0 to 100; above the threshold, it goes to review. The results are summed up in a confusion matrix:
| TP | True positive: fraud, and the model caught it. ✓ |
| FP | False positive: a clean payment the model flagged (a false alarm that wastes a reviewer's time). |
| FN | False negative: fraud the model missed. (Usually the costliest) |
| TN | True negative: a clean payment that went through. ✓ |
Accuracy = (TP + TN) ÷ all Precision = TP ÷ (TP + FP) Recall = TP ÷ (TP + FN)
Precision: of everything we flagged, how much was really fraud? Recall: of all the fraud out there, how much did we catch? Move the threshold and watch them trade off:
Actually fraudActually clean
| Actually fraud | Actually clean | |
|---|---|---|
| Model: suspicious | 3TP | 2FP |
| Model: clean | 2FN | 13TN |
Accuracy80%
Precision60%
Recall60%
A balance of the two. The right choice depends on what each kind of error costs.
Watch outThe imbalanced-data trap: if only 1% of payments are fraud, a “dumb” model that always says “clean” is 99% accurate! Yet it catches no fraud at all (recall = 0). That's why for fraud, disease or failures we look at precision and recall, not accuracy alone.
PYTHON
tp, fp, fn, tn = 8, 2, 4, 86 accuracy = (tp + tn) / (tp + fp + fn + tn) precision = tp / (tp + fp) recall = tp / (tp + fn) print("Accuracy: ", round(accuracy * 100, 1), "%") print("Precision:", round(precision * 100, 1), "%") print("Recall: ", round(recall * 100, 1), "%")