When evaluating AI defect detection systems, you'll encounter various accuracy claims. But what do these numbers actually mean? This guide explains the key metrics and how to interpret them.
Why Accuracy Metrics Are Complicated
Unlike simple classification tasks, defect detection involves finding and localizing objects in images. This creates multiple dimensions of accuracy:
- Detection accuracy: Did we find all the defects?
- Localization accuracy: Did we pinpoint them correctly?
- Classification accuracy: Did we identify the defect type correctly?
- Severity accuracy: Did we assess the severity correctly?
A single "95% accuracy" claim doesn't capture this complexity.
Key Metrics Explained
Mean Average Precision (mAP)
What it measures: Overall detection and localization accuracy across all defect classes.
How it works:
- For each defect type, calculates precision at different recall thresholds
- Averages these precision values to get Average Precision (AP)
- Averages AP across all defect types to get mAP
The IoU threshold matters:
- mAP@0.5: Predicted box must overlap 50% with ground truth
- mAP@0.75: Requires 75% overlap (stricter)
- mAP@0.5:0.95: Averages across multiple thresholds
MuVeraAI DefectVision Performance:
- mAP@0.5: 95.2%
- mAP@0.75: 87.3%
- mAP@0.5:0.95: 78.6%
Precision and Recall
Precision: Of all defects the AI flagged, what percentage were real?
- High precision = fewer false alarms
Recall: Of all actual defects, what percentage did the AI find?
- High recall = fewer missed defects
The tradeoff: Increasing one often decreases the other. The right balance depends on the use case:
| Application | Priority | Why | |-------------|----------|-----| | Safety-critical | High recall | Missing a defect is dangerous | | High-volume screening | High precision | Can't review many false positives | | Balanced | F1 score | Equal importance |
Confidence Thresholds
AI systems output a confidence score (0-100%) for each detection. The threshold determines which detections are shown.
High threshold (e.g., 90%):
- Fewer detections shown
- Higher precision, lower recall
- Good for: automated reports
Low threshold (e.g., 50%):
- More detections shown
- Lower precision, higher recall
- Good for: assisted review
At MuVeraAI, we default to 70% for production use, with lower-confidence detections flagged for human review.
Evaluating Vendor Claims
When vendors claim accuracy numbers, ask these questions:
1. "What's the test dataset?"
Be skeptical if:
- Dataset isn't described
- It's the same data used for training
- It's laboratory images (not real-world)
Better practice: Performance on hold-out test sets with real field images.
2. "What's the IoU threshold?"
mAP@0.5 is industry standard, but some vendors use loose definitions of "correct" detection.
3. "Which defect types?"
Aggregate accuracy can hide poor performance on specific defect types. Ask for breakdown:
Per-Defect-Type Performance:
- Surface corrosion: 96.1%
- Concrete cracking: 94.8%
- Coating failure: 93.2%
- Pitting: 89.7%
- Delamination: 87.3%
4. "What conditions?"
Accuracy varies with:
- Image quality (resolution, lighting)
- Defect severity (small defects are harder)
- Surface type
- Environmental conditions
Honest vendors acknowledge these variations.
5. "How is it validated?"
Best practices include:
- Independent test set (never seen during training)
- Regular revalidation (quarterly at minimum)
- Real-world field images, not lab samples
- Customer-provided validation sets
Our Approach at MuVeraAI
We report accuracy transparently with full methodology:
Validation methodology:
- Hold-out test set of 10,000+ images
- Labeled by certified inspectors (2x review)
- Updated quarterly with new field data
- Stratified by defect type, severity, conditions
Published metrics:
- Per-defect-type accuracy
- Performance by image quality tier
- Confidence calibration curves
- False positive/negative analysis
Continuous monitoring:
- Daily model performance tracking
- Drift detection alerts
- Customer feedback loop
The Honest Truth About AI Accuracy
No AI system is perfect. Here's what realistic expectations look like:
What AI does well:
- Finding obvious, common defects
- Processing high volume consistently
- Never getting tired or distracted
- Providing a thorough first pass
Where AI struggles:
- Novel defect types not in training data
- Very small or subtle defects
- Poor image quality
- Complex 3D assessment from 2D images
The right approach: AI handles the volume and obvious cases. Humans focus on edge cases, severity assessment, and professional judgment.
Want to understand how DefectVision would perform on your specific inspection types? Request a demo to test with your own images.
