← Case studiesBenchmark · real track photographs

How accurate is Ralliyo's AI on labelled defects?

We run the vision models Ralliyo can use on photographs that people have already labelled, and publish how often each one agrees, misses and raises false alarms. No hand-picked successes.

Models compared on the same sample of photographs, at confidence 0.5

Model	Images	Precision	Recall	F1	Found	Missed	False alarms	Tokens per image	Run time
gpt-4.1-mini railsight-vision	150	22%	30%	25%	24	56	87	1,290	6 min
gpt-5-mini railsight-vision-5mini	148 (2 failed)	15%	66%	24%	52	27	299	2,473	32 min

›

gpt-4.1-mini

F1 25% · precision 22% · recall 30%

Precision22%of AI findings were labelled defects

Recall30%of labelled defects were found

F125%balance of the two

Images150none failed

Modelgpt-4.1-minirules rs-track-1 · run 2026-10-31

Threshold0.5confidence

By observation type, at confidence 0.5

Type	Labelled images	Found	Missed	False alarms	Precision	Recall	Box overlap ≥0.3
Fastener deterioration	0	0	0	41	0%	—	—
Missing fastener	39	9	30	11	45%	23%	0%
Potential sleeper cracking	4	0	4	19	0%	0%	—
Rail surface anomaly	37	15	22	16	48%	41%	7%

Effect of the confidence threshold

Threshold	Precision	Recall	F1
0.3	22%	30%	25%
0.5	22%	30%	25%
0.7	22%	30%	25%

Other findings not scored: Foreign object (5), Visible corrosion (67), Ballast contamination (20), Vegetation encroachment (3), Unusual trackside condition (1).

Examples, including mistakes

Green boxes are the dataset's labels; dashed yellow boxes are the AI's findings.

Benchmark example 192_0_png.rf.8a35880a295ddac679a0900f59d51c67.jpg — CorrectLabelled: Rail surface anomalyAI found: Rail surface anomaly (90%), Visible corrosion (95%)

Benchmark example 180_4_png.rf.bb0d1f4710b49a7446610a49dcbd07c5.jpg — MissedLabelled: Missing fastenerAI found: Fastener deterioration (85%)

Benchmark example 189_1_png.rf.9fbb09debac0d1ee5e0e88b490927d64.jpg — False alarmLabelled: no defectAI found: Fastener deterioration (90%), Visible corrosion (95%)

Benchmark example 192_3_png.rf.605c748b988794e2b0028a52ecee222e.jpg — Correctly clearLabelled: no defectAI found: Visible corrosion (90%)

Benchmark example 192_2_png.rf.4d90e7d7d997a98a3eb41c143dd29a57.jpg — CorrectLabelled: Rail surface anomalyAI found: Rail surface anomaly (90%), Visible corrosion (100%)

Benchmark example 181_1_png.rf.887f20ec2a7f9e04d1c992a05d5d6ee1.jpg — MissedLabelled: Rail surface anomalyAI found: Foreign object (90%)

Benchmark example 190_4_png.rf.ea55d96ead0a72a1c3d593bf67b1461d.jpg — False alarmLabelled: no defectAI found: Fastener deterioration (80%), Visible corrosion (90%)

Benchmark example 198_0_png.rf.f7f5e8a74b3aef2f5ef39c346bd99a49.jpg — Correctly clearLabelled: no defectAI found: Visible corrosion (90%)

Benchmark example 192_4_png.rf.7d39ca0db55492b52363f9212f55710b.jpg — CorrectLabelled: Rail surface anomalyAI found: Rail surface anomaly (90%), Visible corrosion (85%)

Benchmark example 181_3_png.rf.913072e2a18c04a38a3e92f1afeef763.jpg — MissedLabelled: Rail surface anomalyAI found: Fastener deterioration (85%), Visible corrosion (90%)

Benchmark example 191_0_png.rf.bd39591501df55ac97f7bd2c2b18c51c.jpg — False alarmLabelled: no defectAI found: Fastener deterioration (85%)

Benchmark example 201_3_png.rf.46cb4b54e8f04106c230dcb59ef15f38.jpg — Correctly clearLabelled: no defectAI found: Visible corrosion (85%)

›

gpt-5-mini

F1 24% · precision 15% · recall 66%

Precision15%of AI findings were labelled defects

Recall66%of labelled defects were found

F124%balance of the two

Images1482 failed

Modelgpt-5-minirules rs-track-1 · run 2026-11-01

Threshold0.5confidence

By observation type, at confidence 0.5

Type	Labelled images	Found	Missed	False alarms	Precision	Recall	Box overlap ≥0.3
Fastener deterioration	0	0	0	128	0%	—	—
Missing fastener	39	18	21	10	64%	46%	33%
Potential sleeper cracking	4	4	0	75	5%	100%	0%
Rail surface anomaly	36	30	6	86	26%	83%	10%

Effect of the confidence threshold

Threshold	Precision	Recall	F1
0.3	15%	67%	25%
0.5	15%	66%	24%
0.7	16%	60%	25%

Other findings not scored: Visible corrosion (147), Vegetation encroachment (22), Foreign object (53), Ballast contamination (23), Unusual trackside condition (5).

Examples, including mistakes

Green boxes are the dataset's labels; dashed yellow boxes are the AI's findings.

Benchmark example 34_2_png.rf.158d0ed92bc69c0ead88638d7899a669.jpg — CorrectLabelled: Rail surface anomalyAI found: Rail surface anomaly (95%), Visible corrosion (75%), Foreign object (60%)

Benchmark example 44_1_png.rf.390016eb1ee036df6048d6c12dbff45b.jpg — Correctly clearLabelled: no defectAI found: Foreign object (90%), Visible corrosion (75%)

Benchmark example 34_4_png.rf.61ae2f713c35a5167bfc9b37759e7ae0.jpg — CorrectLabelled: Rail surface anomalyAI found: Rail surface anomaly (95%), Visible corrosion (60%)

Benchmark example 183_2_png.rf.2d9cda155b1df9e631b1b924c7f1b03c.jpg — MissedLabelled: Missing fastenerAI found: Rail surface anomaly (90%), Fastener deterioration (85%), Visible corrosion (95%), Ballast contamination (70%), Vegetation encroachment (50%)

Benchmark example 183_3_png.rf.e6c318615d6033c4aab122892995a200.jpg — MissedLabelled: Missing fastenerAI found: Rail surface anomaly (95%), Visible corrosion (90%), Fastener deterioration (85%)

Benchmark example 181_4_png.rf.1dacc1cd36ba7bae41a83f7ff0acc44f.jpg — False alarmLabelled: Rail surface anomalyAI found: Fastener deterioration (70%), Potential sleeper cracking (60%), Rail surface anomaly (70%), Visible corrosion (80%), Vegetation encroachment (85%)

Benchmark example 184_2_png.rf.05555a294fadd6877a65c425a1fc5112.jpg — MissedLabelled: Missing fastenerAI found: Visible corrosion (90%), Fastener deterioration (70%), Fastener deterioration (85%), Ballast contamination (75%)

Benchmark example 182_1_png.rf.0c6c524a911c854c0116c5340d8e4e56.jpg — False alarmLabelled: Missing fastenerAI found: Missing fastener (90%), Visible corrosion (95%), Fastener deterioration (60%), Foreign object (90%)

Benchmark example 185_2_png.rf.f31d074ad626dd2e70471ffb415db619.jpg — MissedLabelled: Missing fastenerAI found: Visible corrosion (90%), Fastener deterioration (90%), Fastener deterioration (90%), Potential sleeper cracking (90%), Rail surface anomaly (60%), Foreign object (60%)

Method

A fixed, stratified subset of the dataset's held-out split: every labelled defect type is represented, plus clear images so false alarms show up. Every model sees the same photographs.
Scored per image: a type counts as found when the AI reports it at or above the confidence threshold.
Only types the dataset labels are scored. Anything else the AI reports is listed separately, not counted as wrong.
Box overlap (IoU) is reported separately because the labels and the AI draw regions differently.

Dataset

ZITMS (Roboflow Universe) · test split (300 images) · licence CC BY 4.0.

Dataset by nam on Roboflow Universe, used under CC BY 4.0.

Dataset source