What mattered more for mapping buildings: cleaner labels or a newer AI model? Poland's benchmark says labels
Across 15 architectures and 14,378 Polish image tiles, fine-tuning with corrected labels improved overlap far more than hyperparameter optimisation. A single annotator and only five independently surveyed buildings limit the claim.
The short version
- What happened
- The benchmark retained 14,378 tiles—11,878 with buildings and 2,500 without—covering 1,896 municipalities and all 16 Polish voivodeships, with cadastral districts separated across splits.
- Who it affects
- Public bodies and companies should audit omitted buildings and boundary errors before using a benchmark to buy or deploy a mapping system.
- What remains unproven
- Manual fine-tuning and test-set denominators are not stated in the main methods or result tables.
Keep in mind
Manual fine-tuning and test-set denominators are not stated in the main methods or result tables.
One GIS specialist performed manual verification, so inter-rater reliability was not measured.
Each variant was trained once with a fixed seed; close rankings lack run-to-run uncertainty.
The independently surveyed geometric check covered five buildings.
Results concern Polish RGB orthophotos and ground-floor footprints and may not transfer elsewhere.
Source analysis · 1 linked source. Primary analysed source date: 7 November 2026.
Read the full report · 7 min The evidence, methods, implications and limitations
The answer: better labels mattered much more than more tuning in this benchmark
A Poland-wide comparison of 15 building-segmentation systems found that reference-mask quality could change both measured accuracy and which architecture appeared best. Fine-tuning each model on manually corrected labels increased intersection-over-union on a held-out manual test set by an average 0.128, with a standard deviation of 0.036. Hyperparameter optimisation added only 0.008 on average, with a standard deviation of 0.010.
Intersection-over-union, or IoU, measures overlap between predicted and reference building pixels. The label-related gain was more than sixteen times the average gain attributed to further tuning. It does not mean cleaning always beats model development by that ratio. It means that for these images, labels, architectures and protocol, reducing omissions and boundary errors had the larger measured effect.
The finding matters beyond remote sensing. Teams compare systems to benchmarks as if the reference were unquestionably correct. Here, changing the reference changed model rankings. A procurement or research decision based only on the original register-derived masks could reward agreement with defects rather than better mapping.[1]
A national dataset was filtered and split to reduce geographic leakage
The researchers began with about 30,000 tiles generated from Poland's Land and Buildings Register and national orthophoto service. Each tile measured 1,024 by 1,024 pixels. Manual review removed records with missing buildings, inaccurate outlines, damaged images and other problems. The final corpus contained 14,378 samples: 11,878 positive tiles and 2,500 tiles checked to contain no buildings.
Coverage spans all 16 voivodeships, 1,896 municipalities and 2,269 cadastral districts, including dense urban and dispersed rural development. All samples from a cadastral district were assigned to only one of training, validation or test. That matters because nearby tiles can share landscape, imaging and administrative characteristics; mixing them across partitions would make generalisation look easier.
Labels retained the register's ground-floor outline rather than tracing visible roof edges. One experienced GIS professional reviewed the data and excluded ambiguous cases. The paper also created a smaller manual fine-tuning set and a disjoint manual test set, but does not state those sets' sample counts in its main methods or results tables. That omission makes it harder to judge how much hand-labelled evidence produced the gain.[1]
The distinction that matters
How the reference label changes what counts as an error
The same prediction can look worse or better depending on omissions and boundary mistakes in the benchmark mask.
Register mask
Score against filtered administrative outlines
May reward remaining defects
Manual label
Fine-tune and test against corrected outlines
Mean IoU gain 0.128
Hyperparameters
Optimise the training configuration
Mean IoU gain 0.008
Survey check
Compare five geodetic outlines
Illustrative, not national validation
Fifteen architectures were tested—and the test reference changed their story
The benchmark included U-Net, FPN, LinkNet, D-LinkNet, attention and multi-scale designs, transformer-based SegFormer and UPerNet variants, and YOLO instance-segmentation models. Models used common pretraining sources and trained for up to 100 epochs with early stopping. Semantic systems used a combined binary cross-entropy and Dice loss.
Each architecture had a baseline trained on register-derived data, a version fine-tuned on manual labels, and a fine-tuned version with optimised hyperparameters. Optuna used 2,000 training and 500 validation samples from the main dataset. YOLO used its own optimisation route, so training was aligned but not identical across every architecture.
Against the original register-labelled test set, fine-tuned systems often scored worse because predictions moved away from remaining reference defects. Against the manual test set, all 15 improved. SegFormer-B3 reached the highest reported IoU, 0.874, but several systems clustered closely. Every variant ran only once with a fixed seed, so small ranking differences are not stable winners with measured run-to-run uncertainty.[1]
A flawed test set can count a correct building as an error
If a reference mask omits a real building, a correct prediction counts as a false positive. If a boundary is displaced, a better geometric outline can receive a worse overlap score. The benchmark makes this visible: adapting to corrected labels lost points against the old masks while gaining substantially against the manual reference.
Ranking also depended on the metric. Transformer and convolutional systems traded places across references, while good overlap did not guarantee good local boundaries. YOLO variants achieved competitive IoU but much larger Hausdorff distances—about 111 to 119 pixels—than similarly scoring semantic models. That distance captures local deviations an average overlap score can hide.
For mapping agencies, insurers, emergency planners or utilities, an omitted outbuilding, a shifted corner and a slightly rough boundary have different consequences. Evaluation should match the intended use: presence, area, corner accuracy, topology, update detection or legal cadastral geometry. One leaderboard metric cannot represent all of them.[1]
The independent check was small, and consequential uses still need people
Manual image annotation is not absolute truth when a roof overhang hides the ground-floor footprint. The researchers therefore compared one fine-tuned model with geodetic surveys for five buildings. IoU ranged from 0.727 to 0.839, with a mean of 0.796 and a reported 95% confidence interval from 0.742 to 0.849. Area differences ranged from an under-estimate of 7.82% to over-estimates above 23%.
Five buildings cannot establish national geometric accuracy, and the authors present them only as illustrative. One annotator improves internal consistency but prevents measurement of inter-rater reliability. Better maps can help emergency response and planning, but automated boundaries affecting tax, rights, eligibility or safety need human review and a correction route.
The Military University of Technology funded the work; the authors declared no competing interests. Code is on GitHub and Zenodo, while datasets are available on request. The evidence would strengthen with exact manual-set denominators, multiple annotators, repeated seeds, a larger surveyed test and external replication. The central lesson—test the test set—is credible and current, but not urgent: Latest, not Breaking.[1]
Keep following the evidence
Join The Impact Brief founding list for selected AI reports, their effect on people and the evidence limits. Read a sample first.
Continue the story
Read next
Continue exploring computer vision: compare the finding and its evidence limits.
Can a lightweight AI read Bangladesh’s road signs?
In a 720-image test, a 4.6-million-parameter classifier reached 99.34% accuracy on a new 48-class Bangladeshi dataset. It classified prepared sign images; it did not detect signs in moving traffic or prove safe vehicle use.
Read the next reportExplore the full source and evidence record
What this means for people
- Public bodies and companies should audit omitted buildings and boundary errors before using a benchmark to buy or deploy a mapping system.
- People can be affected when automated footprints inform planning, emergency response, exposure, tax or eligibility decisions, so consequential cases need review and appeal.
- Technical teams may gain more from representative annotation and independent tests than from repeatedly tuning similar architectures.
Global context
National mapping systems differ in image resolution, update cycles, cadastral definitions and public-data quality. The study's Polish coverage strengthens domestic relevance, while transfer elsewhere requires local labels and surveyed checks. The general lesson—test the test set—is globally applicable even when the ranking is not.
What the evidence does not yet show
- Manual fine-tuning and test-set denominators are not stated in the main methods or result tables.
- One GIS specialist performed manual verification, so inter-rater reliability was not measured.
- Each variant was trained once with a fixed seed; close rankings lack run-to-run uncertainty.
- The independently surveyed geometric check covered five buildings.
- Results concern Polish RGB orthophotos and ground-floor footprints and may not transfer elsewhere.
What to watch next
- Publication of exact manual fine-tuning and test denominators
- Replication with multiple annotators and inter-rater agreement
- Repeated runs and uncertainty for close rankings
- Larger independent geodetic validation across shapes and regions
- Operational studies measuring human correction and appeal in consequential mapping
Living evidence record
Impact record IAI-1OAZG6B
Evidence stage
Studied
Confidence
Supported
Reporting basis
Source analysis
Independent or research support
Present
Record status
Monitoring
Last checked
8 November 2026
Source trail
1 direct source across 1 source type.
People impact
Documented in this record.
Uncertainty
Limits and next checks are explicit.
Stages describe the evidence available—not whether a technology is good or bad. See the public method.
Single-source reporting disclosure
This record analyses one direct source. It can establish what Scientific Reports published or reported, but it is not independent corroboration of every performance claim or predicted outcome. The confidence label will change only when broader evidence is added.
Evidence trail
Sources used for this report
Links checked 8 November 2026
Research question, topics and regions
How reference-label quality changes measured performance and model ranking for 15 neural-network architectures segmenting buildings on Polish orthophotos
Topics: Computer vision · Geospatial AI · Training data · Benchmarking · Public-sector mapping
Regions: Poland · Europe · Global
This report is labelled source analysis. We summarise and analyse source material in our own words; company statements remain attributed claims until independently supported. Translated summaries preserve the meaning of the original source and link back to it. Read our editorial standards.
Reader commentary
What does this mean where you are?
Tell us what you have seen at work, in your community or in everyday life. A question is welcome too. No account is needed; we review notes before publishing them.
Explore commentary across the portal →Published reader notes
0No published reader notes yet. You can start the evidence-led discussion above.
Prefer a private correction or response? Contact the newsroom.