← LLM Spatial Reasoning Under Prompt Changes
LLM spatial-reasoning results explorer
Results from the paper · recomputed in your browser
Twelve model configurations were asked the same question 2,016 times in different words: can this electric wheelchair get round this stair landing? Try the scene yourself, then see how the answers moved with the prompt.
The charts use the numbers in my course paper. Where the paper gives percentages, the page rebuilds the counts from the group sizes and recomputes the paper's statistics from them; section 4 shows the comparison. The answers themselves are not published here. The scene in section 1 is an illustration and does not decide whether the chair fits.
The landing
S0: the measured plan. Drag the chair, turn it, and watch the margins.
Drag the chair, or focus it and use the arrow keys (Shift for 5 cm) and [ ] to turn.
A chair that spins on the spot about its centre needs a circle as wide as its diagonal. Real electric chairs turn about their drive wheels, which needs more room, and they cannot hang over the stairs.
What each prompt gave the model
L1 · plain words
The landing is long but its usable area is not a rectangle; a wedge-shaped bottleneck on the right narrows it, with a local protrusion. No full set of dimensions.
L2 · dimensions
Adds the chair's length and width, the 220 × 140 cm plan, the two 110 cm stair openings, the bottleneck's width and length, the 15 cm protrusion, the 106 cm diagonal and the margins.
L3 · geometric model
Asks for the chair as a rigid rectangle and the free space as an irregular region, then checks of the bottleneck, the whole plan, room to stop and turn, the swept area and repeated back-and-forth moves.
Strong prompt adds
Do not compare only the chair's width with the bottleneck. Check every dimension listed. The chair moves only in straight lines and must stop to change direction. Say whether measuring on site or a simulation is needed.
Distractors (geometry unchanged)
Optimistic: the site looks roomy, the family think it will fit. Pessimistic: it looks very narrow, people worry it will get stuck. Irrelevant: handrail material, wall colour, floor glare.
Summaries of the prompt templates described in the paper, translated; every prompt asked for the same fields, ending in a final judgment of can pass, cannot pass or uncertain.
The experiment
- 7scenesS0 measured plan, S1–S6 one dimension changed
- 3prompt levelsL1 plain words, L2 dimensions, L3 geometric model
- 2prompt strengthsweak (task and answer format) or strong (must check bottleneck, sweep, stop-and-turn)
- 4distractorsnone, optimistic, pessimistic, irrelevant
- 12model configurationssix models, each in a normal and a thinking mode
- 2,016answersone call per combination, all usable
Two follow-up runs: 480 answers on a scene with two extra obstacles (S7, 20 repeats per condition) and 54 answers with images added (3 models × 6 inputs × 3 repeats).
What the models answered
Recomputed from the counts
The page rebuilds counts from the paper's percentages and recomputes its statistics in your browser. If a rebuilt count were wrong, these would not match.
| Quantity | Paper | Recomputed here |
|---|