LLM Spatial Reasoning Under Prompt Changes

A data-mining study of 2,016 answers from 12 LLM configurations to one borderline question: can an electric wheelchair get round a narrow stair landing? The question was asked at three levels of detail, with two prompt strengths and four kinds of distractor.

Period
2026
Context
Data Mining course project
Team
Individual project
My role
Author of the study and the paper
Status
Course paper completed; Data Mining course grade 98/100
  • LLM APIs (DeepSeek
  • Qwen
  • OpenAI)
  • Prompt design
  • Behavioural labelling
  • Chi-square and McNemar tests
  • Logistic regression

Demo & artifacts

The explorer charts the paper's results and recomputes its statistics in your browser. You can also move a wheelchair around the landing used in the prompts. The model answers themselves are not published, and no model is called.

Stacked bar chart of final judgments, cannot pass versus uncertain, for weak and strong prompts at levels L1, L2 and L3.
Diagram Final judgments by prompt level and strength, 336 answers per bar, from the results explorer. With dimensions (L2) answers settle on “cannot pass”; plain words (L1) and the strong prompt push them towards “uncertain”.

My contribution

  • Turned one real, borderline question (a 121 × 68.5 cm electric wheelchair on a 220 × 140 cm landing with a wedge-shaped obstacle) into a controlled experiment: 7 scenes × 3 prompt levels × 2 prompt strengths × 4 distractors × 12 model configurations, 2,016 answers in all.
  • Wrote prompts at three levels (plain words, full dimensions, an explicit geometric model) and a strong variant that names the constraints to check. Every prompt asks for the same structured fields, so each answer can be labelled.
  • Labelled every answer's final judgment and the factors it considered, measured flips caused by distractors against the no-distractor answer, and reviewed failure types by hand.
  • Tested the differences with chi-square tests, Wilson intervals, binary logistic regressions and paired McNemar tests for normal versus thinking mode.
  • Ran two follow-ups: a harder scene with two more obstacles (480 answers, 20 repeats per condition) and a small image-input run (54 answers).

The question

An electric wheelchair, 121 cm long and 68.5 cm wide, has to get round a stair landing 220 cm long and 140 cm deep, with a wedge-shaped obstacle in one corner. It is a real case and a borderline one: to spin on the spot about its centre the chair needs a circle as wide as its diagonal, 139 cm, on a landing 140 cm deep.

The study does not try to settle whether the chair fits. It asks something else: when the same landing is described in different words, do language models give the same answer?

Design

Each answer comes from one combination of five factors:

  • Scene: the measured plan (S0) or one of six variants with a single dimension changed.
  • Prompt level: L1 describes the landing in plain words, L2 gives every dimension, L3 asks for a geometric model of a rigid rectangle in an irregular region.
  • Prompt strength: the weak prompt gives the task and the answer format; the strong one lists the constraints to check, including that the chair moves in straight lines and must stop to turn.
  • Distractor: none, an optimistic remark, a pessimistic remark, or irrelevant detail such as wall colour. None of them changes a dimension.
  • Model configuration: six models, each in a normal and a thinking mode.

Every prompt asks for the same fields, ending in can pass, cannot pass or uncertain. A flip is a change from the model’s own no-distractor answer for the same scene, level and strength.

What came out

The results explorer has the full breakdown. In short, giving the dimensions made answers stable, while plain-language descriptions and a stricter prompt made models admit uncertainty more often. Remarks that changed nothing about the geometry still changed some answers, and some models were far more sensitive than others. Longer reasoning in thinking mode did not make answers more stable.

The follow-ups pointed the same way. On a scene with two extra obstacles, models listed the right factors almost every time but rarely turned them into a decision. Adding images did not help either: the only “can pass” answers appeared when a drawing of the turning area was included.

Results & validation

  • Prompt level mattered most (χ² = 287.2, Cramér's V = 0.38). With dimensions, 93.8 % of weak-prompt answers said “cannot pass”; in plain words, 44.3 % said “uncertain”.
  • The strong prompt made models more cautious, not more decisive: “uncertain” rose from 23.9 % to 46.8 %.
  • Distractors that change no dimension flipped 9.6 % of answers overall, and up to 31 % (13 of 42) for one configuration under a pessimistic description.
  • Thinking mode wrote about 1,570 characters of reasoning per answer but flipped more often (11.4 % against 7.7 %, McNemar p = 0.003).
  • In the harder scene almost every answer named the obstacles, the bottleneck and the swept area, yet most still ended “uncertain”.
  • For the explorer I rebuilt the counts behind the paper's percentages; they reproduce its chi-square values, Wilson intervals and per-model flip rates.

Numbers come from the course paper. The counts in the explorer were rebuilt from its percentages and figures for this website in November 2026 and checked against its reported statistics (section 4 of the explorer). The answer data set was not re-analysed for this page.

Limitations & next steps

  • One call per condition in the main run, and sampling settings (temperature, top-p, system prompt) were not recorded. Without repeated no-distractor answers, part of each flip rate may be ordinary run-to-run variation; a repeated baseline would separate the two.
  • No ground truth by design: the study measures how answers change, not whether they are right. No answer in the main run said the chair can pass, so a three-way model of the judgments could not be fitted.
  • The “error rate” (64.2 %) counts every label except “no obvious error”. That includes cross-level disagreement, which describes a set of answers rather than one wrong answer, and in the multi-label coding every “uncertain” answer, so it is not the share of wrong answers.
  • Labels were reviewed by one person; there is no second annotator or agreement statistic.
  • One scene family and one task. Results may not carry over to other spatial problems, and model versions change over time.