AI orchestration for geospatial data

Large language models (LLMs), the engine of generative AI tools such as ChatGPT, are undoubtedly exceptionally good at generating seemingly-intelligent human language. But, unlike an intelligent human, LLMs have no capacity for manipulating and reasoning about the concepts that lie behind such language. For all their remarkable outputs, LLMs remain fundamentally “synthetic text extruding machines”, as Emily M. Bender puts it.

That means LLMs are, on their own, of limited use when working with spatial language and spatial data. Spatial concepts are central to making sense of our geographic world. So it is not hard to generate perfectly formed but entirely false text output when asking LLMs to reason about about geographic space, as illustrated below.

LLMs such as ChatGPT have no capacity to reason about spatial concepts, and are easily tripped up by spatial language as a result. In the example above, a human might reason that A is either north or south of C (if they interpret north and south as precise bearings); or more often that they know nothing about the direction of A from C (adopting a more usual, looser interpretation of direction).

These limitations were underlined for my team recently, when we put ChatGPT to the test with two spatial language benchmark data sets, bAbI¹ and SpartQA². The results illustrated that ChatGPT often performs no better than flipping a coin for even basic spatial reasoning tasks (summarised below, to appear later this year³). Other research teams have noted similar results⁴.

ChatGPT performs no better than flipping a coin when presented with benchmark spatial reasoning tasks, which in these cases required a Y/N answer (approximately 50% accuracy, top two bars) or one of four direction relations (approximately 25% accuracy, bottom bar), see footnote 3

Knowledge-based GeoAI

Happily, we already have a rich variety of knowledge-based geospatial AI techniques that have the potential to combine with LLMs to create more powerful and more reliable “GeoAI” tools.

Knowledge-based AI techniques have a very long history in geospatial sciences, dating back to the genesis of the field⁵. Where machine-learning techniques, including LLMs, aim to identify patterns from data, knowledge-based techniques rely on automated reasoning with symbolic representations of data. Knowledge-based techniques had fallen out of favour in recent years, as the power and popularity of machine learning, and especially deep learning, has skyrocketed.

However, the remarkable new capabilities of LLMs for language processing are also highlighting the limitations of these techniques for even simple spatial reasoning, analysis, and logic. As our new generative AI tools increasingly lead us to “bump up” against those limitations, so there is an increasing need for combining generative AI and knowledge-based techniques.

Enter AI orchestration

AI “orchestration” is emerging as a foundational approach to combining the exceptional text-extruding capabilities of LLMs with reliable knowledge-based spatial reasoning and analysis.

Orchestration combines multiple AI and analytical tools into workflows. In orchestrated workflows, LLMs are used to help parse and make sense of human natural language inputs and generate readable outputs. In contrast, knowledge-based reasoners and authoritative data repositories are used to generate reliable and trustworthy responses to questions.

Orchestration is already at the core of some of the most advanced AI tools, such as Microsoft’s Copilot. It also the engine behind emerging GeoAI technologies, such as ESRI’s forthcoming generative AI assistants for ArcGIS. Despite these advances, today’s tools do not yet orchestrate seamlessly with the full range of trustworthy spatial reasoning and analytics capabilities. But that vision is getting closer. Using some of the excellent open-source orchestration tools and APIs, such as LangChain, it is already possible to start building simple but effective GeoAI orchestration in a matter of hours (see the short explainer video below).

Where next?

The potential for using orchestration to generate authoritative answers to text-based human spatial questions, such as “Which primary schools are in my suburb?“ or “What’s the total area of retail zoning in the CBD?”, is huge. Today, the technological barriers to retrieving geospatial data from authoritative databases mean that decision-makers across industry and government often have to rely on their GIS team or a human data analyst to generate trustworthy answers to even basic spatial questions. AI orchestration is bringing authoritative, automated spatial queries a step closer for everyone, not only the technologically savvy. These advances could not only release the technical experts to focus more on sophisticated analytics, but also enable rapid and easy access to the authoritative spatial data needed to support evidence-based decisions for anyone in government, industry, and the wider public.

1. Shi, Z., Zhang, Q, & Lipani, A. (2022). Stepgame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts. In Proc. AAAI Conference on Artificial Intelligence, 36(10):11321–29. https://ojs.aaai.org/index.php/AAAI/article/download/21383/21132

2. Mirzaee, R., Faghihi, H. R., Ning, Q., & Kordjmashidi, P. (2021). SpartQA: A textual question answering benchmark for spatial reasoning. arXiv preprint arXiv:2104.05832

3. Beydokhti, M.K., Tao, Y., Duckham, M., & Griffin, A. (2024, in press) Integrating Large Language Models and Qualitative Spatial Reasoning. In Hassan A. Karimi, ed., Big Data Techniques and Technologies in Geoinformatics, CRC Press: Boca Raton, FL.

4. Li, F., Hogg, D.C. and Cohn, A.G. (2024). Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark. In Proc. AAAI Conference on Artificial Intelligence, 38(17): 18500–18507. https://ojs.aaai.org/index.php/AAAI/article/view/29811/31406

5. For example, back in 1987 issue 2 of the flagship academic geospatial sciences journal, the International Journal of Geographic Information Systems, contained an influential research paper about “A knowledge-based geographical information system”.