Using NLP to create preliminary causal system maps for use in policy analysis

The use of causal systems mapping in interdisciplinary and policy research has increased in recent years. Causal system maps typically rely on stakeholder opinion for their creation. This works well but does not make use of all available literature and can be time-consuming. For most topics, there is an abundance of text data in easily identifiable journal papers, grey literature, and policy documents. Using this data to support causal systems mapping exercises has the potential to make them more comprehensive and connected to evidence. There is also potential for the creation of maps using this data, to be done quickly, if the processes used become routine. In this paper, we develop an approach using Natural Language Processing (NLP) techniques and text data from journal papers to create preliminary causal system maps. Using the example topic of power sector decarbonisation policies and comparisons to a related participatory exercise, we consider the best techniques to use, the workflows which might speed up mapping exercises, and potential risks. The approach produces familiar factors and logical individual relationships, but causal maps with structure that mirrors attention in the literature rather than real causal patterns, and which overemphasise connections directly between policies and outcomes, rather than longer more realistic causal chains. We highlight the importance of choice of documents and sections of documents to use, and that the NLP workflow is full of subjective judgements and decisions. We argue that a clear purpose must be identified before beginning, to inform these choices; purely exploratory, which are relatively common with systems mapping exercises, are likely to be flawed.