Large Language Models As a Tool for Argumentation Analysis: A Field Report from the Study of Parliamentary Debates on Abortion

Research on automated analysis of argumentation analysis is expanding rapidly. Current methods, however, depend on identifying argument indicators, such as conjunctions like "so," "then," and "because," which are often missing in real-world communication. As a result, researchers studying argumentation in large corpora face a choice between labour-intensive manual qualitative analysis or (semi)automated methods that can categorise general discourse topics but fail to detect implicit arguments. In this context, large language models (LLMs) offer a promising solution, as they can identify and categorise arguments by analysing natural language on an unprecedented scale.

I will evaluate this approach by comparing the results of argument identification and classification obtained from LLMs with findings from my research on the same corpora, which used both manual methods (qualitative rhetorical analysis) and automated techniques (topic modelling, semantic network analysis, and corpus linguistics methods such as collocate and keyword analysis). These analyses will focus on recent parliamentary debates about abortion law in Poland and Ireland. Parliamentary debates provide an ideal foundation for this study, as they are easily accessible through official transcripts and rich in pragmatically evident yet lexically implicit argumentation. For instance, when a speaker cites a court ruling aligned with their view, it is clear to listeners that the ruling is used as a premise in the speaker's argumentation, even without explicit argumentation markers, and that this line of argumentation might be classified as legal. Poland and Ireland are compelling cases for comparison, as both operate within the same global and regional human rights systems but have recently taken opposing paths in abortion legislation.

The proposed analysis will evaluate the strengths and weaknesses of using LLMs to analyse real-world argumentation and provide initial guidelines for their application. Additionally, it will explore the use of LLMs in automated text analysis as a means of methodological triangulation.