Measuring the Quality of Answers in Political Q&As with Large Language Models

This paper proposes a novel methodology for assessing the quality of answers in political question-and-answer sessions. Our approach consists of measuring the quality of an answer based on how accurately it can be identified among all observed answers given the question. This reflects the relevance and depth of engagement of the answer to the question. Similarly to semantic search, this measurement approach can be implemented by training a language model on the corpus of observed questions and answers without additional labeled data. We showcase and validate our methodology using data from the Question Period in the Canadian House of Commons. Our analysis reveals that while some answers have a weak semantic connection with questions, hinting at some evasion or obfuscation, answers are generally relevant, far surpassing what would be expected from random replies. Besides, our findings provide valuable insights into the correlates of answer quality. We find significant variations based on the party affiliation of the members of Parliament posing the questions. Finally, we uncover a meaningful correlation between the quality of answers and the topic of the questions.