The Challenging Adaptation of Research Methods from Human Subjects to AI Systems

Today’s Artificial Intelligence (AI) systems stem from recent advancements in Natural Language Processing (NLP) capabilities and are enabling applications across a wide range of disciplines. While social sciences have a long history of adopting this and similar emerging technologies, the possibility of interrogating the systems themselves by exploiting their conversational or instruction-following capabilities is a relatively new possibility. To some extent, many research methods typically involving human subjects can now be considered for many classes of AI systems, with varying degrees of complexity. This approach can be useful in detecting the representation, be it manifest or latent, that a system or its underlying model has of a given social phenomenon. Some theoretical frameworks can be mobilized in support of a similar application, while there is limited guidance for it on an operational level.

It is argued that, when adapting human methods to AI systems,, a complete operative equivalence with human subjects is never possible, and properties of cultural products can also be lacking or inadequate. A hybrid category is therefore needed to account for many of the peculiar properties of AI systems. These properties can be system-agnostic or present with varying degrees in different systems and determine the range of considerable methods, and while many of them present practical limitations to a given administration, others can render it easier, scalable and reproducible. This arrangement of properties also offers opportunities for research methods that are considered obsolete or impractical for human administration, and for the adoption of dedicated computational methods. Some examples concerning both mainstream and exotic methods are presented, and key methodological takeaways are outlined along with suggestions for improvements and further research.