Man or Machine: Evaluations of Human and Machine-Generated Movie Reviews

Recent advances in generative language models, such as ChatGPT have demonstrated an uncanny ability to produce texts that appear to be comparable to those produced by humans. Nevertheless, machine generated texts differ from those produced by humans in important aspects, such as routinely including references to nonexistent sources. In this paper, we use both psycholinguistic measurements and participant responses to compare texts generated by machine with equivalent texts generated by humans. Our analysis demonstrates some of the ways in which machine-generated texts differ from human-generated ones in both style (e.g., increased use of positive connectives) and content (e.g., increased confidence). We also note multiple ways in which texts generated by these models are similar to those generated by humans (e.g., their use of emotion words). We believe this research provides insights that can be useful to understanding how language is generated by both humans and machines.