Hello! You must login or signup first!

Monitoring AI-Modified Content at Scale: A Case Study on the Impact of ChatGPT on AI Conference Peer Reviews Weixin Liang 1* Zachary Izzo 2* Yaohui Zhang 3* Haley Lepp 4 Hancheng Cao 15 Xuandong Zhao6 Lingjiao Chen Haotian Ye¹ Sheng Liu Zhi Huang 7 Daniel A. McFarland 489 James Y. Zou 137 Abstract We present an approach for estimating the frac- tion of text in a large corpus which is likely to be substantially modified or produced by a large language model (LLM). Our maximum likelihood model leverages expert-written and AI-generated reference texts to accurately and efficiently exam- ine real-world LLM-use at the corpus level. We apply this approach to a case study of scientific peer review in AI conferences that took place af- ter the release of ChatGPT: ICLR 2024, NeurIPS 2023, CoRL 2023 and EMNLP 2023. Our results suggest that between 6.5% and 16.9% of text sub- mitted as peer reviews to these conferences could have been substantially modified by LLMs, i.e. beyond spell-checking or minor writing updates. The circumstances in which generated text occurs offer insight into user behavior: the estimated frac- tion of LLM-generated text is higher in reviews commendable 160 12 innovative meticulous 120 80 15- 40 2020 2021 40 2022 2023 2024 2020 2021 intricate 80 2022 2023 2024 notable 020 2021 2022 2023 2024 30 versatile 24 60 18 40 12 20- 2020 2021 2022 2023 2024 2020 2021 2022 2023 2024 2020 2021 2022 2023 2024 Figure 1: Shift in Adjective Frequency in ICLR 2024 Peer Reviews. We find a significant shift in the frequency of certain tokens in ICLR 2024, with adjectives such as "commendable", "meticulous”, and “intricate” showing 9.8, 34.7, and 11.2-fold increases in probability of occurring in a sentence. We find a similar trend in NeurIPS but not in Nature Portfolio journals. Supp. Table 2 and Supp. Fig- ure 12 in the Appendix provide a visualization of the top 100 adjectives produced disproportionately by AI. 1. Introduction

+1

Seems ChatGPT has been used in scientific papers.


Uploaded by Evilthing Mar 21, 2024 at 09:15AM EDT

Share Pin Download More

Origin Entry: ChatGPT

File type: png

Resolution: (1105px x 800px)

File size: 164 KB


Matt Novak @paleofuture.bsky.social Follow Al folks have now discovered "thinking" Steph Smith ❤> @stephsmithio Follow ... Sometimes in the process of writing a good enough prompt for ChatGPT, I end up solving my own problem, without even needing to submit it. 2:16 PM 7/29/24 1.7K Views Jul 29, 2024 at 4:43 PM ALT
View All 95 Images


More:

0
+1

ChatGPT - Seems ChatGPT has been used in scientific papers.

PROTIP: Press the ← and → keys to navigate the gallery, 'g' to view the gallery, or 'r' to view a random image.


◄ Previous View Gallery Random Image Next ►

More:


+ Add a Comment

Comments (0)

There are no comments currently available.

Display Comments

Add a Comment


+ Add an Image

Image Details

128 views (13 from today)

Uploaded Mar 21, 2024 at 09:15AM EDT

Origin Entry

ChatGPT

Source

Source

Tags

arxiv, paper, scientific, chart, graph

Claim Authorship Edit History
Textile Embed