The use of large language models (LLMs) is skyrocketing, and with good reason; It’s really good. Over the past couple of weeks, ChatGPT has become my favorite tool. At work, I asked him how to build an obscure piece of Linux software against a modern kernel, and he told me how. He even created code blocks with the bash commands needed to complete the task. I also told him to do all sorts of stupid things. For example, he created a fictional resume for Hulk Hogan where he has no prior IT experience but wants to transition into an Azure Cloud Engineer role. He did the same and it was hilarious. In fact, it’s so good that it can produce clear and persuasive papers for your college coursework. Because of this, there is now a need for systems that detect machine-generated text.
Recently, a team of Stanford researchers proposed a new method called Locate the GPT., which aims to be among the first tools for tackling text developed in higher education. This method is based on the idea that the text produced by LLMs typically revolves around certain regions of negative curvature of the model’s log-likelihood function. Using this insight, the team developed a new barometer for judging whether text is machine-generated that doesn’t rely on training an AI or collecting large data sets to compare text. . We can only guess that this means that human written text occupies regions with positive curvature, but the source is not clear on this.
This method, called “zero shot,” allows DetectGPT to detect machine-written text without knowledge of the AI that was used to generate it. This works in stark contrast to other methods that require training ‘classifiers’ and datasets of real and simulated segments.
The team tested Detect GPT on a dataset of fake news articles (Probably anything that came out of CNET during the last year.) and outperformed other zero-shot methods for machine-generated text detection. Specifically, they found that DetectGPT improved the detection of fake news articles generated by the 20B parameter GPT-NeoX from 0.81 AUROC for the strongest zero-shot baseline to 0.95 AUROC for DetectGPT. Honestly, this is all French to me, but it claims a substantial improvement in detection performance and suggests that DetectGPT may be a promising way forward for machine-generated text.
In summary, DetectGPT is a novel machine-generated text detection method that exploits the unique properties of text generated by LLMs. It is a zero-shot method that requires no additional data or training, making it an effective and efficient tool for machine-generated text recognition. As the use of LLMs continues to grow, the importance of relevant systems for machine-generated text detection will become increasingly important. DetectGPT is a promising method that could have significant impact in many fields, and its further development could be beneficial for many fields.
Source: DetectGPT (ericmitchell.ai)
