By RNZ

Universities’ arms-race against cheats took a step forward today with the activation of software that can detect the use of Artificial Intelligence systems such as ChatGPT.

Turnitin, which all eight New Zealand universities use, said it could now spot AI-generated material with 98 per cent accuracy and it had switched on that ability for its New Zealand customers.

Academics told RNZ the update would help, but they doubted it would be effective for long or for students who knew sophisticated ways of using the tools.

Turnitin regional vice-president for the Asia Pacific, James Thorley, said the software estimated what percentage of a text was written by AI and highlighted the offending sentences.

Thorley said AI-generated work was harder to detect than other types of cheating.

“Each time that ChatGPT or any other AI writing tool generates anything it is unique so no two creations will ever be the same, so you cannot detect it in the same way that you detect copy-paste plagiarism from whatever source,” he said.

“You’re really looking for this difference between how an AI writer would write and how a human would write but obviously that’s a lot more complex than simply matching the same text.”

Chatgpt bot. Photo / 123RF

Testing indicated Turnitin would be effective even if students edited AI-produced work to make it look like their own, Thorley said.

Surveys in the United States indicated 25-50 per cent of university students had used AI for various tasks, he said.

“Everyone’s actually very excited about the potential of these kinds of tools. Detecting doesn’t necessarily mean that there’s misconduct happening, that it’s wrong. It’s all about getting a bigger picture of what’s going on and starting conversations.”

Catherine Moran from the University of Canterbury said it was redesigning its assessments because of AI and she expected Turnitin would be used for assignments that had not yet been changed.

Turnitin update only likely to work short-term as students adapt - senior lecturer

A senior lecturer in software engineering at Victoria University, Simon McCallum, said the update to Turnitin would help catch misuse of AI, but he expected it would only be effective for a short time and when students were doing it in an unsophisticated manner.

“There’s a whole bunch of these other AI tools that when you layer them on top of each other make any of those detections much less certain and the problem we have is that if you’re going to ping someone for plagiarism you have to be certain that they break the rules not just have a hint that it’s probably an AI that wrote this.”

McCallum said a large number of students were using ChatGPT and similar “large language model” programmes.

“Students are using GPT a lot. Some of them are getting very good at it and the ones who are good at prompt-engineering, creating good input to the AI, are able to generate very realistic work that looks very much like their own work in much shorter time than they would if they were having to write it themselves.”

He said students should learn to use AI because it would be widely used in many jobs, but he worried some were doing it without putting in any thought.

“I’m worried when people are abdicating thinking. They’re not using their mind or making the effort to learn.

“We might need to consider every assignment a group assignment because we have to think that every student has access to a group member, an AI, that can do some of the task for them and so then you have to say, well instead of treating it like your own work, in the group how did you contribute.”

ChatGPT and the like were so important the entire education system should shut down so teachers and lecturers could figure out what to do, McCallum said.

“This is a pandemic-level event. It’s the kind of thing which is going to fundamentally change a lot of what we do. In the pandemic we suddenly had to go online and we suddenly had to teach in a whole new way and this is the same level of change that’s needed.

“Let’s just close schools for a month, five, six weeks, and just spend that time doing professional development with all the teachers and with the university lecturers and say all of you need to get up to speed with this because it has fundamentally changed education.”

- John Gerritsen, RNZ