Ai detectors suck

[u/Prs8863765]

Me and my Tutor worked on the whole essay and my teacher also helped me with it. I never even used AI. All of my friends and this class all used AI and guess what I’m the only one who got a zero. I just put my essay into multiple detectors and four out of five say 90% + human and the other one says 90% AI.

[u/waynemr]

Smash them in the face with facts.

At a high level, detectors function on a kind of watermarking that is not an industry standard or universally applied, further its extremely easy to to prompt a model to abandon its form and any watermarks it has. Finally most pattern matching is based on the training and test data sets, the vast majority of which are common literature and formal writing. Formal writing is by design meant to have a uniformity in structure and tone, making detection for these use cases even more difficult.

https://arxiv.org/abs/2303.11156
https://arxiv.org/abs/2310.15264
https://arxiv.org/abs/2310.05030
general search term: "arxiv AI detection not possible"

It's worth noting that what is done in these evals is very similar to the kinds of eval benchmarks done to test how "smart" a model is, a quick look into the arguments and debates on how to even evaluate an LLM against others should warn most thinking folks off from using a content evaluator in this way.

I do feel it is possible to detect if an output is from a specific model however this requires full access to the model's weights and more computation time than what would be cost and time effective for the task.

IMO embracing tools like detectors is an attempt to preserve the "old" way of teaching in the face of a world demanding an entirely new paradigm.

See also https://hai.stanford.edu/news/ai-detectors-biased-against-non-native-english-writers and https://www.vanderbilt.edu/brightspace/2023/08/16/guidance-on-ai-detection-and-why-were-disabling-turnitins-ai-detector/

[u/mrchuckmorris → u/waynemr]

Caveman: [invents wheel]

Caveman Teachers: "No use wheel! Carry all things!"

[u/ArcticHuntsman → u/mrchuckmorris]

except if using the wheel led to the decay and atrophy of important muscle groups leading to no longer being able to use the wheel as effectively. Acting as if getting an AI to write a full essay is fine within an educational space is dangerous.

[u/StoryLineOne → u/ArcticHuntsman]

How we teach has to be radically different. What you or I learned in school is probably not what they should be teaching anymore - if an LLM can spit out an essay, then maybe Essays are pointless. So now it becomes - how can you test an individuals knowledge, knowing all the tools that are available?

[u/fmfbrestel → u/StoryLineOne]

The essay was always pointless, it was the content of the essay that was the lesson. It was learning to think in a structured way about a focused topic. -- Lessons still important even post AI.

[u/Single_Management891 → u/fmfbrestel]

You are 100% on point. If all people know how to do is ask questions to a machine we end up with a lack of critical thinking. As time goes on the ai will also get dumber due to all the bs on the internet and ban idiocracy is now our reality, like the people actually water plants with something akin to Gatorade.