PSA: Stack Exchange Inc. have announced a network-wide policy for AI content

Question

A couple of days ago, Stack Exchange Inc., the company running these sites, announced a new policy for handling AI content. This policy goes against our own, local policy on how to deal with AI content, where the U&L community made it very clear that such content was not welcome: How do we want to handle AI-generated answers?

I am writing this post to make sure that the U&L community is aware of this new network-wide policy and to provide a place where we can discuss it and maybe decide how to handle it. The full text of the new policy can be read below, but I urge you to visit (and vote on) the original on Meta Stack Exchange as well:

We recently performed a set of analyses on the current approach to AI-generated content moderation. The conclusions of these analyses strongly indicate to us that AI-generated content is not being properly identified across the network, and that the potential for false-positives is very high. Through no fault of moderators' own, we also suspect that there have been biases for or against residents of specific countries as a potential result of the heuristics being applied to these posts. Finally, internal evidence strongly suggests that the overapplication of suspensions for AI-generated content may be turning away a large number of legitimate contributors to the site.

In order to help mitigate the issue, we've asked moderators to apply a very strict standard of evidence to determining whether a post is AI-authored when deciding to suspend a user. This standard of evidence excludes the use of moderators' best guesses based on users' writing styles and behavioral indicators, because we could not validate that these indicators are actually successfully identifying AI-generated posts when they are written. This standard would exclude most suspensions issued to date.

We've also identified that current GPT detectors have an unacceptably high false positive rate for content on our network and should not be regarded as reliable indicators of GPT authorship. While these aren't the sole tools that moderators rely upon to identify AI-generated content, some of the heuristics used have been developed with their assistance.

We've reminded moderators that suspensions (and typically mod messages as well) are for real, verifiable malfeasance only, and should not be enacted on the basis of hunches, guesses, intuition, or unverified heuristics. Therefore, we are not confident that either GPT detectors or best-guess heuristics can be used to definitively identify suspicious content for the purposes of suspension.

As always, moderators who identify that a user has a problematic pattern of low-quality posts should continue to act on such users as they otherwise would. Indicators moderators currently use to determine that a post was authored with the help of AI can in some cases form a reliable set of indicators that the content quality may be poor, and moderators should feel free to review posts as such. If someone is repeatedly contributing low-quality content, we already have policies in place to help handle it, including a suspension reason that can, in those cases, be used.

Answer 1

Personally, I consider the previous policy as both appropriate and existentially necessary for the future of the community. I support and encourage any moderators here in joining the general strike.

While I respect the right of each person to choose for themselves, I'm looking for the exit. I just don't see that there is much meaningful room for compromise here on the community side. None of us are interested in spending our time competing with, much less refining, garbage generated output - for others to claim both the credit and the profit. From what I gather that sentiment applies to the mods as well.

I'm still watching for now, but I'm already starting to move over to Codidact and see how I can contribute there. As a result, I now care more about the data dumps, so this is doubly bad timing for SE. In order for me to resume meaningfully contributing here as well, I would need to know my effort will be equally available to all in the dumps going forward.

Basically, because of this AI debacle, I trust SE significantly less than before. Because I trust them less, the dumps are more vital consideration for my participation than ever before. Thus unless they reverse course on both issues, I will probably, finally, fade away from here entirely.

Either way, best of luck to all those working to resolve the situation with SE, and I remain grateful to everyone who has contributed to this wonderful community resource over the years.

Answer 2

May I start by thanking you for bringing this to the community's attention. This is a very serious matter that concerns us all. We are all very worried about the extremely detrimental effect that AI content could have in the future.

I personally am worried about the issue generally after I stumbled on a website for a supposed "School" containing thousands of articles which I believe might have been largely generated through AI content for the sake of one or perhaps a few people's CV. This does not bode well for the future of knowledge sites like U&L.

But Keep your powder dry...

May I urge our moderators here to exercise caution and restraint on this topic. Keep your heads cool as possible.

I acknowledge the current "general strike" happening over this matter and I realise our own moderators might have chosen to take part. I would urge you not to take part in future action until, at least, some more has been made clear:

Give SE opportunity to fail

I find this teaching one of the most excruciating to follow as a mentor, but it is invaluable. Until the real impact of this policy can be directly seen combined with the [lack of] tooling SE have chosen to adopt, none of us really know the future.

SE have asked moderators not to take action and there is open discussion that they have set a strict policy that general user base is not privy too. What that does is to put the issue squarely in the hands of SE. SE might well have good reason for keeping even more hidden from moderators to prevent leaks to the public.

It will be really painful to watch them fail at this, but this is really a topic where we MUST let them try [and fail] rather than not even letting them try.

SE are not wrong in the [publicised] justification for this

I think our moderators here are ace! Really some of the best on SE.

SE have highlighted two particularly thorny issues:

• There is a worryingly high number of false positives
• There is a real concern about racial bias

Nobody is calling anybody racist! The problem is that people have used writing style as an indicator of ChatGPT content and this is absolutely prone to false positives. ChatGPT was trained from real language after all!

What's really worrying about this is the idea that people who were educated by particular schools or who learned English through a particular educational system are more prone to having their content falsely flagged as AI generated... leading to a really pernicious racial bias that cannot be easily identified or controlled.

The extent of this problem is not yet well understood, but at this time it's of sufficient concern that it could genuinely get big organisations sued.

There are better tools than people for this

SE have not publicised, nor are they likely to publicise, the tools they engage to begin catching AI content. But theoretically technology is likely to be better than people at this.

Again, we have great moderators. But I want our moderators to spend their time moderating user content, not chasing down bots and those who copy-paste from ChatGPT.

There are many features of cheats that become so much easier to spot with technology and with data that is not available even to moderators. E.g. ChatGPT does not commonly spend time writing its posts and forming saved drafts as they go. Click-through time between reading a question and posting sizeable answers is much lower. The relationship between which questions appeared on screen and how many immediately got answered by that user is wholly different. The time spent browsing questions is different.

Strike action hurts us all

As a gentle reminder, those who get hurt by this strike most are the user base. Historically strike action is as likely to anger customers as much as change the minds of management... and that anger, historically, gets directed at those striking.

I really think our mods here on U&L are great. I am genuinely worried that further action could create "friction" between users and moderators.

This isn't to say anyone should put up with poor treatment. But where strikes have such a poor success rate and high collateral cost, I'd advise anyone to exhaust as many possible alternatives as you can find.

And finally

The final solution to this hellish nightmare MUST NOT be asking moderators to clean up a flood using a mop and bucket. I really do empathise with moderators who feel so very let down by SE right now.

Instead of using this strike action to demand back your shiny mod hammer, consider demanding that SE do more to implement better tools themselves that will prevent it even getting to your inbox.

---

You guys are AWESOME! Hope to see you back here soon.