Bot Problems

[soc_puppet]

I briefly saw (and participated in, for exactly one reblog) a thread on Tumblr about whether AO3 should allow works produced by Generative AI. OP was in favor, as was the first reblogger, where as at least one reblogger was firmly against all AI generated works on AO3, and several (myself included) shared the reasons why AI generated works currently are allowed on AO3. One of the rebloggers who is firmly against any AI generated works being allowed on AO3 said, "Like anything else in a shared social space, an enforced social norm requires actual social consequences for breaking it. This? Is just capitulation."

Anyway, that person also said that they're done talking about it, and I wanted to respect that, so I'm taking my thoughts over here. If you recognize the quote in question, or know who the person is, I will politely ask you to respect their boundaries and not go looking for the post and/or reblogging it from them with your own commentary; we can talk in the comments of this post, which they can safely ignore. I'm personally struggling with the point they raised, and will probably not be able to get to sleep without working some of it out, so I'm doing so over here where, again, they should be able to safely ignore it.

So. AI generated works on AO3.

For moral and ethical reasons, AO3 should not host AI generated works. This I fully agree with. For the same reasons why LLMs should not be used in place of human creativity (destroying the environment at a much faster rate than general electronic use, plagiarism, etc etc), AO3 should not give these works a safe home.

Unfortunately, that leaves us with the practical questions related to How to enforce a Gen AI works ban?

The biggest practical question is, "How can we tell something is AI generated?" To the best of my knowledge, outside of people admitting that they generated a work using ChatGPT and the like, there is no reliable way.

With many LLMs trained on AO3 itself, legalities aside, that means that a lot of the commonly touted "tells" come from AO3's "house style". So using those "easy tells" is out.

On top of that, other "AI detectors" will get stuck on stylistic differences that actually indicate someone is a non-native English speaker; AO3 already has a demonstrated racism problem, and even if we assume that everyone reporting fics for being AI generated is doing so because they actually believe the fic is AI generated (having run it through one of those AI detectors, for example), a lot of fics that shouldn't be are going to end up targeted.

And speaking of reporting in good faith, let's talk about reporting in bad faith. There are plenty of anti-fans who want AO3 to stop hosting much of the very content it was built for; some of them are hopefully getting a clue over the Itch.io payment processor thing, but that will never get through to all of them. (Hell, some of them might even be behind the Itch.io payment processor thing!) There are also already reports of (potentially bot-generated) comments on fics accusing them of being AI generated, which is not helping the morale of any writers. I can all too easily picture malicious actors going around and reporting any fic they don't like as being AI generated. What a waste of time for everyone involved! And with the Policy and Abuse team already poorly trained and overworked, adding this to their plate just seems... cruel, almost.

Practically speaking, this also means that the only works that would get taken down are those whose posters admit are generated by AI. And, given AO3's stance on not addressing off-site harassment, it would possibly be only works whose creators admit on AO3 itself are AI generated; I don't know how AO3 deals with any off-site plagiarism admissions, which is what I'd compare it to, so I'm at a loss there.

Which brings us to the big argument AO3 is using in favor of hosting AI generated works: At least when AI generated works are allowed, people will admit that's what they are, and people who don't like that can filter them out or just plain not read.

I honestly feel like this is potentially the weakest argument of the bunch, and possibly the only one I have an actual solution for; if AO3 disallows AI generated works, then yes, people would still post them without admitting that they're AI generated. Hell, they might even still admit to it! Look at how many people get caught sharing Ko-Fi and Patreon links in their notes! At least that's a report-able offense.

But right now I'm stuck on the question: Are the people who use Gen AI to "write" fics admitting it? Or is Gen AI unpopular enough that they've largely stopped being honest about it? If it's the latter, AO3 may as well roll out a "No AI Generated Content" rule, with the clause that any reporting must have evidence of self-admitted use of AI. Otherwise, we're basically already living in the universe where we can't tell for sure whether any content is AI generated, just without any recourse if anyone who does post it slips up and admits it.


Anyway, those are my current thoughts. So now probably the thing that will keep me up is trying to decide whether to create a Tumblr poll about that last point ("Do you post AI generated works to AO3, and if so, do you admit that they're AI generated?")

Comments

[izzet_bedtime_yet]

There was a rash of spam comments a while back that would accuse fics of being AI-generated and urge readers to use their (AI) tool to analyze other fics. I remember getting a bit of a laugh out of them trying it on one of my fics that was written before the advent of the tool it claimed I'd used, but also being really angry because that sort of thing is absolutely poison to normal fan interactions. Which is to say, I agree with you about the risks of bad-faith and mistaken reports, between that and fandom's long tradition of brigading.

In a perfect world, people would all tag their AI-generated stuff properly, and in a realistic world... well, we can work on it?

[soc_puppet]

Right? Unless we can get either actual, accurate AI detection, or a firm, clear rule about when it's allowed to report, then allowing reports of AI generated works is going to be a minefield.

Honestly, one of the polls I'm thinking of doing is, "Do you post AI generated work to AO3, and if so, do you say it's AI generated?" With options being "Yes and Yes", "Yes and No", and "No". Just so I can get more of a feel on how things stand.

No answers here, either.

[pangolin20]

Oh, that's quite hard to find a solution for! The best thing I can think of is indeed encouraging people to tag them as being such, because then you at least know mostly what they are. That doesn't answer the question of what to do with them then, though... In order to do anything about these fics, they need to be out in the open, and any attempt to do anything about them will probably drive them into obscurity (I don't think many people will put up tagged AI fics if they know it'll be deleted anyway). The other way something can be done about them is social pressure, but I don't know enough about AO3 to say how much effect it would have on people who post AI fics.

Re: No answers here, either.

[soc_puppet]

Right? To the best of my knowledge, that's how things are on AO3 currently, but like. Is that capitulation? Would it be better to outright ban AI generated content? Would making AI generated works against the TOS meaningfully reduce the number of them that are hosted on AO3? If so, to what degree?

So many questions, none of which should be answered by gen AI itself 😑

[pangolin20]

The best answer to the latter questions would be for someone to compare AO3 with a platform that does ban AI fics, I think. I don't know if that'd be very feasible, but it might be worth trying. (Of course, I don't know which sites ban AI fics, and AO3's a quite large site, so the dynamics of comparison sites will probably be different.)

And having looked up their statement... I do feel allowing AI fanworks conflicts with AO3's spirit, and I do feel that it's not a great position to take on such a controversial subject.

[winterfirelight]

Gosh I have so many thoughts on this. I read college essays as part of my job, and we've been having SO many conversations on this topic. In our case there have been some fairly clear tells indicating AI use, but this is getting increasingly harder as the tech develops, and even if we're 100% sure an essay is AI-generated, in part or in full... well, how can we prove that in a meaningful enough way that we can fairly reject essays on those grounds alone?

At least for now, we've come to a similar conclusion of allowing AI use, but providing education on ways to use it more mindfully and judiciously. We also ask students to disclose if/how/how much they used it - partially to make them think about it harder, but mostly just so we can (attempt to) get a sense of how widespread it is and how it impacts essay form/quality/etc. Similar to the polls you're thinking about, but in an academic context rather than a creative one. We get some really interesting data out of it, but we have to take it all with a grain of salt since we know that students won't all tell us honestly no matter how much we reassure them they won't be punished for it.

[soc_puppet]

It's a sticky situation! I like the solution I've seen some people use, where they get their students to write something using AI, and then dissect everything that's wrong with what the AI wrote, but that works best for preventing AI use in the first place, not detecting it.

[winterfirelight]

Yeah, I love that idea of having students examine AI's output so they can learn it's not infallible and they have to be careful when using it or not use it all. But you're right, that doesn't help with detection. And honestly, it's not just students who need to learn that lesson!! I see so many experienced professionals these days thinking you can use AI as a search engine. Or seeming convinced that AI will do their work faster and better than they could do it themselves, but not doing any kind of review to check whether that's actually the case. If you have to correct a million mistakes, or spend time crafting the exact perfect prompt (and follow-up prompts) to get AI to put out what you want, is that really faster and easier than just doing it yourself with your human brain that's actually capable of critical thinking???

Drives me NUTS to see all these smart people actively make themselves dumber on purpose.

But I digress :) All that to say, this is a very important conversation, and I'm glad to see people talking about it. I'm no longer on tumblr myself, but I'd love to hear the results if you put out a poll on this.

[pikkugen]

Very much this. I'm honestly afraid of some people I call friends who put everything they do through AI and proudly claim so. It's a use-it-or-lose-it situation, and I'd much prefer to lose the damn plagiarization machine and all its problems than my very own little brain. (Not to mention all the global problems to the environment it creates.)

Also I genuinely enjoy writing stories myself. I can't draw very well, but I love doodling whatever I doodle myself. I'm sure if they made an AI robot that knitted or crafted some people would externalize their hobbies to it. Personally, I don't understand the need to. I miss the time when robot labor was seen as a way to give people more free time, not steal away their creativity, but maybe I'm just old.

[winterfirelight]

Yes, exactly! I fundamentally do not understand why anyone would want to use AI for creative pursuits in the first place. I *like* putting in the work to make a story happen. Having something else write a story for me, or even having it give me ideas for what to write, defeats the whole entire purpose as far as I'm concerned.

I feel older every day, with this stuff.

[mistressofmuses]

I pretty much feel the same.

I loathe genAI in "creative" endeavors, and am strongly morally opposed to their inclusion in the archive. (I agree with points made that AO3 is for fanworks, and genAI is not a "fan" of anything; even if a "fan" put in a prompt to the AI, AO3 does not allow prompts to be posted as if they were fanworks, so something that had no actual fan participation in the creation beyond the prompt itself is not a fanwork. Even outside of that, genAI is plagiarism and plagiarism is not allowed!)

However... there is no foolproof way to determine whether genAI was used, and grudge-reporting and false-positives are a serious concern. There can and *should* never be a "quality test" to determine what stories are allowed. (I would hate for people to be afraid of posting their works out of fear they aren't "good" or "unique" enough. I'd hate for people to be afraid of posting fic in a language other than their native one because some of their phrasing might be different.)

I do understand the frustration people have with "we can't make a rule against it because people will break the rule anyway," which seems to be what the argument often gets boiled down to. The archive DOES have rules, and takes action when they're broken!

But "making a rule against it is very difficult to enforce because there is no objective way to determine what works are breaking that rule" is a different concern.

[soc_puppet]

But "making a rule against it is very difficult to enforce because there is no objective way to determine what works are breaking that rule" is a different concern.

That's making me feel better, honestly. Especially with the extremely limited amount of work I'd personally like to allow that was in any way directly touched(?) by Gen AI.

Like, if someone used Gen AI to get inspiration or to figure their way out of a block, then the actual created stuff is still 99% human effort, you know? I'd be willing to accept that with a tag, or some other note denoting the use of Gen AI, especially if the work was particularly time consuming, and double especially if the creator did it before realizing/learning how bad Gen AI is. Or if they're working off of prompts from AI Weirdness's Botober (which, to the best of my knowledge, actually got all of its data more-or-less ethically, and has definitely been using AI stuff for educational purposes).

Anyway, much to think about.

Thanks for helping me understand

[jesse_the_k]

...how charged this issue is.

Re: Thanks for helping me understand

[soc_puppet]

Sure thing! It's a headache and a half, no mistake. (As were my Tumblr notifications after making a poll related to it, oof 😩)