Ethical Concerns with Writing Assistants

I am a PhD student, working on natural language processing and argumentation, essentially trying to get computers (artificial intelligence) to understand how humans reason about ideas.

A few months ago, I was debating whether or not to join a project that involved working on a writing assistant. As part of my project proposal, I wrote down some ethical concerns I had with writing assistants in general.

I thought it would be good to post these concerns here so that other people can see them and think about them. In addition, it also provides a reference so that if I mess up and create AI systems that exacerbate these concerns, people can point me back to my own words and hold me accountable.

1. Consent

We’re using people’s writing to train an AI. Do people know that their writing is being used this way? Do they want their writing to be used this way? What does consent look like here?

2. Should we be trying to bend people’s writing towards a particular style?

I know, personally, I have the writing suggestions “Smart Compose” turned off in all my Google applications because I find the AI makes my voice more standard, and I end up using its suggestions because they sound good, rather than writing what I actually wanted to write in my own voice. Are we limiting what voices are heard, not explicitly, but through nudges? Is this a good thing? Can we develop our model such that it helps users express their own voice rather than push them to express like others have in the past? How do we gauge how important it is to do what is “normal” in writing so that people can easily understand your writing, versus doing something that’s novel or not commonly done because it fits the situation well, even if there is a “normal” way of saying the same thing?

(Note that while many of the topics previously discussed can be brought into the sphere of code-editing AIs such as Copilot and Tabnine, this one is fairly unique to writing. When coding, you almost always want to do what is standard and normal so that everyone understands what you’re doing, and when you don’t want to do something standard, usually doing it the standard way isn’t “good enough,” and it’s clear when you shouldn’t do it the standard way. In writing, this is not the case.)

3. Diversity

(This question is related to question 2, but assumes that at least representational harm at a large group level is something we want to avoid.)

How can we ensure that we’re being diverse enough in our dataset and are releasing models that can be updated if trained on future datasets? It’s highly likely that our initial datasets will be from extremely selective and non-diverse populations. How do we make sure that we have tools (and share tools!) to expand the diversity of our datasets and models in the future so that we don’t nudge minority writing styles towards majority writing styles?

For a clear example of this, I’ve noticed that captioning systems often don’t realize when someone is speaking African-American English, and will incorrectly caption their words. (For example, by replacing “ain’t” with “isn’t.”) If we are able to obtain a dataset of writing and editing from African-American English speakers, how can we ensure that our system will be able to support that style of writing?