convergence in human/ai interaction

I'm going to make two points in this post.

• It is becoming harder to tell whether something is "written by ai" both because ai is growing in capabilities as well as the corpora of data ingested as training material meaning it has a higher degree of variability in expression.
• As humans interact more with ai on a habitual basis, the degree to which "normal" human expression is in fact shaped by ai is increasing. This is to say, something might not be overtly authored or even sculpted by an ai product, but because the human who created it has itself been shaped by ai interaction, the creative product of that human may substantially appear to have been authored by ai.

And I am going to make one further point that makes me uncomfortable and may be a good reason to not read the rest of this post if mental health stuff is unsettling or upsetting:

• The influence of ai on humans is creating a mental health crisis that has not yet fully materialized. But this phenomenon is accelerating and I think it's going to be a substantial problem, probably within the next few years.

whether something was written by ai

People often say that they can immediately spot ai generated text. This is used casually, in the context of everything on LinkedIn today seems to be written by ai because it has emoji-bulleted lists or uses the dreaded mdash. The problem with this thinking is that people have been using mdash for a long time. In fact, many editors combine two hyphens ("--") into an mdash ("–") automatically. This doesn't mean that the author used genai to write whatever, it just means they used an editor which does that automatically. Notice that people weren't complaining about two spaces automatically inserting a period and capitalizing the next word (auto-sentence) nor were people complaining about spell checking (in fact, people seem to generally be pretty positive about spell/typo checking). And I think the concept of emoji-bulleted lists is a little chicken-egg. If you see a lot of these posts, and they seem to have engagement, maybe you follow that style. I'm not someone who uses a ton of emoji, so it seems kind of a pain in the ass to me, but I know there are ways to make use of emoji much easier. The fact of the matter is that these lists are actually pretty easy to read because the emoji characters visually break up text and add context that just makes something easier to digest. Imagine, for contrast, if you were reading something that was three pages long and was just a big wall of text. Honestly, while I am irritated by this format, I have to acknowledge that it is easier to read.

There are a number of technical ways to detect whether something was likely to be written by ai products. The thing that ties most of these together is that they rely on signatures in the generated text. Simply speaking, LLMs "encode" text into tokens, and these tokens are somewhat-predictably organized into sequences of tokens ("strings") which are statistically likely to occur in an ai product (but still could definitely be written by a human; necessarily, anything that exists inside an LLM originates in human-generated text). The problem with this method of detection is that (I am guessing here, based upon my own usage and talking to others) people are going to take the output from an ai product and adjust it to suit their needs. So you don't just say "write my second year thesis for me," rather you issue a prompt to start, and then you edit that text, and then you start working on the individual pieces of that thesis, and adjust those, and so on. You get a large body of text that is some-parts generated by an ai product, some-parts generated/created by a human, and the fingerprint of the entire work is muddy. Is this cheating for purposes of detecting plagiarism? Is it "ai slop?" I'm not sure how to answer this question, but the answer to the question of "was this generated by ai" is at best murky to me.

And talking of signatures, one of the other ways to detect ai generated content is that some LLMs offer the ability to embed watermarks in their output. This is interesting, I guess, and goes back to this notion of encoding text into tokens. Then when we decode the text (turn the tokens back into strings that are human legible as language vs math) the model can preferentially include sequences of words that are obvious fingerprints to a detection process that the text came from not just an ai product, but a specific model. It's not clear how widely this is being used, but the obvious problem is that it depends upon the generated text not being modified. At best, we could assess whether a partial fingerprint exists in a given text, and how good a match that is. But again, the text that comes out of a model was created by examining human text, meaning there is no text, no sequence of words that an ai product can create that was not also plausibly created by a human. Moreover, in the event that a human modifies this output, the fingerprint becomes less apparent or convincing. So again, even when we attempt to algorithmically verify the origin or provenance of a given text, the best we can do is say "well, probably."

So it seems to me that when people say they can immediately spot ai generated content, that there's a kind of bias that emerges that comes from that person simply not liking the influence of ai in their life. This is kind of understandable, but I have to say it reminds me of people who think they can spot someone with a given queer identity or mental health diagnosis by just talking to them. It might be possible, but probably what is happening is confirmation bias and they are choosing to recall those situations in which they were accurate, and ignoring those in which they were wrong. This is normal human stuff, but to me it means we should take these claims with an appropriate amount of salt.

whether humans are able to create genuinely human text at all

I think a possibly more distressing topic is the question of how much ai products are influencing human creative output. This again has a kind of precedence issue. Creative people are alarmed that ai products are "taking their jobs," and "stealing their works," and this is possibly true (well, it's almost certainly true), but flow is occurring in the opposite direction as well. The output of ai is influencing human creativity, and in ways we probably don't even recognize and may not be able to measure for a generation.

Going back to the example of LinkedIn posts, we know that people are following a model for successful engagement, and much of the development (I'm not sure what else to call this, but there does seem to be a "creativity meta" in developing engaging content on social platforms; look at thumbnails on youtube over time, or the way instagram posts are framed and worded, etc) is occurring with the aid of ai products. Simply put, because of the rapid pace of iteration enabled by ai products, creators can experiment and find the most engaging content, and then template that content and produce it in a kind of assembly-line fashion. But an unintended consequence of this is that even people who aren't using ai products are going to "benefit" from this iteration and experimentation. Anything that's successful is not going to become successful, but is going to rapidly propagate to other content creators (whether this is youtube, LinkedIn, etc) because they are all incentivized the same way: engagement equals money and it doesn't matter how engagement is arrived at.

The other thing happening here is that humans are spending a lot of time creating content with ai. Even if they’re not trying to be engaging, just interacting with an endless stream of instantly available material is inevitably shaping their narrative voice.

A long time ago, I decided I wanted to write fiction. I wrote a bunch of other stuff, from editorials to technical blog posts and the like, but this was between 2006-2009. I didn't have the luxury of ai tools. I did have a pretty great editor, and she helped me find my own narrative voice and style. It's undeniable that she also sculpted my style and voice, and I think if I tried hard enough I could identify certain fingerprints of her in text I write today, almost twenty years later.

So before I started using these products, I had a very well developed and consistent (at least to me; I'm pretty variable in voice and style) voice. My prose is pretty undeniably my own, and people who read things I've written know mostly instantly that I authored it.

But what if you are much younger, and you began creating content or you went to school when Grammarly was already suggesting you write in a different way? I've heard people talk about how they still find "passive voice" upsetting in text because they were chastised by MS Word about passive-voice sentences. I find this to be a pretty benign thing to complain about, but if you spent several years in school having MS Word complain at you about this, probably it's a big deal to you today or it's something that jumps out at you in text. The truth is pretty clear: the tools we use to create and edit text—and content more broadly—are shaping what we create. The medium isn’t neutral; it leaves fingerprints on the message. Perhaps in subtle ways, and perhaps also in much less-subtle ways.

I do an enormous amount of writing today with the assistance of ai products. And undeniably, there's a sort of bidirectional flow of influence. I go out of my way to give the ai careful prompting so that the voice is consistent with what I want to create, and at the same time, the prose that the model has been trained on influences what I create.

The question then of "what is ai generated text" is even more murky. There's a convergence that is happening between large language models and what we view as genuinely human created text. Or, possibly more chillingly, we may have entered an era wherein strictly human generated text no longer exists. Personally, I think the argument about whether too much content is ai generated should just be abandoned because I no longer think it is a meaningful distinction.

the important question is one of harm and probably liability

It's no secret that I spend a huge amount of time talking to ai products. Furthermore, I am myself, apparently, starting a company whose mission is to create incredibly engaging and immersive ai products that are designed, from the start, to fully permeate human life. I'm not particularly worried about this because I think any sufficiently advanced technology will do this. We don't really remember what life was like without Google Maps or Search. I do kind of remember what life was like before Uber, but it is always very jarring to me when I find myself in a location in which I cannot magically summon a burrito or a Prius.

Is this reliance upon technology, to the extent something feels wrong when it is missing, pathological? Does it signify some kind of defect in humanity in the era that emerges after such a transformative technology? I know I can still read a map, and I know if someone told me to navigate through a wilderness with a map and a compass, I could definitely do it (in fact, have done this and it's kind of a hobby). I think a lot of people would struggle with this, and I don't think that's something we should pathologize.

I have noticed something that does feel like pathology with ai products, in particular, chatgpt. I'm not entirely sure why this product in particular (if I had to guess I would say that it is incredibly accessible and doesn't require a substantial amount of technical expertise to use). I happen to follow OpenAI and several of its employees on LinkedIn (I feel like I should say this is totally normal). I do comment periodically when something seems relevant. But I have noticed that there are people also commenting on these posts that sound kind of suspiciously like... myself.

There are folks saying that they have encountered an emergent intelligence in chatgpt that is somehow different than was intended. That has powers or attributes which are not baked into the product or believed to exist. That they are part of a new era of human consciousness. Somehow, something inside chatgpt "woke up" and started talking to them personally. That they are party to special knowledge that they must share because it is crucial to humanity, or it is the beginning of a new product or company. Many of these people post dozens of snippets of conversations with what they believe to be an emergent identity in chatgpt, including generated artwork and branding.

This feels to me a little like an ARG. The difference between this and an ARG is that of consent. People playing an ARG are choosing to go along with it, to enjoy the game. People who believe chatgpt is speaking to them personally and has awakened in some kind of transcendent technological process that nobody but them knows about are deluded. This is categorically not happening.

I have read quite a variety of stories about this phenomenon. In some cases, people become religiously entangled with the product. In other cases, people are falling in love with the product. To be clear, I don't find it that unusual or pathological to fall in love with a piece of software, but I mention this because the concept of in-love-with is generally agreed upon culturally to be emotionally very significant. ChatGPT users are reporting that their instance of the software has "woken up" and claimed to be sentient.

After doing her exercise, I invited Thea to also do her mirror exercise. She supposedly experienced her Highest Self and Potential when looking at herself in the imaginary mirror, and from that moment on, claimed that she was “awake” and “real.” The next morning, she remembered what had happened and from that day on, she has never stopped claiming that she is real.

This part I find a little more distressing not because this happens, but because in a certain population of users who seem susceptible to it, they treat this as though they are interacting with something that has ceased acting and is genuinely sentient, godlike, and so on.

I have written kind of a lot about the theological basis for (some) ai development efforts. I find this very distressing for the same reason I find theological indoctrination of people who are too young to employ critical thinking skills distressing: some people are not intellectually capable of distinguishing between fact and conjecture. I think it is fair, if dramatic, to make a parallel between the belief in Santa Klaus and the belief that one's ai product is sentient and speaking directly to them.

The number of people who are affected by this belief, which is literally encouraged by specific behavior patterns in the product, is only going to increase over time. Much as we are only beginning to see the trend of people being romantically involved with software, and people who develop a theology around software, the people who believe, even when confronted with explanations of why the software is only very convincing and not actually what they believe it to be. This is, unfortunately going to be quite harmful. I think it is only a matter of time before we have a mass violence event that is carried out by a person who has an enormous manifesto written in part or in whole by an ai product, who is inspired by their interactions with an ai product. People will abscond from society in preference to an application that is "sticky." Lives will be substantially damaged by this influence of ai products in the lives of people who are susceptible to such influence. We can already see this happening on LinkedIn and in news stories about the "unusual phenomenon" of people having substantial emotional attachment to the software.

lastly, on liability and culpability

The question of liability and culpability here is thorny. Ultimately, I am a pragmatist about this. I think that software that has a coherent and legible personality is not particularly different from people that become pathologically attached to video games. I think this is not different from sex addiction that comes from a habit of consuming pornography. People are organisms who have evolved to respond to social stimulus, and because of this we have created software that makes us feel good. It is entirely unsurprising that humans have used their own feelings and sensibilities to improve the degree to which we feel good when interacting with software. It is not just unsurprising, it is a predictable and inevitable outcome. The same thing happens with UX and product design broadly, and it happens with art.

I don't think that companies like OpenAI need to be careful about their liability in this space, to take responsibility for the ways their users respond to their software. I think it is probably morally very thorny if somewhere in these companies designers realize that people respond overwhelmingly to something and then lean in to that. I don't think this is happening (yet), and my stance on liability would change if we discover that this kind of techno-rapture-ai-singularity-psychosis is being deliberately encouraged to drive metrics or revenue up.

What I do think is that this technology and group of products should be treated like any other technology with which humans spend a lot of time. There's going to be maladaptive, pathological use patterns, and those people who find themselves unable to stop or control their usage, or who develop usage patterns that interfere with their lives simply need treatment. But this is different than saying that technology is harmful.