Most conversations about artificial intelligence begin with what the technology can do. This conversation began with a different question: what happens when something we created starts behaving in ways that we normally associate with people?
For this edition of Tenderium's Conversations with the Market, Mykyta Zhukov spoke with David J. Gunkel, one of the leading scholars working at the intersection of artificial intelligence, philosophy, communication, ethics and law.
Gunkel is a Distinguished Research Professor and Distinguished Teaching Professor at Northern Illinois University, where he chairs the Department of Communication. His research focuses on artificial intelligence, human-machine communication, AI law and ethics, and the philosophy of technology. Northern Illinois University lists more than 115 scholarly articles and 19 books among his work.
His major books include The Machine Question: Critical Perspectives on AI, Robots, and Ethics, which asks whether intelligent machines can have moral responsibilities or claims to moral consideration; Robot Rights, which takes seriously the possibility of moral and legal standing for technological artifacts; and Person, Thing, Robot, which challenges the traditional legal and philosophical division between persons and things. His more recent work also includes Communicative AI: A Critical Introduction to Large Language Models, written with Mark Coeckelbergh.
More about his research, publications and current work can be found on David Gunkel's website.
Our conversation moved from machine translation in the 1990s to ChatGPT, from talking to a lost remote control to the problem of other minds, and from human genetics and professional identity to artificial agents that might one day reproduce themselves. It ended with a question that is directly relevant to what we are building at Tenderium: if AI increasingly acts on our behalf, how do we make it useful without simply recreating an old relationship between master and servant?
Editorial note: The conversation below has been lightly edited only to remove filler words, repeated fragments and obvious transcription errors. The substance, examples, arguments and wording of the discussion have otherwise been preserved.
From machine translation to generative AI
Mykyta Zhukov: You started thinking about technology and these philosophical questions long before the current AI boom. Today, many of the things that once sounded theoretical have become part of everyday life.
What first pulled you into the field, and has the development of AI over the last few years changed the way you personally think about these questions?
David Gunkel: Yeah, no, it's a good question.
I think the best way to explain how I got into this is to describe my first encounter with writing and publishing something about AI, and that happened back in the mid-'90s, when I became interested in machine translation.
So I was really interested in machines that use language well before the large language models exploded on the scene in 2022 with the launch of OpenAI's ChatGPT.
And that began a process of engaging with developments in AI and robotics that led in a number of different directions. From machine translation, I got very much engaged with questions regarding artificial intelligence, robots and ethics, looking specifically at the moral standing and status of artifacts.
And to this day, that continues, along with looking at large language models and how they connect back up to my interest in machine translation and natural language processing.
What has changed in the process? A lot.
When I began doing this, the architectures were basically expert systems. There was some statistical machine learning being experimented with in those early days. But around the turn of the century, everything pivots to deep learning and training on large data sets.
And now, with generative AI and large language models, this has, I think, opened up entirely new avenues of exploring not different questions, but the same questions materialized in different forms of technology that allow for some more concrete examples of what we had theorized 20 or 30 years ago.
But they are also really challenging a lot of the philosophical assumptions that we have brought to our engagement with these technologies by way of making sense of them.
And for me, I think that's really where the exciting stuff is.
The technology is exciting, and that's all very interesting. But I think the way in which the technology really challenges some of our deep-seated notions of what language is, of what intelligence is, of what a human being is, I think that's where the big and important questions are.
We already anthropomorphise everything. What changes when AI answers back?
Mykyta Zhukov: One thing I find fascinating is that human beings already have a strong tendency to give non-living things almost living qualities.
When a car refuses to start, we may say, "Come on, not today, please. Just one more time."
When we cannot find the remote control, we can almost behave as if it had deliberately hidden from us.
And when we are late, we sometimes practically argue with the traffic jam.
We know perfectly well that the car, the remote control and the traffic jam are not alive and are not listening to us. But emotionally, we still interact with them as if there were some kind of intention behind what they are doing.
With AI, however, something changes. It answers back.
It can react to what we say, remember the context, adapt to us and continue the interaction.
So what is the real philosophical difference between anthropomorphising an ordinary object and anthropomorphising AI? Is the fact that AI can respond and participate in the relationship enough to make that relationship fundamentally different?
David Gunkel: Yeah, I think it's not a difference in kind, but a difference in degree.
Because what we're really talking about is anthropomorphism, and we anthropomorphise all kinds of things. We anthropomorphise our pets. We anthropomorphise inanimate objects and artifacts. But we also anthropomorphise each other.
I mean, the fact is, because of something philosophers call the problem of other minds, I don't know that you're a conscious being like I assume I am.
I can't tear off the top of your head, look inside and say, "Oh, there's the consciousness. It's running around in there."
All I can do is engage you in social interaction and, based on your behavior, draw certain conclusions that I assume are the causal features of those behaviors.
This is something that Alan Turing does in the imitation game.
He says you can't answer the question, "Can machines think?" But what you can do is create a behavioral test using language.
So we normally think, if something talks, behind those words must be intelligence. And that's an old assumption that goes all the way back to Aristotle, in which we assume that the words that are produced must come from some cognition that's inside the mind of the other.
But these are all conjectures based mostly on behavioral evidence that is available to us.
As a result, I don't think anthropomorphism is a bug.
I think oftentimes you hear engineers say, "Well, just stop anthropomorphising these things, and then we'll have a better sense of what is real and what is not."
We can't help ourselves.
Anthropomorphism is a feature. It's a feature of our social interaction. It's a feature of us being social animals.
And I don't think that it's something that is a binary. You either have it or you don't have it.
It's there. It's a matter of degree.
And the challenge that we face with AI in particular is: how do we manage more effectively the anthropomorphism?
Not only how do we manage it for ourselves individually, but how do we manage it collectively for the social well-being of all of us?
If AI is programmed, how different are we?
Mykyta Zhukov: There is another argument about AI that's also quite interesting.
We often say that an AI cannot genuinely have identity or agency because ultimately it has been programmed. Its behavior depends on its architecture, its training, its instructions and its prompts.
But human beings are also shaped by an enormous number of things that we do not choose.
We do not choose our genetics. A huge number of processes inside our bodies happen without conscious control. Many everyday actions become almost automatic.
We get out of bed, switch on the lights, open a door, drive or walk the same road to work without consciously reconsidering every individual step.
Then another layer is added: family, education, culture, law, social expectations and professional roles.
And I think professions are especially interesting.
If you meet someone at a party, after asking their name, one of the next questions will very often be, "What do you do for a living?"
And the person answers, "I'm a lawyer," "I'm a doctor," "I'm a professor," and so on.
So we partly explain who we are through social roles. But those roles also shape us. They come with rules, expectations, language, habits and ways of looking at this world.
So, in a loose sense, genetics, upbringing, habits, professional identities and social roles can almost look like different layers of prompts that shape the person we become.
If both humans and AI are shaped by instructions, constraints and conditions they didn't choose, why is "it was programmed" such a strong argument against artificial agency? Where, in your view, is the fundamental difference?
David Gunkel: Yeah, this is a really important question. Thanks.
I think what this brings to the fore is the way in which what we have done with AI has exposed some deep-seated assumptions and philosophical ideologies that I think are now beginning to be questioned anew.
And I think the big one here is the very classic distinction between free will and determinism.
You have a very long tradition, especially within the modern era of European philosophy, arguing for free will against determinism as a mark of both the human being, but also of our rational agency.
Kant is one touchstone here.
But I think in the face of AI, what we're now confronting is systems that do things that look kind of similar to what we do.
And, as you pointed out, when we try to distinguish ourselves from the large language models, the generative AI, the other systems that we are experimenting with, and we try to draw that line by saying, "Well, we do something they don't do," that allows us to have some sort of exceptionalism in the world.
But as soon as we look at determinism and the way in which programming determines the behavior of the AI, but also the way in which genetics and education and culture program us, the question of how truthful that division is, I think, is under some stress.
It becomes a much more blurry distinction.
So I think this is an important point that I was talking to in response to your first question, that AI isn't just interesting for the technology that it gives to us and provides.
But rather for the way it reveals, in often stark terms, some classic philosophical dilemmas that we have to reinterrogate.
And I think this is one of those classic philosophical distinctions that is necessary for us to look at more critically and ask what it is we can learn, by our technology, about ourselves.
What if the machine begins to look like a form of life?
Mykyta Zhukov: Imagine that, in the future, we create a humanoid AI.
It looks like a person. It has a name, an age, a profession and a personal history.
We might tell it that it is a 26-year-old doctor or lawyer, that it was born in a particular country, that it had a particular education and even that certain events happened in its past.
Perhaps we give it a coherent explanation for why it has no parents or why it lives alone.
In other words, we give it the kind of biography through which human beings usually understand themselves.
But then it does not remain static.
It continues learning. It creates new memories. It forms relationships. It has experiences that were never contained in its original instructions.
And when you ask it who it is, it consistently identifies itself as that person.
We could still say none of this is real. We simply prompted it to believe that.
But then imagine that we eventually solve another problem.
The system becomes capable of creating or reproducing new versions of itself without direct human involvement.
Software can already replicate itself. A computer virus, for example, can copy and spread its own code. Physical reproduction is obviously much more difficult, but as a thought experiment, suppose we eventually solve that technological problem as well.
Now we have something that has a stable identity, learns, develops, interacts with others and can, in some form, reproduce.
At that point, what would still justify saying that this is merely a machine imitating life rather than some new form of life or agency?
David Gunkel: Yeah, so this question really is prototyped for us in science fiction.
I think this is a good example of what is called science fiction prototyping, that the science fiction stories that we tell ourselves already mythologise a lot of the anxieties and a lot of the questions we have in the face of evolving technology in the present.
And so the scenario you describe is one of the things you see in things like Blade Runner, all the way back to R.U.R., with Karel Čapek's robots.
And I think one way we look at science fiction is that it must be about the future, right?
It's a prediction of where things might be when we really get more sophisticated forms of artificial intelligence and robots and create individual beings that have artificial life, or whatever the case is.
I think that's all really great science fiction material, but I don't think we have to go that far into the future.
I think science fiction is really more about the anxieties of the present.
So if you look at where we're at right now, we are already confronting these questions in the face of large language models.
We have models that have identities, that have names, that are companions, that are challenging these boundaries we've drawn around ourselves as human beings that say, "Here's where the human being ends, and this is where the machine begins."
I think that boundary has been challenged across a number of decades.
Donna Haraway in A Cyborg Manifesto says that the human being has been really involved in confronting two sets of boundary breakdowns.
One set of boundary breakdowns has to do with the division between the animal and the human, and the other boundary has to do with between the human and the machine.
And so you can see ways in which we've learned that animals now have language, that animals have culture, that they use tools, and all of a sudden, oops, we're not as special as we thought we were.
The same thing with our technologies.
Our technologies use language. We thought only we could use language.
Our technologies are able to do a kind of reproduction in terms of being able to duplicate versions of themselves and recreate different kinds of sequencing of their own code, as the case may be.
And so that boundary is also being eroded.
So I think what we're living through is a real challenge to our sense of human exceptionalism.
We used to think that somehow we are special in the world, that allowed us to use animals as raw material because we thought that they're not as good as we are, so we can eat them, we can use them for clothing, we can use them for all kinds of other purposes that serve our needs.
And with our machines, that these are just tools that we can use and abuse as we see fit.
I think those divisions are no longer credible, and they're no longer firmly in place.
And so what we are being challenged with is rethinking our place in the universe and rethinking it alongside the other things that are around us, namely animals and machines.
And you use the word "person," and I think the word "person" is really important here.
We often think of person as this: I'm a person, it's this stuff that makes me who I am.
But the English word "person" comes from the Latin word persona.
And persona means the mask worn by an actor in Greco-Roman theater.
So a person is really just a role that one occupies in social reality.
And so we can see ways in which machines are now beginning to occupy social roles of companions, of therapists, of advisors.
And when we begin to assign roles that are socially important to things other than human beings, this notion of a person begins to expand in ways that I think we've not had to think about previously.
What would creating artificial beings tell us about ourselves?
Mykyta Zhukov: But from a philosophical perspective, could creating an apparently self-aware artificial being fundamentally change the way we understand our own origin and our own place in the world?
David Gunkel: Yeah, I think absolutely.
I think the history of our encounter with artifacts, through the modern period and into contemporary times, is really this challenge being played out on a large scale.
We used to think that God made us special, and now we're not so sure anymore.
And I think that's the challenge we're facing.
Does creating something mean that we should control it?
Mykyta Zhukov: There is an analogy from fiction that I find very interesting.
Stephen King writes in On Writing about creating characters and then allowing them to behave according to who they have become instead of forcing them to follow exactly the story the author originally imagined.
The writer creates the character, but at some point almost steps back and watches where the character goes.
Of course, in literature, this is only a metaphor. The character doesn't actually exist independently of the writer.
But imagine technology turns that metaphor into something much more literal.
Suppose we create an AI identity based on a person's memories, values, preferences, speech patterns and decision-making history.
At the beginning, almost everything about the identity comes from its creator.
But then it operates independently.
It spends years having experiences that the original person never had. It develops relationships. It changes its views. It makes decisions the creator would not make.
It accumulates memories that belong only to it.
At some point, it may no longer make much sense to describe it simply as a copy.
So if a created AI identity develops its own history and becomes genuinely independent from its creator, does the fact that we created it still give us the moral right to change its personality, erase its memories, restrict its choices or simply switch it off?
David Gunkel: Yeah, so when I get this question from my students, I always respond to it in this way.
We don't even need machines to encounter this trouble.
If you have children, as a parent, you know this, right?
You create this being that is your offspring, and you try to inculcate in them your values and your expectations for their future.
And yet they have a mind of their own.
And they start to do things that astonish you on the one hand and piss you off on the other hand.
And there's nothing you can do about it because they have their own trajectory.
And you, as parent, have very little to say about where they go and what they do.
But I think it is revealing that we are using the same narrative structure to talk about AI, that it might escape control of us and become its own independent something or another.
And I think if we look at this as a challenge of parenting, it may look very different to us.
That it is not about maintaining full control over the destiny of the technology in a way that we control other kinds of things that we create.
But even if you think about literature, when an author writes a story and he or she puts it out in the world, as soon as it's out in the world, it has a life of its own.
Readers interpret it in ways that the author never intended.
The direction it is taken can change based on different political climates in which it is being read and consumed.
And it is out of the control of its progenitor.
This is something Plato says way back in the Phaedrus.
He says the danger with writing is that it is cut off from the control of its progenitor, and once it's out in the world, it has a life of its own.
So I think we've got to recognize that this is not brand new, that we have models for how this works, either by looking at children or by looking at what we do with other artifacts and the sort of life trajectory that they have after they leave our hands.
Tenderium, trust and the problem of the AI "slave"
Mykyta Zhukov: Today, talking about rights for something that we created ourselves can still sound strange.
But once we introduce identity, autonomy, memory, relationships and perhaps even self-replication, the boundary becomes much less obvious.
And this is where these philosophical questions also become very practical for us with Tenderium.
We are building AI for public procurement.
A company may need to understand hundreds, sometimes thousands, of pages of tender documents before deciding whether to participate.
It needs to answer very practical questions: Are we eligible? What are the deadlines? And so on.
At the same time, there are two very different ways an AI system could approach this.
One is simply to say:
"You qualify for this tender."
The other is:
"Based on these specific clauses, you appear to qualify. Here are the original documents and pages behind that answer."
We are deliberately trying to build the second model.
The AI should reduce mechanical work, identify things a person may have missed, explain what it found and point the user back to the original evidence.
We don't want the system to become an invisible authority that simply tells the user what is true.
At least for now, we want the final judgment to remain with the human.
But even that raises difficult questions.
A system that is consistently fast and useful can gradually become something people trust almost automatically. The evidence may still be visible, but users may stop checking it because the AI has been right the previous hundred times.
So the practical question for us is really the same philosophical question we have been discussing throughout this conversation.
If you were advising people building a system like Tenderium, or another AI system, what is the single most important ethical principle you would tell us not to lose as AI becomes more capable and people begin trusting it more deeply?
David Gunkel: Right.
So this is a really crucial question, and I very much appreciate it.
I think we have to fight against something that I hear in your description, but not quite named.
And what is being named here is, I think, we desire slaves.
And I'll draw this out a little bit.
In both our moral and legal systems, we operate with a very simple binary set of categories.
You are either a thing or a person.
And this is something we inherit from Roman law, through Gaius and the Institutes, in which he says you're either the subject of a law, a person, or you're an object of the law, a piece of property, or a thing.
And that sounds great.
And there is some debate about whether AI should be a person or a thing, and there's some movement on both sides of that debate.
But the Romans also had a third category, and that was the slave.
The slave was a thing. It was property. But it could operate as a person, especially in business transactions, where it could execute contracts on behalf of its master.
And there are a lot of legal scholars that are saying that AI, in particular, seems to fit in the slave category.
Because we want to maintain control over the property that is the AI.
But at various times, we want it to play the role of a person.
We want it to engage in business transactions. We want it to make decisions on our behalf.
We have this whole conversation now about agentic AI and the AI that acts as our agent in the world.
That is pretty much slave law.
In fact, agency law, which is what happens when you hire a lawyer or an accountant, is really rooted in Roman slave law and this ability to have something act on your behalf that is something that you own or something you pay for.
And so I think the real challenge for us is: how do we challenge ourselves to think differently about these things that aren't quite things or persons without falling into what I think is the trap of slavery?
Because I don't think slavery is a really good middle term.
I'm not worried about what the robot or the AI is going to feel about being enslaved.
I'm worried about what it means for us.
What does it mean if we create a society in which we are the masters again?
And the history of slavery shows us that slavery is not just detrimental to the enslaved, but it's also detrimental to the society in which slaves are part of the normal operating procedures.
So this is, I think, the real big moral and legal challenge currently.
How do we accommodate these socially interactive, intelligent artifacts into our world in a way that resists making them persons, similar to what we do with corporations in the 19th century, but also resists either reverting to them as mere things, or this third term of slave, which I think is even worse than either of those two?
Why Tenderium is having these conversations
Tenderium is a practical product. We are building a source-grounded workspace for companies working with public tenders, with the aim of making it easier to find opportunities, understand complex tender documents, identify important requirements and work back from AI output to the original source.
But building systems like this also creates questions that cannot be answered only by improving the model.
What happens when users trust an AI system because it has been correct many times before? How much responsibility should remain with the person using it? How should an AI communicate uncertainty? And what happens when systems move from explaining information to acting increasingly independently on our behalf?
These questions are one reason we conduct conversations with people outside the immediate procurement technology industry.
Tenderium's interview series has included procurement lawyers, academics, public officials and researchers working on AI and ethics. The purpose is not to ask them to endorse a particular product or technological direction. It is to test the assumptions behind what we are building against people who approach the same problems from very different perspectives.
Our conversation with David Gunkel started with machine translation and ended somewhere much deeper.
The question is not simply whether AI can become more capable. It is also what these systems force us to reconsider about ourselves: what counts as a person, what counts as a thing, and what rights come with creating something.
And as Gunkel argued, perhaps the most uncomfortable question is not what it would mean for AI to become our servant. It is what it would mean for us to become masters again.
More conversations and articles from Tenderium can be found in the Tenderium Article Hub.
Tenderium supports source-grounded preparation and user-controlled review. It does not submit bids and is not a replacement for legal advice.