Structo talks to Oscar Schwartz

“If there seems to be a lack of coherence in a poem, people are more likely to believe that it was written by a computer.”

This interview first appeared in issue 17, published February 2017.

Oscar Schwartz is a PhD candidate at RMIT, the Royal Melbourne Institute of Technology, where he’s been working on the research question ‘Can a computer write poetry?’ All of those thoughts and questions that flooded into your mind just now? That’s why I wanted to talk to Oscar. — Euan Monaghan

What is the Literary Turing Test?

The Literary Turing Test, which we call Bot or Not, is a project that a friend – Benjamin Laird, who is also a PhD student at RMIT – and I made in 2014. You go on the website and you’re presented with a poem with a title but with no author, then you read the poem and you have two options at the bottom. You can select whether it was written by a Bot: that is, a computer; or whether it was written by Not, which is a human. We collect the responses, and so once you click, you’re told whether you were incorrect or correct, and also the percentage of other people who thought it was a Bot or a Not. We also have a leader board with four different categories: the most human-like human poems, the most computer-like computer poems, the most computer-like human poems and the most human-like computer poems. And those that are the highest ranking in the most human-like computer poems purportedly pass the Literary Turing Test. That is, they can fool a certain percentage of people into believing they were written by a human when in fact they were written by a computer. There are some [poems] that are capable of fooling 66 percent of people into thinking they were written by a human when they were written by a computer. Of course this whole Bot or Not experiment is derived from Alan Turing’s test for computer intelligence, which says if you have a conversation and you can’t tell if [your conversational partner] is a human or a computer, then the computer can be said to be intelligent.

Are there traits which people mark as being particularly human-like or particularly Bot-like?

That’s where it starts to get complicated, because it’s hard to formalize aesthetics. That’s what so much of my project has been about: whether you can or can’t formalize aesthetics. To take a simple view: yes, there are certain traits in poems that will encourage people to select them as human. If they rhyme, if they have a theme that involves death and love or suffering. Generally, women score higher in humanness than men; and late nineteenth century women poets, in particular writing sentimentalist poetry, were strongly identified as being human. Computerness is easier to explain: it’s randomness. If there seems to be a lack of coherence in a poem, people are more likely to believe that it was written by a computer.

Where did the Bot poems come from? Do you code?

Not from my own code; I’m not a programmer, and certainly when I started this I had basically zero programming ability. They come from a variety of sources. I emailed a number of computer poets and asked them if they would give me unpublished work. There are also a lot of free generators online. I used these and took the [resulting] poems, crediting who made it, et cetera. I also came up with a few of my own little methods that didn’t involve writing code but what they produced was nonetheless what I would define as a computational poem. For instance, if you put a sentence into Google Translate and rotate it through 150 different languages and permutations, then it degrades. That passes under my definition of what a computational poem is, so I did a few of them as well just to mix it up a bit.

Torrential Bed


It loves like a serenity around the peace
a shoulder and a hand
carrying the thicket?
A wind of smooth stones!
From her finger and her mouth magnify
veins of the earth.
The romantic wells rejected.


And you perservere like a bed.


Written by Poetry Generator
Programmed by Zackary Scholl

So a variety of different methods, then. Do you know the generation mechanisms the other generators used? Machine learning?

There are a variety of different methods. Some are simple cut-up methods: cut up and collaging. [With others] there’s a certain dictionary and set of transition rules and language rules that are coded into the computer. It then generates sentences according to those rules and within its limited dictionary. Then there are more complicated algorithms. I didn’t use any in Bot or Not that involve machine learning, but there are people starting to experiment with that. What is common is the use of Markov chains or what’s called ‘one-gram generation’. Let’s say I show the software the entire corpus of Shakespeare’s poetry. It will create a statistical model of the language; the regularity with which certain words appear in this corpus. If the computer begins with the word ‘I’, it would ask statistically: what is the most probable next word that Shakespeare would have used? The next word might be ‘shall’, and it will ask: what is, statistically, after ‘shall’, most common? That would be an example of one-gram generation – it works off one word. But it can get to bi-gram generation or nine-gram generation where it looks at the nine preceding words and then asks: if I have, “shall I compare thee to a summer’s”, what is the most likely next word? And it would be “day”. This is a really interesting trade-off. You don’t want it to emulate precisely what’s happened previously, nor do you want it to degrade Shakespeare entirely so it just sounds like random language. You try and find a sweet spot in the middle where it’s Shakespearesque, but it’s something new in Shakespeare’s voice.

There are some strict rhyme schemes which are almost as generative.

Absolutely. That’s why I’ve always been surprised that rhyming poetry does well in “humanness” levels on Bot or Not. Because rhyme has got nothing to do with whether a computer can understand what it’s writing about or whether it actually has been in love. Rhyme is very easy to reduce formally or abstractly, or to give a definition of, in an algorithm. That was some of the trouble I had in my research. Suddenly I had all this data and I realized that the methods that the algorithms were using could be done – all of them could be done – by humans on paper following a set of instructions. It would just take a lot longer.

An algorithm is just a set of instructions.

Exactly. So it kind of made the Bot or Not opposition suddenly seem irrelevant.

Have any of the generated poems surprised you?

Yes, there have been some poems that I have chuckled at or been surprised by. It’s always been in some sense a conceptual project because I’m enjoying not only the output itself but knowledge of how it was made. I think what’s interesting for me about computer poetry isn’t the poetry itself but computer poetics, or computational poetics. That is, the way something is made. I think that’s what’s super-interesting about this area.

Your background is English literature?

Yes, I studied English lit. and also philosophy, and within philosophy I studied logic. I did kind of have a background. I mean, I had a super-steep learning curve in terms of understanding how computers work, and I’m still hardly a programmer at all, but it’s been fun to learn.

Has it changed the way you write your own poetry?

When I first started the project it did, very much so. I started writing poetry for friends on the internet at a similar time to when I was starting my PhD. People in this community were writing highly sentimental poetry that wasn’t formally tight at all in any way. It was all about communicating everyday experiences. I think the connection there is that I was experimenting with ideas of how you write the most human poem possible to someone you’ve never met, who you only know via the internet, in order to authenticate your humanness when you could well be an algorithm.

All very Blade Runner.

[Laughs] Yeah, exactly.

And has it changed the way you look at poetry and literature more generally?

Well, it’s completely changed the way that I look at … not all literature or not all poetry, but my research has ended up being historical. The argument in my thesis is that while computational poetics seems like a new activity and something that is almost science fiction – Blade Runner as you say – it’s actually very ancient. It’s an ancient form of literary practice going back to ancient divinatory practices that involved chance-based generation of symbols and poetic language and interpretation. It has changed my understanding of literature in terms of broadening my understanding of literary practice throughout history.

Can you give an example?

There’s a type of Kabbalistic practice called Abulafia meditation. Abraham Abulafia was a Kabbalist who lived in Barcelona in the thirteenth century. He studied the oldest book of the Kabbalah, which is called Sefer Yetzirah, which means Book of Creation. The Book of Creation describes the way that God purportedly invented the universe. The way that it describes that, obliquely, is that the twenty-two letters of the Hebrew alphabet existed and God commuted them in all possible combinations and from that permutation the world was formed. Abulafia – and people before him as well – came up with methods for letter permutation in order to understand God’s ways and God’s creative capacities more thoroughly. He would sit in a room for days on end and commute letters of God’s name and create reams and reams of essentially computational poetry. Through the process he would gain prophetic knowledge of God’s creative powers. That’s one example, and also it interestingly raises the same types of philosophical and linguistic problems. Literary theoretical problems. Who is responsible for the writings that Abulafia created? Are they in the methods or was it him? How is one supposed to read them and to whom does one attribute agency while they are reading? Is it to the method or is it, you know… if you’re reading these permutations and you’re taking meaning from them is it just because you – the reader – have agency, or because Abulafia had agency? It raises a lot of what seem to be contemporary literary theoretical questions.

There’s a lot of interesting work in the area of generative literature right now. The Infinite Monkey Generator has pretty much written Shakespeare.

The infinite monkey theorem is something that I’ve had to engage with in my research as well. For me the underlying tension and what makes all these questions really interesting are questions of agency. Where does the agency lie in the poetic act? Is it with the author, is it in the text itself, is it with the reader, or is it somehow an interaction between these three things? When you introduce a non-human agent into the poetic act, and give it poetic agency, we have a certain resistance to it. I think it’s that tension between the human and the non-human, and poetic agency, which makes these ideas interesting.

Can you define agency in this context?

Agency is just the capacity to make a difference, to be involved in activity and produce consequences.

How do you think literary work in the generative arts compares to progress in other fields?

I think music is the place where generative methods flourish and there seems to be least resistance and more joy in the act than kind of an, ‘ah, I tricked you’ kind of thing. There’s a long of history of generative music. Mozart used dice to compose some of his pieces by chance. I think because music is not representational people have less resistance in allowing non-human elements into it. Whereas with visual arts, if it’s not abstract, if it’s representational and mimetic of nature, then people have more resistance.