Structo talks to Tivon Rice

“At times a strictly Neural Storyteller voice peeks through, and it’s a very strange one”

This interview first appeared in issue 19, published April 2019.

Find videos and more at tivonrice.com

Last September I found myself in a former anatomical theatre nestled at the top of a 15th-century tower in Amsterdam. It was a suitably strange place to host a workshop on worldbuilding. Forming part of the Coded Matter(s) Terra Fiction series, over two days a motley group of artists, writers, coders, designers, scientists and others were tasked with constructing narratives of an imagined eco-future. One of the tutors was Tivon Rice, an artist based between the Netherlands and the USA. He talked about his work on algorithmic storytelling and we had the chance to experience one of his recent pieces in virtual reality. My notes from that day had the word ‘interview’ scrawled in the margin in all-caps. Two exclamation marks. We talked a few months later at his studio in Den Haag. — Euan

The workshop was all about developing a coherent environment of place and of people behind a story. And it seems that, from a different point of view, that’s what you’re doing in your work. Is that fair?

Yeah, I’d say so. I mean, I come from a visual arts background, and my work tries to figure out how things either emerge or disappear from our visual field. And that’s a very broad way of looking at images, contemporary images and those sorts of things; but, at some point, an artist can either document the things they see emerging or receding around them, or can start to imbue those things with some sense of narrative as well, narrative that colours the things that you see happening out there in the world. It seems to me that all artists are worldbuilders in some sense. And recently I’ve thought about this a bit more consciously in my work—about how to bring fiction, or how to bring speculative realities, into my work.

Can you describe the piece that you brought with you to the workshop?

Sure. It began before I arrived here in the Netherlands about a year and a half ago. I was spending a lot of time on Google Earth, like I imagine many people do, to virtually explore a place before they actually go there. And you get this top-down view of the city you’re about to live in, the space you’re about to inhabit, the places you might explore. And that gives you one particular, virtual sense of a place, but you never really have a true sense of the place until you touch down, walk around and get your eyeballs on an environment. So, when I got here, I biked out to the dunes and saw this row of Atlantic wall bunkers that are still there, these sort of persistent, accidental monuments.
And of course, you think of Paul Virilio’s Bunker Archaeology and these sort of typologies of those concrete structures. But I was struck with this idea… or this sense, this feeling, that I wasn’t really seeing them directly or I wasn’t seeing all the political and historical implications around them. I was seeing them more through the lenses of cinema, fiction, and all the different stories that had been built around those structures, fantastic or truthful or what have you.

Again, I was struck with the sensation that I’m seeing these for the first time, but I’m really not seeing them for the first time. That feeling stuck with me and I was trying to figure out—in what other ways does that type of sensation manifest itself in our contemporary, media-filled environment?
So, it made sense for me to reimagine those landscapes through the lens of another media. Literature was what came to mind, and specifically J.G. Ballard’s landscapes. These are quite often defined by physical materials: concrete structures dominating a character’s existence; or the post-war landscapes which appeared in much of his writing; or natural disasters, which certainly factored into his work. That’s where I made this connection between those Atlantic wall structures, my perception of them, and the writing of J.G. Ballard, which has certainly influenced the way I think about them, or the way my natural vision of those structures is processed in my mind. So that was the starting point. Of course, there’s a lot more to what the artwork actually is, but that’s how I came to it.

I’m going to attempt to explain how it felt to be in that. It’s a [virtual reality] in the sense that you have a VR headset and some headphones, and you have freedom of head movement, and you’re very slowly taken on a walk around one of these bunkers, at least in the section that I experienced. It’s at a pace which, I think you mentioned at the time, is slow enough that the people who do get a little bit nauseous; it’s a pace which they can deal with. The bunkers are quite big and monolithic. And while you’re going around you can hear ambient sounds and then it’s kind of punctuated by what you would assume is a reading from J.G. Ballard. It’s very atmospheric. But it’s not actually the writing of J.G. Ballard in the sense that you might think. Perhaps you can give a brief summary of what is actually happening there.

Sure. It is indeed Ballardian because it is the output of a recurrent neural network and a machine learning system that was trained on the complete works of J.G. Ballard. I plugged in the raw text of everything he ever wrote into this machine learning system and it creates a model that tries to emulate his particular vocabulary, his punctuation, his writing style, and where he goes with certain descriptive images and these sorts of things. That’s a starting point for the system.

The system was actually a master’s thesis project at the University of Toronto created by one of my collaborators named Yukun Zhu. He and I were introduced through the Google Artists and Machine Intelligence project, that was originally set up in Seattle and aimed to connect working artists, musicians, filmmakers, writers, with those who have a more technical understanding of how these AI things work. So that’s sort of how I got started with it.
But the process of actually using the system is quite interesting as well, and I’ve been able to draw a lot of connections with what I described earlier about our own natural perception, and relating the things that we see to the things that we know. That’s kind of how the system works, in a way. So, once it’s been trained on—whatever—the complete writings of Ballard, for example, the next step is to show the system an image. And I of course showed it images of the bunkers.

I flew a drone over each of these five bunkers, and photographed about 150 images of each one, from all these different angles. And back at my studio, I reconstructed 3D models, or photogrammetric scans, of each of the structures. The textured models look quite photorealistic, but they are scenes that I can then dive into and turn around and zoom into on the computer, almost like I can make cinematographic decisions about it after I’ve left the beach. And those were the images that I showed to the text-generating software. Once that system, called Neural Storyteller, sees those images, it creates some very simple captions. The things that we would maybe call our first impressions of a place. What are the materials I’m seeing? What is the shape of this scene? “A concrete structure surrounded by grass”, or “the ocean ends next to an old building”, or something like that. It first generates these very essential captions or ideas about the image. And then the neural network starts to work on those very short segments, turning them into longer, more hallucinatory imaginations of what might’ve happened or who might be residing, what sort of narrative might be unfolding in these places. That’s the part of the system which becomes biased by the J.G. Ballard machine learning model. That’s where it starts to skew the language towards phrases that he might have used.

Something I had briefly talked about in the chat with Ken Lu was the idea of writers working with algorithms, whatever that encompasses. And you didn’t just run the machine and press print and then put it up in a gallery. You worked actively with the outcomes from the system in terms of editing and rearranging. What was that process like?

I think this is an essential decision. And you mentioned this idea of just pushing “print” and printing whatever comes out, the good or the bad or the glitchy. That’s certainly one strategy, especially since artists and programmers are just understanding the shape of these algorithms, of these systems. I think this kind of proof-of-concept or demonstration of the system is quite valid—to let these things run in real-time and see what comes out. And you might look at the work of Ross Goodwin who’s another Artist and Machine Intelligence collaborator working with these types of real-time systems. He lets his cameras look out at the world and then, on a little ticker tape, a machine intelligence stream of consciousness tells us what it sees.

That’s one strategy with these systems. But, in my work, I’m interested in understanding the impression that these texts leave me with, or the impression that a visual scene leaves me with, or the impression that the output from an artificial intelligence system leaves me with. I felt that this was a very important moment for me to reinsert myself into the loop, creatively or artistically. This tends to be my current strategy. So, when I read the output from the system, I can see how Ballard is influencing it. I know Ballard very well, to the point that when I went to the bunkers for the first time, I kind of self-narrated my experience in an internal Ballardian voice. Now, when I see that tone of writing come through the output from Neural Storyteller, I reach for the highlighter.

And then when I see it not working, I put that in another bin—a possibly interesting bin of texts that emerge when the system starts to repeat itself or it starts to lose its train of thought. I categorize these types of outputs and, depending on what I want my final project to be, I then choose the things that are extremely Ballardian or these things that are interestingly glitchy or these moments where it repeats itself. I think that the errors are also interesting and create their own vocabulary parallel to that of Ballard’s.

Along these lines, I also start to see where the system really starts to show itself, where Neural Storyteller shows itself. I’ve done a number of other projects with this system where it wasn’t trained on Ballard, where it was public documents or a different author. And, of course, you can compare the outputs and see where it’s much more like Ballard or much more like Philip K. Dick or much more like city planning documents, but at times a strictly Neural Storyteller voice peeks through, and it’s a very strange one.

What is that voice?

That voice was trained on something like 10,000 novels from 16 different genres. At the University of Toronto, where it was developed, they created this large book archive that became the very initial training set for Neural Storyteller just so the algorithm knows how English operates, how sentences are structured, how punctuation should work, and this large underlying data set is apparently necessary to have some cohesive output. Then you have this much smaller set, the Ballard set, the training bias, that can throw it off in another direction or skew it towards that particular author’s style. But the underlying style peeks through every once in a while. If describing a building, it will use these same turns of phrase about approaching a structure or going underneath a structure. I only started to see these patterns by using this system a lot, by giving it a lot of images and watching it grow with different training sets. In that sense, it starts to feel like watching a child learn how to speak on its own. You give it things to describe, you see how it responds, you hear it repeat itself, you watch it fail.

Coming back to my artistic relationship with these systems: it’s a combination of understanding the input that I’ve given it—an image, a large set of text, or some other data—and then trying to make sense of the output on many different levels. Understanding the output as a response to the inputs, or as a function of the underlying structure of the algorithm and then, ultimately, trying to bend it towards what I want the tone of my artwork to be. Do I want the tone to be cryptic and sort of abstract or do I want it to very specifically talk about some invisible subject, some character that seems to be emerging from all this text?
structo: And for the artwork we talked about, the Ballard, can you break down where your time was spent in making it? Or were you working on more than one thing at a time?
rice: Yeah, it was all quite parallel. I’d say that it’s very easy to go out and fly a drone; it does all the work for you, in many ways. So that was really done in one day and the photogrammetric reconstruction of the bunkers was done in a couple of days. After that, the visual content was established. Then, the Neural Storyteller process is much more arduous. It’s a much longer process, for a number of different reasons. I mean, the machine that’s running [the models] takes hours to start up, to technically embed these language models into the system.

But I think more time consuming than that technical aspect is parsing the text afterwards and understanding what sort of patterns and what sort of atmosphere is being generated in the language. What will the viewer’s experience of these texts be? I didn’t know this was going to be a VR thing from the beginning. I thought it might be an installation: one video projection, and a room filled with the “voice of Ballard” creating a kind of immersion through the relationship between images, space, and sound.

But again, I came back to the experience that I had when encountering those bunkers for the first time. My body was there, I looked at them, but my head was back in all of the different media, all the different movies that I had seen those things represented in. And I thought that layer of virtuality that was going on in my very real encountering of the bunkers was something to shine a light on, or something to reinforce through the display of the work.
It was my first VR-based artwork. I’m still very critical of the medium for both personal and, I don’t know, institutional reasons. But I think it’s interesting to try to sort of poke and prod at media like that and figure out where it allows for some critical reflection on the images we’re seeing and the experience we’re having with it.

Your most recent piece was a physical sculpture combined with observations from the storyteller. The idea that you can get a piece of code to speak in the voice of a dead author is an interesting, complex one. Do you ever think that that the author should be left alone?

Yeah, like how algorithms trained on Rembrandt paintings can now create new, passable Rembrandts. In my work, I’ve specifically chosen authors who either directly wrote about human-machine hybrids, or who I think would have been conceptually interested in this kind of digital afterlife: Ballard, Burroughs…

You mentioned Philip K. Dick before.

Yeah. Totally. So, these impassioned or critical arguments about an author’s work or a painter’s work being assumed by one of these systems are good conversation points when discussing AI. They help us wrap our heads around why we care very deeply about human creativity and originality. But there are also other systems—political, economic, social—that machine learning and artificial intelligence have their fingers in that are maybe a lot more problematic or more troubling.

Just a little bit.

I’m not trying to diminish your question or anything like that, but I think artists can work with the problematics of these systems in a way that’s thought provoking, but also hopefully creative at the same time. [At the Waag workshop] Klaas Kuitenbrouwer was talking about his speculations around terramorphing and workshops he’s conducted on that topic. And he said there was always this danger of participants falling into Black Mirror territory time and time again, because we think of these systems as being so insidious and so pervasive and so potentially world changing, in a bad way. But he described how he tried to skew his participants towards a white mirror model. How can we have these same speculations about the future through a more creative or positive lens? Not to distract ourselves from the importance of criticality, but as a counterpoint to that.

Criticality needs exploration otherwise you have no data, right?

If you ask the question: what can we do with these things? The potential to go in a number of directions is worthy of exploration. In one case, I try to show these systems images that I believe those authors would have had something to say about. And that’s probably cherry-picking or a very easy thing to do. I show Ballard images of World War II bunkers because I know he wrote about those. And I’m trying to tease out those things that I know are there. The second project you talked about: that became a video installation, focused on a high-rise building just north of Den Haag. [Ballard] wrote a lot about those types of architectures. So again, I’m trying to tease those ghosts out of the larger corpus of his texts and see what comes about.

When I’ve done workshops and we brainstorm about what other language models would be interesting to train, I started with a kind of anonymous set of corpora, which focused on Seattle city planners and public comments on urban development. They had a different type of character because the data was aggregated and it didn’t point to one particular person. And then we think about specific styles: what if we just choose science fiction or off-the-shelf romance schlock or something like that, regardless of author, what would that evoke? Again, sort of these aggregated bits of language.

Then as you start getting towards specific authors, it becomes, I think, far more about our relationship with their body of work—knowing what to expect from it, and knowing how to identify the unexpected outputs that might come out of it. Then, an entirely different category would be living authors and at that point I think you are probably in territory where you should contact that person and propose working with them and the software in an agreed-upon collaborative mode.

I feel like I’m collaborating with these dead authors in some way and I think this is exactly what any artist working with large sets of specific data should be asking themselves. What am I really collaborating with, or who am I really building a relationship with?

The same researchers and curators at Google who invited me into this collaboration (and supported and funded it) put on at least two or three of these very high-gravity workshops and presentations. The first was in San Francisco about three years ago called “Music, Art & Machine Intelligence”—a two-day symposium about the technical and epistemological implications of these technologies. And the speakers were outstanding. Timothy Morton led a panel discussion focused on the critical potential of these technologies, Ian Cheng was there talking about his real-time simulations, that was wonderful.

It’s an interesting time to be working with these tools.

Yeah. Not being a writer myself and not having a formal background in using language to put a world together, I normally use physical materials and visual components to build the worlds in my installations or video works. It is very empowering to get a little boost from an artificial agent, in a way, to get pushed towards a different medium. It was almost like I could start to see the logic of the media that I have been using for a while—the logic of the visual media, the time-based media that I’ve used in my work in an inter-textual relationship with literature. Maybe I create images based on things that I’ve read or films that I’ve seen; and now, with this system, I’m generating texts by feeding it images or by cultivating images that I think might evoke something important or interesting. It doesn’t make me a good writer. The system doesn’t inherently make me a writer, but it brings me to these inter-textual connections between images, sounds, the voice, writing, literature, and cinema. And that’s something I’m just figuring out through this work.

It was very nice to have the little thought experiment that

Pippa Goldschmidt brought to the workshop. Her tool was to give us a fragment of a sentence and then we would write for a minute just off the top of our head, to build upon that fragment. And that felt to me a lot like Neural Storyteller, which begins with a very simple phrase and then it hallucinates these much longer sentences.

But then it reminded me of what I was doing when editing the Neural Storyteller text as well. I’m given a full page of potential gold and potential garbage from the system. Then it’s my own internal process that decides what of that is useful, what of the gold is useful and also what of the really glitchy, problematic language might be potentially useful.

So again, thinking of this as a tool for an artist who may not have worked in that particular medium: it launched my interest in working directly with language.

It almost gives you a structure to work within or to start from.

Yeah. And it also felt like it was demonstrating how other pairings of media might work as well. In some sort of higher-level observation of it, you’re really deciding about what an input should be, how that input should be processed, and how you relate to the output. And that could be anything… that doesn’t have to be turning images into text, that could be any number of different inputs, processes, and outputs.

I remember studying, in my visual arts background, the cut-up technique of Brion Gysin and how Burroughs used this writing strategy. I studied this not as a writer and focused less on the actual writing that came of it, but rather the strategy of it: studying this approach to input, process, and output that could potentially be applied, to images, or the way a film is put together in time.

So, using this as a way of studying strategic relationships with content is maybe a far more essential thing that I’m doing rather than specifically trying to emulate a writer or something like that.

Are you working on your next thing or are you taking a breath?

I’m still assessing the second Ballard project, which I think, in a narrative sense, is more successful than the bunkers project. It’s trying to put an invisible character in a space as opposed to reciting all these very ambient, very atmospheric observations about a space.
So, over the course of a year, the former Dutch statistics building was being demolished right along the railroad. Anybody travelling from here to Leiden, or Schiphol or Amsterdam got to observe this very slow cinema, the observable erosion of this building that took over a year to accomplish.

It was actually really nice to talk with people at the project’s opening. Because it was a very recognizable building, dozens of people come up to me like “That’s the building that I’ve been watching from the train!” It was a very bizarre visual situation that stood out for me, like watching someone trying to tear down a building with a plastic spoon. And the cinematic nature of a windowed frame in the train car, moving past this scene made me think—that is a very Ballardian scene, this is like the aftermath of his novel Highrise. So yeah, I’m still processing how it turned out. This may be the last Ballard piece, this may be the last Neural Storyteller piece because I think there are newer language-based neural networks that might start to give me different, more diverse output. But certainly, I don’t think it’s my last piece that tries to take data as an input and see what a machine learning system can give me as an output.

It seems like a great deal of fun as well.

Yeah, it is. And it’s always good for me to see how people experience it as well. I can tell myself I’ve got a pretty cohesive strategy for making these works, but ultimately it comes back to how people perceive them. If I show a new artwork that used this technology but let’s say I don’t foreground the tech—the “machine learning” or the “artificial intelligence”—if it was well-received and successful, does it matter or is it really important to say in the by-line or on my materials lists: neon, video, computer-generated text? What switch does my use of AI turn in the viewer’s head to perceive the piece differently? Do they perceive and engage with it differently because it turns their imaginations on to what’s possible through these emerging systems? Or does the tech detract from the work, if it was a strong narrative in its own right?