JARON LANIER: (VOICEOVER) I’m worried about where we are right now. I just feel like our present Internet is destroying societies and democracies and economies. I think it’s bringing down civilization. It’s bad. We really screwed this thing up.

[MUSIC]

KEVIN SCOTT: Hi, everyone. Welcome to Behind the Tech. I’m your host, Kevin Scott, Chief Technology Officer for Microsoft.

In this podcast, we’re going to get behind the tech. We’ll talk with some of the people who have made our modern tech world possible and understand what motivated them to create what they did. So join me to maybe learn a little bit about the history of computing and get a few behind-the-scenes insights into what’s happening today. Stick around.

[MUSIC]

CHRISTINA WARREN: Hello, and welcome to the show. I’m Christina Warren, Senior Cloud Advocate at Microsoft.

KEVIN SCOTT: And I’m Kevin Scott.

CHRISTINA WARREN: And today we have an amazing guest.

KEVIN SCOTT: Indeed, we do.

CHRISTINA WARREN: Okay, but before we hear from him, I wanted to ask you a question first, Kevin. All right, so you have a ton of interests, you’re into cooking and photography and there’s a long list of stuff.

Do you ever feel like there isn’t enough time in the day to do all the stuff that you want to do, like, you know, to pursue your passions?

KEVIN SCOTT: Yeah, this is sort of the curse of my life. (Laughter.) I wish there were more hours in the day.

CHRISTINA WARREN: Same, but if you could stop and spend time during the day to do just the thing that you’re most passionate about at this moment, what would that be?

KEVIN SCOTT: Yeah, it’s sort of weird – it would definitely be making something and, you know, with me, what it is that I would make changes over time.

So, at one point in time, it might be a dish to share with someone together at the dinner table, at another point in time, it might be a piece of furniture. Right now, interestingly enough, I’m sort of obsessed with machining. So, I would go to my metal shop and fabricate something mechanical.

CHRISTINA WARREN: I love it. I love it. That sounds really cool. I think for me, I don’t know, it would be boring, I’d probably just write, continue, just focus on writing –

KEVIN SCOTT: I think that sounds great.

CHRISTINA WARREN: Yeah.

KEVIN SCOTT: That’s what I should be doing, given that I’m writing a book (laughter) right now. I should like, you know, go into a nice quiet place and try to pound out the last chapter of my book.

CHRISTINA WARREN: Okay, but you know, maybe do some machining, too. Maybe machine something to – machine your own typewriter or something, I don’t know.

KEVIN SCOTT: Yeah, maybe. Maybe.

CHRISTINA WARREN: Well, today’s guest, Jaron Lanier, is kind of beyond belief in regards to the things that he’s accomplished.

Like, how does he find the time? Speaking of not having enough hours in the day.

KEVIN SCOTT: Yeah, I – Jaron is awe-inspiring in the breadth of his intellectual interests and when I first met him, I wasn’t really prepared for the full brunt of his intellect, really. Like, I knew just from things that I read from him and from admiring him like all the way back in the ’90s when you know sort of there was a VR boom happening at the time.

And, like, he was frequently on TV and, you know, sort of like during the rest of the tech boom, sort of conveying the promise and possibility of virtual reality.

But I think the first time I met him, like, I learned that he’s got this, like, crazy musical instrument collection and, like, we share this interest in classical piano. And, like, he is – unlike me, like, I am – I am a rank amateur. And, you know, so spend most of my time listening and very little of my time playing.

Like, Jaron’s an accomplished performer, he’s actually a composer, like, he’s done performance with folks like Phillip Glass. He’s recorded records, he’s been a session musician.

And, like, this is on top of him being, like, one of the best computer scientists in the world. Like, one of the best writers in the world, one of the best, like, philosophical thinkers about our, you know, sort of digital modernity.

You know, and, and, and, like, I’m not even covering everything. So, it’s sort of unbelievable. Like, I’m totally jealous. (Laughter.)

CHRISTINA WARREN: I am, too. I am, too. Okay, so, without further delay, let’s meet our next guest.

[MUSIC]

KEVIN SCOTT: I’m delighted to introduce our guest, Jaron Lanier. Jaron is a scientist, musician and author best known for his work in virtual reality and his advocacy of humanism and sustainable economics in a digital context. His 1980 startup, VPL Research, created the first commercial VR products and introduced avatars, multi-person virtual world experiences and prototypes of major VR applications such as surgical simulation. Some say his mind is as boundless as the Internet. Welcome, Jaron.

JARON LANIER: Oh, hey, I am so happy to be here. Thank you for having me here.

KEVIN SCOTT: So let’s talk a little bit about your history. You are, I say this to many of my guests because I have interesting guests, but you have one of the most interesting careers and like set of life experiences of anyone I’ve ever met. So like let’s start with you as a kid. Tell us a little bit about how you grew up.

JARON LANIER: Oh, my. Okay, well, my parents were both refugees from antisemitic violence. My mother was a Holocaust survivor. My dad’s family was mostly wiped out in pogroms in Ukraine.

KEVIN SCOTT: And your mother like was in a concentration camp.

JARON LANIER: Yeah. Yeah. She was taken at 13. Well, we can talk about the stories, but they’re truly just horrible, beyond understanding. And my parents met in the Bohemian New York of the 1950s and I was born in ‘60 in Harlem Hospital. And they immediately fled. And I think the idea, although I never really got a clear answer from them, is that they wanted to be as far from civilization as possible but not so far that they weren’t next to a good university.

So they ended up in Southern New Mexico.

JARON LANIER: And I would catch a bus across the border every day because there were better schools in Mexico.

KEVIN SCOTT: And what was the strength of the schools back then, because obviously you had this career in science and music and technology. Did that sort of get sparked there?

JARON LANIER: Well, my mom came from an educated family in Vienna. She was a protégé pianist and that sort of thing. And she had very high standards and when I was a kid the Mexico school system was about two years ahead for any given grade in terms of its curriculum compared to the Texas or New Mexico schools. So a lot of the families that cared about education sent their kids across the border to Mexico. And this was like just one continuous place. It wasn’t – there wasn’t like some big prison wall between the two. It just was a continuation.

So she died in a car accident. I was about nine. And many years later it turned out that there had been a mechanical flaw in that model of car that’s likely to have been the cause.

So there’s a layer of tragedy in that in that she had deliberately gotten a Volkswagen out of a sense of wanting to find reconciliation with humanity.

KEVIN SCOTT: Right.

JARON LANIER: So this was of course devasting, but devasting on some levels that were unusual in that era because she was also the family breadwinner. My dad was always kind of the multi-career, slightly weird artist type. He had all these little careers. He was – he designed windows for Macys and was an architect for a while and he was a science fact writer for Hugo Gernsback.

KEVIN SCOTT: How interesting.

JARON LANIER: Yeah, he would write the science fact columns in Fantastic and Amazing and Astounding in the ’50s. So he knew all the Golden Age science fiction writers. He was in their circle of friends. But none of that was particularly lucrative for him. And my mom was kind of a systems thinker and figured out how to play the stock market and she would do with phone calls from the desert in New Mexico which nobody did at that time. These days that’s normal. In those days, it was highly innovative and especially for a woman. And I suspect a lot of the people at trading desks in New York didn’t know she was a woman. I think she just figured out ways to do it.

At any rate, we suddenly ended up quite impoverished. Meanwhile, I was hit by my mother’s death very, very, very hard. I think she held me very close because of her background.

KEVIN SCOTT: But you went from there to right around the time you were 12 years old or so you were taking college courses.

JARON LANIER: Well, so what happened afterwards, my dad was kind of backed into a corner. And what he did is he got certified to teach school so he could get a job because it was the only thing he could see that he could do out there. And he bought a piece of cheap desert land and we moved onto the land in tents. And we gradually started building a house. He let me design it, it was this crazy thing with geometry. And it lasted for like 30 years and then collapsed. So don’t let your 11-year-old, I think by that time I was about 11, don’t let your 11-year-old design your house.

KEVIN SCOTT: But it’s somewhat remarkable that an 11-year-old could design a house that would stand for 30 years.

JARON LANIER: Yeah, well, I let my daughter at 11 design part of our new house, but I checked her work. Let’s just say, like I think it’s great, let your kids design the house, but in terms of actually living in the house, check their work.

Anyway, it happened by luck that this place we were in in the desert was the perfect place for me. One of our near neighbors was Clyde Tombaugh who discovered the planet Pluto and was the head of optics research at White Sands Missile Range, and he started showing me how to make telescopes and lenses, which is the background that I used to be able to make virtual reality headsets later on when I needed to be able to do that. So I learned optics as a kid directly from somebody who was as much a master as exists.

KEVIN SCOTT: And had you been a precocious kid before that?

JARON LANIER: My mom had made a demand that I would be so. Like this wasn’t an option. I was informed. You have to have your concert at Carnegie Hall, and you have – I expect a Nobel Prize and not one of those sissy ones in economics or something, like a real one. And like that was kind of the expectation. There wasn’t really negotiating room on that. That was how I was raised.

And I think a lot of Jewish kids of that generation were raised that way. It was compensatory for all that had just happened and there was just this expectation, this is what you’re going to do.

JARON LANIER: I don’t know how different I’ would be if my mom hadn’t pushed me as much. I really just don’t know.

KEVIN SCOTT: So you were 11 or 12 years old when you’re sort of learning about optics and like getting into science, this is in the early ’70s, right?

JARON LANIER: Yeah.

KEVIN SCOTT: And so like the personal computer revolution is like a decade away at this point.

JARON LANIER: Well, yeah, this is before you could make your own computer. This was before the little Altairs.

KEVIN SCOTT: But so were you toying around with like computers at that point, like the big mainframes of the day?

JARON LANIER: So what happened was, in order to design this crazy house, which was a mixture of geodesic domes and these other geometric shapes, I had to learn trigonometry. And so I did. I just forced myself to learn it. And it was really tedious calculating all these angles and lengths for this thing. And I was really interested in computers.

When I was 14 I went to a summer course in chemistry. They had like this chemistry summer camp at the university there at New Mexico State. And that was great and I did all of the things that chemistry professors worry a 14-year-old would do in a well-equipped chemistry lab. And I was directed to do those things in the empty lot next door rather than in the building. And I used to visit it and there were still a few pock marks but I think now it finally has a building on it.

But I learned to make flavors and explosives and I was just fascinated by the geometry of molecules which was a little like the house that I worked on and all that. So the summer came to an end and it just seemed somehow absurd to go to school. So at my age I would have been going to high school. So I never went to high school. I just never – I just skipped it. And I sort of enrolled.

So the question is, how did I get into the college? And I’m not sure. I just sort of signed up for courses and either it’s possible I might have falsified a document or two or they might have forgotten to check. I’m not sure. They had me back to the campus and I think by now whatever happened would be forgiven. But at any rate, I just started attending.

KEVIN SCOTT: And which school was this?

JARON LANIER: This is New Mexico State University. And as it happened –

KEVIN SCOTT: How old were you?

JARON LANIER: Well, I was 14. And it happened by wonderful coincidence that because New Mexico State was supporting the White Sands Missile Range it had one of the earliest good computer science departments, like actually way ahead of a lot of the fancy places, way ahead of some place like Caltech as I learned later. So there were actually good computers, good computer scientists and an excellent math department.

So I was like in heaven. I remember I would like just haunt the basement of the math building programming.

KEVIN SCOTT: And what was the first substantial program that you remember writing?

JARON LANIER: A psychedelic graphics thing. And in those days, so the way you programmed in those days was on decks of cards. And so this is something that’s hard to describe these days. What would happen is you’d have these stacks of cards, and you’d have to take them to this window in a place where someone, a graduate student, would take them and then run your thing and they’d give you another stack of cards that came out.

And one of the things about Southern New Mexico is can get quite windy, so as you approached this cinderblock building with your stack of precious –

KEVIN SCOTT: And order matters.

JARON LANIER: Well, listen, there would be like these clouds of these punch cards just flying everywhere. So there was a certain degree of cross-pollination between people’s programs I suppose, but it was actually – I mean, I remember seeing actual code tornadoes made of punch cards in those days.

KEVIN SCOTT: Wow, that’s incredible.

JARON LANIER: Yeah. So when we talk about cloud computing, we had fast cloud computing back then, low latency. So as the ’70s progressed there started to be a few options for real-time computer graphics and that really turned me on, and we started to be able to do some very simplistic, like even just a biplot or moving like a rectangle or stuff on the screen was still kind of challenging but we could start to do it.

By the way, do you know how the biplot was invented?

KEVIN SCOTT: I think I do, but why don’t you tell the audience.

JARON LANIER: Larry Tessler, one of the original Xerox PARC people invented the biplot, which is just moving a rectangle of pixels around on a screen, and he’d originally done it to control those little color cards that fans hold up at the Stanford stadium. So it actually started as a stadium ritual before it was applied to –

KEVIN SCOTT: That I actually did not know.

JARON LANIER: Yeah.

KEVIN SCOTT: That’s fascinating.

JARON LANIER: I’m the sort of like antiquities department here. So there was a good library at NMSG which kind of became my comfort zone in a way, and you could just like get lost in these stacks of all different kinds of journals and crazy art books and everything. And there I discovered Ivan Sutherland’s work.

And Ivan Sutherland was the founder of computer graphics and the founder of graphical interaction on computers and user interface design with graphics and many other things. He’s kind of the father of a lot of the experience of modernity, and he’s still with us. He’s teaching at a little school in Oregon these days and I keep up with him once and a while.

But he had described computer graphics and there was like this picture of a cube just painted by a computer, and when I was a kid, this was before we could get a computer graphics machine, that happened like a year or two after I arrived, I would run down and people, the strangest mysteries, look at this, you can make images with computers. I’m like so excited. I was like, because there was no Internet, you couldn’t just reach. You had to just attack strangers on the street with weird journal articles.

So Ivan in something like ‘65 had proposed a head track computer graphics headset and he actually built one in ‘69. And that really, really turned me on because ever since my mom had died, I’ve had the sort of feeling of incredible isolation from people. I was super socially awkward and a very weird kid. And I always had this sensation that people were like the stars. You can see the stars, you know there must be interesting things in the stars, but they’re too far away to reach. And people felt that way to me.

And I always imagined that if there was some kind of new medium, some way of sharing dreams maybe, it would be like a starship where you could reach those distant stars. And to me when I read about Ivan’s headset, I thought, okay, if you could network these things, maybe you could have that shared dream thing. So that was how I got into this whole virtual reality whatever it is, this crazy adventure.

KEVIN SCOTT: So like you are I think rightfully so credited with being the father of virtual reality. And I remember like the first time I was aware of you was in the ’90s. I saw you on – like I forget what the name of this – there was a technology network that Leo LaPorte was at and I think Leo was interviewing you. And I was like, oh my god, like this is the most unusual and incredible thing I’ve ever seen. And it was you.

So like when did this whole notion of virtual reality crystalize for you? When did you coin the name?

JARON LANIER: Well, the name – so the deal is, Ivan Sutherland called the thing you saw through the headset the virtual world. And he got that from an art theorist named Susanne Langer who was writing about virtual worlds in the ’40s and ’50s. And I thought, well, if there was a network or a social version of it, a shared virtual world, maybe that would be virtual reality. And so I started like writing little things about it in magazines and all kinds of stuff. And that probably started in like the late ’70s or something.

But I didn’t actually have any way to do it. And so what happened next – and by the way, in those days it was almost impossible to explain to somebody what this was. If you tried to explain how quantum computing works today you get a feeling for what it was like trying to explain virtual reality in the late ’70s or the early ’80s. It’s really hard.

KEVIN SCOTT: My sense was, even like the first time that I saw you being interviewed in the ’90s after the Internet had started to take off with these – like the graphical web browser, that it was still difficult to explain virtual reality. It’s not the easiest thing in the world even now, but like there are far more examples and like there’s a richer continuum I think of like different types of experiences that people are trying to build.

JARON LANIER: People can actually try it now. It’s not rarified.

KEVIN SCOTT: Right. Like you don’t have to go into a room full of a million dollars’ worth of equipment.

JARON LANIER: Right, exactly. So it seems kind of almost mundane these days, which is probably good, it’s probably a stage it has to go through.

KEVIN SCOTT: But it’s sort of – I always find things like these sort of disruptive innovations – so sometimes like when you look at startups you sort of see a company that’s doing something and like they might be a few years early. And just being a few years early is enough to like kill them.

JARON LANIER: Oh, yeah.

KEVIN SCOTT: And you were 30 years early maybe. And it’s almost incomprehensible to me like how you maintained the intellectual stamina for like all this time and you’re still pushing on it today. Like what made you sort of stick with it that long? It was so hard.

JARON LANIER: Right. It was really hard. It was a very, very hard area to innovate in back then. I had good fortune in the early ’80s, I made some money in videogames. So I had one of the top ten games in ‘83, ‘83, yeah. And so I suddenly have some capital. So a bunch of buddies and I just invested this money from videogames in the first VR company which we incorporated in ‘84 although we’d been sort of doing garage experiments for some years before that and that was called VPL Research. If it had more of a ring to it, it would probably be better known these days.

And VPL was quite an entity. It actually, even though it was always a small company, it had a kind of a feeling of being a larger, more influential company. Everybody knew about it and it was like one of the things in the Valley for a while. And of course it was way too early and the story of VPL is still emotional for me, because I still feel some guilt about – I wonder in my head if I could have done things differently to keep it going.

A lot of people thought it was going to become one of the big Silicon Valley companies. And a lot of – there were just a lot of challenges and difficulties that mounted up simultaneously and it just became too hard a thing to keep going. But there were just wonderful people and we did a lot of stuff. I mean I’m still reading today about these new innovations that are actually things we’d already done back then.

KEVIN SCOTT: Give us an example.

JARON LANIER: Well, I just got a pitch in the mail from somebody who has a new way of visualizing tumors in VR that’s supposed to help radiologists and they’ve just gotten all this money and all this research and I read the papers and looked through it and it’s almost the same thing as a project that we had done in the ’80s with visualization for radiologists.

Of course, it’s cheaper and higher resolution and more responsive and all that. Everything is better now than it could have been back then and vastly less expensive than it was back then. But, it’s sort of like a history that’s forgotten. But I think the right attitude to have about that is that the purpose of history is to enhance the present. So if people want to experience the myth that what they’re doing is entirely new and if that helps them then the better use of history is to be forgotten.

If there are some lessons we learned or some inventions or something that can help them, then the better use of history is to be remembered. But I think obviously forgetting has to be part of life. We can’t live in a way that sort of subdues the present moment for us. It has to be fresh. It has to be its own thing. So I’m fine with the new waves of VR not knowing what the old waves of VR, to a degree. I’d like it if they remember it once in a while. But it’s not important to me that they remember everything.

KEVIN SCOTT: Yeah, well it really is I think an incredible thing that you all did so much stuff in the ’80s. And like you had thought about – like you had this clear vision even before VPL of what could be. And like you’re still even to this day like we just launched the second version of the HoloLens at the Mobile World Congress in Barcelona. And in a sense like that’s just another waypoint along this sort of vision that you had decades ago.

JARON LANIER: Yeah, I still have a triptych of drawings at home, meaning three drawings that are meant to be seen together, showing two people first in natural reality then in mixed reality then in virtual reality. But the term mixed reality is mine, as well, from back then. It predates augmented reality by at least a decade. And it had kind of a roadmap for these things and a vision of what they could mean to people.

And I mean the first arrivers often can see more clearly, because there’s just less clutter and less crap. So I think Ted Nelson did the first design for a digital network and in many ways, I think it’s more insightful and decent and reasonable than many of the ones that came later, and many other examples. I think the generation of Turing and Weiner and these first people had a sense of what computation was all about. In a way it was kind of deeper than a lot of what came later.

I like reading computer science conference proceedings from before I was born. Like the ones in the ’50s are fascinating, because it’s people coming upon this thing really afresh and having to think about it without preconceptions and before there was all this money to be made and all. And it’s amazing. I mean it’s actually – a lot of the old stuff reads as quite radical today.

KEVIN SCOTT: Yeah, and even the stuff that doesn’t read is radical. If you try to put yourself in context, I remember I was – when I was working on my Ph.D. there was a question about Dijkstra’s algorithm on my like doctoral qualifying exams and I remember in that moment being irritated at like why is this relatively simple thing like enshrined as one of the most important accomplishments in computer science, because it looks so simple.

But you think about it and not knowing that it existed, like it wasn’t obvious at all, and I think sometimes even I think you’re totally right in your point about some of these older things reading as very radical today. But I think even the simple things that look simple from our perspective they were quite radical back then.

JARON LANIER: Yeah, I think computer science these days suffers from a couple of problems. One is just there’s such a profit motive. There’s just so much money to be made and people are so attracted to the things that are probably going to make money. That’s one issue. But then another issue is that there are just so many people in it and so many strong personalities and so much baggage on everything that it’s kind of like trying to navigate in some giant city with no infrastructure or something. It’s just at every turn you’re just facing other people who are trying to go somewhere else.

It felt more like this open wilderness. I mean even like when I used to write videogames, or when we wrote our operating system for virtual reality and our applications, we did everything. We did everything and we did it in a way that was completely unique to its time. We had a different architecture than anyone would use today in order to get the efficiency we needed.

We had a high-level incremental compiler that’s unlike anything else I’m aware of that was amazing. We’d actually see something that looked like code or graphical programming, but at the bottom it was instantly turning into op codes that would never crash. And there was like this whole really interesting way of doing code that I’ve never seen anywhere else. And these days you can’t do that. These days it’s all about all the preexisting tools and libraries that you learn. So you’re kind of more in this downstream position. So it’s both better and worse in a way.

KEVIN SCOTT: And it’s sort of funny. This has come up in a bunch of the conversations that I’ve had on this podcast about one of the interesting things with contemporary computer science and software engineering is that the level of abstraction at which people are operating is so much higher now, because you aren’t writing the operating system kernel or the code generator for the compiler or whatnot. There’s some people who do that, but you can do an awful lot by just sort of assembling a bunch of the building blocks that are already there for you.

And it’s in a sense almost miraculous. Like I even look over the course of my career, just sort of thinking about the startup that I did in 2007, I went from Google, where we had had to build all of this distributed computing infrastructure from scratch, because it didn’t exist. And four years later when I did the startup like there were all these open source things that were replicating some of those things. And so like I had a little bit of a start. But I was still in ‘07 we were building our own data centers and racking computers and whatnot. And if like you were doing that today you build it on a public cloud and not have to sort of spend a bunch of your startup’s money sort of standing up a bunch of physical infrastructure.

So on the one hand it’s great. You’re just sort of putting this incredible amount of power into the hands of individual creators, like fewer people can make bigger things, inside of some sort of constraints. But the downside is like if you’re not curious about what’s on the other side of these abstraction boundaries, like you can limit yourself in really interesting ways. And that’s sort of the thing.

When you were developing this stuff you didn’t feel constrained at all, right? You were just –

JARON LANIER: Not at all. I mean I remember feeling like even the built-in architecture in some chip, in a CPU bothered me, because it was enforcing structure on me. And I would hate that. I would resist it. I would say these people at Intel, they’re jerks. Why would they make us – well it has to be something. Yeah, I think if you live inside an abstraction for long enough it becomes like a religion for you. It becomes the framework within which your mind functions, right?

There was a thing that’s just been – there’s a controversy in the last week in AI where somebody is saying that we must not understand. When I say somebody, there’s a certain group of people who are arguing with another group of people. And the debate is since there are mechanisms in our machine learning implementations, like back propagation, that don’t seem to have direct analogues in biological neurology, maybe we don’t understand biological neurology or something.

And I’m like, what do you mean? This is just this code we made. What does it have to do with anything? It doesn’t even work that well if we’re honest. Like, why is this even – how can this even be a controversy? But there’s this thing where you start to believe in your abstraction so much that they loom larger than they should.

KEVIN SCOTT: Yeah and I could not more strongly agree with that point. And I think it’s sort of especially interesting right now in all of these discussions about AI. These machines, these machine learning systems are far less mysterious than some people think they are. I mean it’s like all the way at the bottom it’s just a bunch of things doing linear algebra.

JARON LANIER: And there’s nothing wrong with that.

KEVIN SCOTT: There’s nothing wrong with that.

JARON LANIER: It’s all great.

KEVIN SCOTT: It’s trying to emulate some aspects of intelligent behavior, but that doesn’t necessarily mean that – like the connections I think are sort of arbitrary between biological intelligence and like a supervised machine learning algorithm.

JARON LANIER: I’ve always been bothered by not – I don’t have any – I’m not opposed to the research program of the algorithms that are called AI, but I have a problem with the sort of concept and the culture of the term AI. And a little later, when I was a teenager, my main mentor early on was Marvin Minsky, who kind of invented a lot of the mythology of AI, about this idea that we’re building these little creatures in the computer, and I just find it –

I’m actually thinking of writing a book that might be called something like AI Is Not A Thing, because what happens is we choose some bundle of algorithmic techniques and say this is AI, but it’s kind of arbitrary. Sometimes things go in, sometimes they’re taken out. And then we allow ourselves to believe in it like a monster or a god or something. The AI will take your job. AI will this and that, whereas, in fact, there’s not any particular thing that’s AI. It’s just another example of people coming up with ways of using computation to do things and this whole storytelling I think makes us a little nuts.

KEVIN SCOTT: Yeah, you should definitely write that book. I’m writing a book right now and I say something to that effect in one chapter of my book. But you are the far more eloquent writer and I think an entire book on that topic would be amazing, totally amazing.

JARON LANIER: Maybe it will happen.

KEVIN SCOTT: Yeah, in your copious free time.

JARON LANIER: Yeah, that’s an issue.

KEVIN SCOTT: Yeah, it is an issue. So after VPL you went on to do a whole bunch of other things. And I don’t want to like skim past anything super-important, but where I want to get to before we run out of time here is, you have become quite the sort of technical philosopher for modern digital life. You’ve written these amazing books like you are promoting a bunch of super-interesting ideas like recently in some of the work that you’re doing at Microsoft you’ve sort of coined the phrase “data dignity,” which is like a really interesting concept. So I’d love for you to talk about like why it is that you took on this responsibility for like being sort of the humane intellectual thinking about our digital world?

JARON LANIER: I’m not sure if I coined data dignity, by the way. I think either Glen Weyl, or maybe even Satya Nadella did. Digital dignity was a term – was going to be the title of Who Owns the Future, but the editor didn’t like it, so I turned it to Who Owns the Future.

At any rate, so this is a whole long tale, as well. In the ’80s and ’90s there were a couple of really vociferous, intense movements within hacker culture, within technical culture, about what networking should be like whenever it really comes about.

One of them was this idea that everything should be open and free and that was started from a number of sources. One of them was a guy who is a friend of mine, Richard Stallman, back in Boston. And there were a few other source points for that, as well. And then another was this kind of intense Libertarian philosophy that governments shouldn’t be involved, we should leave everything to entrepreneurs.

And in the last ’80s and early ’90s I ended up spending time with somebody named Al Gore, who is at that time a Senator from Tennessee. He eventually became a Vice President. And he was really interested in these things and he came up with this idea of throwing some government money at people with nascent packet switch networks to bribe them to become

interoperable. And that was the Internet. So that was funded by the Gore Bill.

And so we used to debate what this thing should be and because of the extremely intense, those two dogmas. There was this feeling, well, it will be minimalist, it won’t have accounts, for instance. It won’t represent people. That will be left to private industry. There won’t be any persistent data on it. That will be left to private industry. There won’t be any transactions on it. That will be left to private industry and on and on and on. There won’t be any memory on it. There won’t be any contextualization on it. That will be left to private industry.

And I remember saying to him we’re creating this gift of many hundreds of billions of dollars to persons unknown, because there will be natural network monopolies that form to fill these obviously needed functions, but whatever. That was – there was just this feeling that that was the better way to do things. And since the experiment wasn’t run the other way, we don’t know.

But then the other, the “everything should be free” I think sent us down a terrible path, because it feels all socialist at the first. It feels like this friendly, socialist, lefty thing, but since it’s being mixed with this Libertarian philosophy you end up with only one possible business plan, which is advertising.

So everything feels free, but actually the money is made by third parties who want to influence the users using user data. And it ends up – it starts cute and ends up evolving into the sort of monstrous universal behavior modification scheme. Anyway, this is the stuff I talk about all the time, where I think we’ve gone wrong and we’ve created a network that’s more about deception than it is about reality.

KEVIN SCOTT: So what do you think we can do about that?

JARON LANIER: Well, we’re kind of in a pickle now, to use an expression from when I was a kid. It’s tricky. I mean there are a lot of schools of thought about it. I think we can’t combine socialism and libertarianism in the awkward way we did and expect to get anything useful.

And I think we should just choose one of them. And I personally think we’re better off choosing markets.

I’m worried about where we are right now. I just feel like our present Internet is destroying societies and democracies and economies. I think it’s bringing down civilization. It’s bad. We really screwed this thing up.

KEVIN SCOTT: So you’ve been working on a bunch of concrete things to try to figure out like how to introduce these new incentive structures. Can you elaborate on that a little bit more?

JARON LANIER: Yeah. Well, the problem is how to get from here to there. I kind of have in my head an image of what a society would be like with paid data. There’s a few things to say about it. One is, there are a lot of people out there who pretend to be making a living online but aren’t because they’re fakers. It’s all a big illusion. It’s what we used to call Horatio Alger illusion, where you create this illusion that there’s this way of making a living when, in fact, there isn’t. It’s only for a token, small number of people.

However, there’s another population of people out there who are making a living, but not within the rules dictated by a central hub, but as actors. For instance, there are tens of millions, maybe – we don’t know the total number. But at least 50 million people in the world who are actually making a living delivering online video lessons and counseling and guidance. This is anything from legal consulting to religious training to yoga teachers, to musical instrument teachers.

All those people have sort of cobbled together something that has to fight against the grain of everything, because there’s no –

KEVIN SCOTT: There’s no infrastructure to support them.

JARON LANIER: There’s no infrastructure, so each one of them has had to invent their own infrastructure by cobbling together little pieces from the different digital companies. And that population interests me. In a way I see them as the future. Those are the people who don’t have to worry about their jobs being taken by robots, unless I mean they could be, all we have to do is create some machine learning thing that steals all their data and makes a fake clarinet teacher, without paying them for their data, just stealing their value. And that’s what we’ve done in so many other areas.

So the future I would see is to, first of all, try to support, identify those groups and support them and also identify those communities that are trying to create new structures to help people cooperate in decentralized ways. And here the blockchain community, not the get rich quick blockchain, but the other blockchain, the blockchain of people who are interested in new ways of cooperation that can be mediated by networks, those people could be really important and helpful.

I think we need to invent new structures. The reason that we treat data as being worthless, even though the companies that collect the data become the most valuable ones in the world, is that there’s no collective bargaining for people whose data is taken. So in any other economic example, in order to have a productive economy you have to have some – you have to invent some kind of structure so that people can cooperate and have it not be this Hobbesian race to the bottom where each person is against each other person. If you believe more in capital than labor you call that a corporation or a legal partnership or something. So these people are incentivized to cooperate instead of try to kill each other. If you believe in labor over capital you call it the union. And you call it collective bargaining. But on the Internet the difference is academic.

I was playing around with terms like Unorp and Corporion and they’re terrible. So we just came up with – my research partner Glen Weyl and I came up with the term MID, actually my wife came up with that, mediator of individual data. So you’d have something that’s a way for people to band into a group so as to not have the value of their data descend to zero through interpersonal competition, but instead have a degree of local cooperation.

So we need to create those things. And MIDs can serve another function here. I’m talking fast, because I know we’re almost out of time. But, one of the things that’s really terrible about what’s happened in the world is we’ve been petitioning tech companies to become the arbiters of culture and politics.

But the thing is, do we really want tech companies to become the new de facto government? Is that what we want? I don’t think so. So the MIDs could also become brands in themselves where people who have bonded together to create a MID not only are collectively bargaining for the value of their data, but the MID itself has become a channel, like, if you like, like a guild or a union, or like a corporation or a brand that represents a certain thing.

It might say, whatever data comes through here is scientifically rigorous and has been checked, or whatever data comes through here is fashionista approved and is very beautiful, or whatever data comes through here is guaranteed to be really amusing and suitable for your whole family, or whatever. What it creates is these in-between sized structures that can take on this function of quality maintenance, because you don’t want a centralized source being the maintainer of quality. That’s a recipe for some kind of dysfunction or too much centralized power.

So the MIDs both solve the economic problem and the quality problem. And we need to start creating them. So there are fledgling attempts to create them. Right now they have no infrastructure tools to help them along. I’d like to change that. And that’s just one little corner of the problem.

I’m mostly just trying to – honestly, I’m just trying to get the tech companies to see the light. And here some of them are better than others

KEVIN SCOTT: Yeah. So let’s switch a little bit into like all of these other interests that you have. I think one of the fascinating things about you that folks under-appreciate is that you are a composer and a musician and you have one of the largest, maybe the largest collection of musical instruments in the world in your home. So your mother was a piano virtuoso. But how has this remained a thread in your life all these years?

JARON LANIER: After she died, I feel like music is my main connection to her. And I still play the piano, but not so much straight classical playing anymore. I have my own style and it’s pretty unusual. But I started just learning new instruments and I have this voracious, perhaps not always healthy need to always be learning a new instrument. And so whether it’s the largest instrument collection I’m a little doubtful of, because there are some pretty big instrument museums. But, in terms of playable collection, I’m pretty sure it is. And I don’t know how many there are. But there are a lot of instruments and I do – I can play them.

KEVIN SCOTT: And we’re talking like hundreds, if not thousands.

JARON LANIER: Certainly in the thousands, yeah.

KEVIN SCOTT: Which is sort of a mind-boggling, interesting thought in and of itself that there are like thousands of distinct instruments that one could collect.

JARON LANIER: Well, they’re the best user interfaces that have ever been created. They’re the ones that support peak human performance like no other invention ever. And they’re profoundly beautiful. And each one has a story. And each one is kind of a form of time travel, because you learn to move and breathe like the people who originally played it wherever it’s from. So it’s a kind of a cultural record that’s unlike any other one. It’s a haptic record, if you like.

KEVIN SCOTT: I mean I’ve always been fascinated with piano and I think the reason is it’s always struck me that a piano is not too dissimilar from a computer. It’s like this complicated machine that requires some non-trivial degree of mastery to get anything out of it. And like sometimes that struggle to achieve mastery is like this sort of long, isolating activity. Like I’ve read biographies of famous pianists and some describe it as you sort of sit alone in this room and struggle against this machine to get it to bend to your will. And I’m like, holy crap, that’s sort of what you do sometimes as a programmer.

JARON LANIER: Well, I’ve experienced feeling alone with a computer, but I’ve never experienced feeling alone with a piano.

KEVIN SCOTT: Which is interesting, even when you’re practicing?

JARON LANIER: Yeah, pianos are a bit mysterious, because they’re sort of the button box that transcends button boxness. They have some kind of a life in them that they shouldn’t have. I think that’s one of the reasons that they’re so provocative to computer scientists. Of course, the piano led to the computer pretty directly, because around Mozart’s time somebody made a non-deterministic player piano, which is what inspired the jacquard loom, which inspired the Babbage generalized calculator, which inspired Turing, et cetera. So you can blame the piano for all this if you want to.

KEVIN SCOTT: So what has been your favorite performing experience over all of the years?

JARON LANIER: Performing? I’ve had the good fortune to perform with a lot of interesting people, although – and I was living as a professional musician for a while in the ’90s. But I’ve never been like a major one. I’ve never been a major star or anything. But, as a side man I’ve had incredible performance experiences. I think my favorite one was when I toured with Yoko Ono and her Plastic Band and Yoko and I would do these duets that were – a number of people said they were the strangest thing ever on stage. And I think we got there. I think we did a – John Perry Barlow called it the heavyweight championship of weirdness. And I think we got – yeah, that was good.

And I played with Ornette Coleman, this wonderful – the father of free jazz. And I’ve done a lot with Philip Glass, including just recently. In theory we’re doing a new record together. We did one in the ’90s.

KEVIN SCOTT: That’s so awesome.

JARON LANIER: I just did a show with Philip where I brought the pedal steel guitar in to the minimalist music aesthetic for the first time and it worked great. It was so fun. And I just did a thing with T-Bone Burnett a while ago for a Sara Bareilles record that was really, really cool. And I’ve done – I’ve played with all kinds of people. I’ve played with George Clinton and P Funk. I’ve done more than you’d imagine, but always kind of in the background as a performer. I’ve done some solo stuff, too. But, as a performer I can only do so much.

I’ve had this career in computer science and another one as a writer at a certain point you can’t really do everything always and then have a family. So unfortunately, I haven’t been doing as much lately, but still a little bit now and then.

KEVIN SCOTT: But, even though – so the writer, philosopher, computer scientist, composer, performer, like you still have other interests, which I guess your mother really did like –

JARON LANIER: She did a number on me.

KEVIN SCOTT: She certainly convinced you that you had to sort of exhibit excellence in a bunch of different domains. Maybe one of the most interesting things that you and I have chatted about is the Neural Information Processing Symposium is like the big, deep learning conference for the field.

And the best paper at this year’s symposium was this thing called Neural Ordinary Differential Equations, which is this – the short idea is that neural networks are usually these sort of layers of connected artificial neurons and you’re trying to sort of figure out like the activation weights between the connections of all of these layers and like which non-linear functions you’re using to sort of accumulate the weights into a network node and whether or not you should use things like dropout to like impose some sort of memory loss as you’re doing training and blah, blah, blah, right and like there’s back propagation and all this other stuff.

And so like this paper is sort of interesting in that it sort of models a bunch of the interior state of these deep neural networks as a system of ordinary differential equations. And it was sort of a sensational result, because it has some really big performance implications for training deep neural networks and like the amount of computation that’s required to train them is one of the sort of big things on people’s minds. So that’s great. But when you saw it the connection you made was to quantum field theory.

JARON LANIER: Yeah, so here it might be a bit premature to speak about this, but one of the dimensions in my life is I have a lot of physicist friends. I’ve done sort of weird projects with theoretical physicists from time to time. And this goes – this all started, because my first serious girlfriend was a daughter of the head of the Physics Department at Caltech. So when I was a teenager, I was sort of hanging out here and I got an informal chance to learn directly from Feynman and Gell-Mann and all these amazing people. And ever since –

KEVIN SCOTT: Which is insane.

JARON LANIER: Yeah, ever since then I’ve had this kind of – so one of my best friends is this guy named Lee Smolin, who cofounded the loop quantum gravity approach to trying to find a unified theory, which is probably – since string theory kind of burned itself out, it might be one of the most prominent ones found as an alternative.

Anyway, so there’s this thing in physics where we’ve always had pretty – we want really simple equations to describe let’s say fields in that we want them to have emergent behavior that’s complicated enough to be reality. But the thing is we’ve kind of burned – it’s kind of like in a house where you put all your mess in one room to pretend the house is clean. So in string theory they tried to simplify some stuff and they ended up with this insane.

KEVIN SCOTT: Room full of mess.

JARON LANIER: Really messy room of like of like an unbounded number of possible – this was a really messy room, the sort of range of possible string theories. And it’s kind of like playing whack-a-mole, when you try to – you often end up creating this big mess in another corner when you’re cleaning up one part of it.

KEVIN SCOTT: Yeah, I think it’s fantastic how broadly your mind wanders. And with that I think we should wrap up. Thank you.

JARON LANIER: Sure.

KEVIN SCOTT: Thanks so much, Jaron, for being on the podcast.

JARON LANIER: Delighted to be here. Thank you for having me.

KEVIN SCOTT: Awesome.

[MUSIC]

CHRISTINA WARREN: Well, thanks for joining us for Behind the Tech. You just heard Microsoft CTO Kevin Scott speaking with Jaron Lanier. So, what really struck me about that conversation was how far ahead some of the visions for virtual reality were, you know, decades ago and how similar those visions are to what we’re actually seeing in the market now with both VR and with augmented reality.

KEVIN SCOTT: Yeah, it’s the – it’s this really fascinating thing with true visionaries like Jaron. Like, he saw this thing, like, way, way, way, way before anyone else did, and it’s not just the – the vision of sort of seeing this thing that one day might be, but just the sort of his consistency and tenacity over time to sort of stick with the vision.

It’s not like he’s wavered. Like, he’s been doing this for almost four decades now. And, like, he’s had this vision and he’s kept pushing, you know, episodically for, like, this very, very long period of time. And, like, I find that almost as amazing as the vision itself, just the willingness to, like, believe in something for that long and to just push against it as hard as you can.

CHRISTINA WARREN: Yeah, it’s so interesting to me. You know, when you – when he was talking about how, you know, he sees – he’s getting pitches for certain, you know, uses of VR and he’s, like, “Oh, yeah, I had a – had a paper, you know, kind of predicting that, you know, 30 years ago.” And – and he was right.

And as you said, he continued to push and be committed to that, which is just kind of incredible.

KEVIN SCOTT: Yeah, you know, and sometimes the frustrating thing with technology is timing matters way more than you would like to think.

CHRISTINA WARREN: Yeah.

KEVIN SCOTT: Like, unfortunately, like, vision and persistence aren’t enough. Like, sometimes, like, the technology that you need and, like, the set of conditions in the ecosystem you need to exist in order for sometimes to become broadly adopted by a whole bunch of people is just – just isn’t there. And it’s really interesting, like, it’s sort of – you can almost see it right now that, like, you know, mixed reality, augmented reality, you know, like the whole grand, you know, the whole grand virtual reality vision, like, might actually be within reach now. But it’s, like, taken all of that time.

CHRISTINA WARREN: And, yeah, no, it’s so true. I mean, as you said, you know, Microsoft just showed off the new version of the HoloLens and we – it feels so much closer, and yet it’s just still somewhat, you know, you can kind of see what Jaron’s vision has been all this time, and it seems to just be within grasp. It’s really exciting.

KEVIN SCOTT: The other thing that’s sort of fascinating about his vision for VR is on the one hand, I think it is like a very deeply technical thing, but I think you know, as you heard in the conversation, the thing, perhaps, even more than technology that motivated it is like this very, you know, humane desire that he had, like, to connect with other people.

CHRISTINA WARREN: Yes.

KEVIN SCOTT: And, like, that’s something that, you know, you don’t always get from folks who are trying to do something with, like, deeply, deeply, deeply technical technology. (Laughter.)

CHRISTINA WARREN: No, you’re – you’re exactly right. It kind of reminds me a little bit of, you know, Tim Berners-Lee and the Worldwide Web, which is a similar thing and that’s, you know, celebrating an anniversary right now, too.

And it’s, like, you’re right, a lot of times, it’s rare to see these intersections between these highly technical things and these also highly social and personal and deeply connective things.

KEVIN SCOTT: But sometimes, like, those are the things that, like, have the biggest impact on the world is, like, you’ve got this desire to, you know, sort of facilitate more of our own humanity to, like, empower and ennoble, like, individuals and groups and, like, those technologies can be really, really profoundly transformative.

CHRISTINA WARREN: I mean, I would actually argue, I think, that what you just described, is kind of the basis for the most transformative technology, whether we’re talking about radio or television or transistors or anything else.

KEVIN SCOTT: Or the PC.

CHRISTINA WARREN: PC, absolutely is finding a way to facilitate humanity.

KEVIN SCOTT: Yeah, for sure.

CHRISTINA WARREN: All right, so I think we’re out of time for this episode, but we’re going to meet another icon on our next show.

KEVIN SCOTT: That’s right. I’ll sit down with Reid Hoffman, investor, author, and entrepreneur, someone I consider a true friend.

[MUSIC]

KEVIN SCOTT: Be sure to join us next time on Behind the Tech. And please, help spread the word. Tell all of the geeks and non-geeks you know. And don’t forget to subscribe. See you next time!