The following is a conversation with Ian Goodfellow.

He’s the author of the popular textbook on deep learning

simply titled Deep Learning.

He coined the term of Generative Adversarial Networks,

otherwise known as GANs,

and with his 2014 paper is responsible

for launching the incredible growth

of research and innovation in this subfield

of deep learning.

He got his BS and MS at Stanford,

his PhD at University of Montreal

with Yoshua Bengio and Aaron Kerrville.

He held several research positions

including at OpenAI, Google Brain,

and now at Apple as the Director of Machine Learning.

This recording happened while Ian was still at Google Brain,

but we don’t talk about anything specific to Google

or any other organization.

This conversation is part

of the Artificial Intelligence Podcast.

If you enjoy it, subscribe on YouTube, iTunes,

or simply connect with me on Twitter at Lex Friedman,

spelled F R I D.

And now here’s my conversation with Ian Goodfellow.

You open your popular deep learning book

with a Russian doll type diagram

that shows deep learning is a subset

of representation learning,

which in turn is a subset of machine learning

and finally a subset of AI.

So this kind of implies that there may be limits

to deep learning in the context of AI.

So what do you think is the current limits of deep learning

and are those limits something

that we can overcome with time?

Yeah, I think one of the biggest limitations

of deep learning is that right now it requires

really a lot of data, especially labeled data.

There are some unsupervised

and semi supervised learning algorithms

that can reduce the amount of labeled data you need,

but they still require a lot of unlabeled data,

reinforcement learning algorithms.

They don’t need labels,

but they need really a lot of experiences.

As human beings, we don’t learn to play Pong

by failing at Pong 2 million times.

So just getting the generalization ability better

is one of the most important bottlenecks

in the capability of the technology today.

And then I guess I’d also say deep learning

is like a component of a bigger system.

So far, nobody is really proposing to have

only what you’d call deep learning

as the entire ingredient of intelligence.

You use deep learning as sub modules of other systems,

like AlphaGo has a deep learning model

that estimates the value function.

Most reinforcement learning algorithms

have a deep learning module

that estimates which action to take next,

but you might have other components.

So you’re basically building a function estimator.

Do you think it’s possible,

you said nobody’s kind of been thinking about this so far,

but do you think neural networks could be made to reason

in the way symbolic systems did in the 80s and 90s

to do more, create more like programs

as opposed to functions?

Yeah, I think we already see that a little bit.

I already kind of think of neural nets

as a kind of program.

I think of deep learning as basically learning programs

that have more than one step.

So if you draw a flow chart

or if you draw a TensorFlow graph

describing your machine learning model,

I think of the depth of that graph

as describing the number of steps that run in sequence.

And then the width of that graph

is the number of steps that run in parallel.

Now it’s been long enough

that we’ve had deep learning working

that it’s a little bit silly

to even discuss shallow learning anymore.

But back when I first got involved in AI,

when we used machine learning,

we were usually learning things like support vector machines.

You could have a lot of input features to the model

and you could multiply each feature by a different weight.

All those multiplications were done

in parallel to each other.

There wasn’t a lot done in series.

I think what we got with deep learning

was really the ability to have steps of a program

that run in sequence.

And I think that we’ve actually started to see

that what’s important with deep learning

is more the fact that we have a multi step program

rather than the fact that we’ve learned a representation.

If you look at things like resonance, for example,

they take one particular kind of representation

and they update it several times.

Back when deep learning first really took off

in the academic world in 2006,

when Jeff Hinton showed that you could train

deep belief networks,

everybody who was interested in the idea

thought of it as each layer

learns a different level of abstraction.

That the first layer trained on images

learns something like edges

and the second layer learns corners.

And eventually you get these kind of grandmother cell units

that recognize specific objects.

Today I think most people think of it more

as a computer program where as you add more layers

you can do more updates before you output your final number.

But I don’t think anybody believes that

layer 150 of the ResNet is a grandmother cell

and layer 100 is contours or something like that.

Okay, so you’re not thinking of it

as a singular representation that keeps building.

You think of it as a program,

sort of almost like a state.

Representation is a state of understanding.

Yeah, I think of it as a program

that makes several updates

and arrives at better and better understandings,

but it’s not replacing the representation at each step.

It’s refining it.

And in some sense, that’s a little bit like reasoning.

It’s not reasoning in the form of deduction,

but it’s reasoning in the form of taking a thought

and refining it and refining it carefully

until it’s good enough to use.

So do you think, and I hope you don’t mind,

we’ll jump philosophical every once in a while.

Do you think of cognition, human cognition,

or even consciousness as simply a result

of this kind of sequential representation learning?

Do you think that can emerge?

Cognition, yes, I think so.

Consciousness, it’s really hard to even define

what we mean by that.

I guess there’s, consciousness is often defined

as things like having self awareness,

and that’s relatively easy to turn into something actionable

for a computer scientist to reason about.

People also define consciousness

in terms of having qualitative states of experience,

like qualia, and there’s all these philosophical problems,

like could you imagine a zombie

who does all the same information processing as a human,

but doesn’t really have the qualitative experiences

that we have?

That sort of thing, I have no idea how to formalize

or turn it into a scientific question.

I don’t know how you could run an experiment

to tell whether a person is a zombie or not.

And similarly, I don’t know how you could run

an experiment to tell whether an advanced AI system

had become conscious in the sense of qualia or not.

But in the more practical sense,

like almost like self attention,

you think consciousness and cognition can,

in an impressive way, emerge from current types

of architectures that we think of as learning.

Or if you think of consciousness

in terms of self awareness and just making plans

based on the fact that the agent itself exists in the world,

reinforcement learning algorithms

are already more or less forced

to model the agent’s effect on the environment.

So that more limited version of consciousness

is already something that we get limited versions of

with reinforcement learning algorithms

if they’re trained well.

But you say limited, so the big question really

is how you jump from limited to human level, right?

And whether it’s possible,

even just building common sense reasoning

seems to be exceptionally difficult.

So if we scale things up,

if we get much better on supervised learning,

if we get better at labeling,

if we get bigger data sets, more compute,

do you think we’ll start to see really impressive things

that go from limited to something,

echoes of human level cognition?

I think so, yeah.

I’m optimistic about what can happen

just with more computation and more data.

I do think it’ll be important

to get the right kind of data.

Today, most of the machine learning systems we train

are mostly trained on one type of data for each model.

But the human brain, we get all of our different senses

and we have many different experiences

like riding a bike, driving a car,

talking to people, reading.

I think when we get that kind of integrated data set,

working with a machine learning model

that can actually close the loop and interact,

we may find that algorithms not so different

from what we have today learn really interesting things

when you scale them up a lot

and train them on a large amount of multimodal data.

So multimodal is really interesting,

but within, like you’re working adversarial examples.

So selecting within modal, within one mode of data,

selecting better at what are the difficult cases

from which you’re most useful to learn from.

Oh yeah, like could we get a whole lot of mileage

out of designing a model that’s resistant

to adversarial examples or something like that?

Right, that’s the question.

My thinking on that has evolved a lot

over the last few years.

When I first started to really invest

in studying adversarial examples,

I was thinking of it mostly as adversarial examples

reveal a big problem with machine learning

and we would like to close the gap

between how machine learning models respond

to adversarial examples and how humans respond.

After studying the problem more,

I still think that adversarial examples are important.

I think of them now more of as a security liability

than as an issue that necessarily shows

there’s something uniquely wrong

with machine learning as opposed to humans.

Also, do you see them as a tool

to improve the performance of the system?

Not on the security side, but literally just accuracy.

I do see them as a kind of tool on that side,

but maybe not quite as much as I used to think.

We’ve started to find that there’s a trade off

between accuracy on adversarial examples

and accuracy on clean examples.

Back in 2014, when I did the first

adversarily trained classifier that showed resistance

to some kinds of adversarial examples,

it also got better at the clean data on MNIST.

And that’s something we’ve replicated several times

on MNIST, that when we train

against weak adversarial examples,

MNIST classifiers get more accurate.

So far that hasn’t really held up on other data sets

and hasn’t held up when we train

against stronger adversaries.

It seems like when you confront

a really strong adversary,

you tend to have to give something up.

Interesting.

But it’s such a compelling idea

because it feels like that’s how us humans learn

is through the difficult cases.

We try to think of what would we screw up

and then we make sure we fix that.

It’s also in a lot of branches of engineering,

you do a worst case analysis

and make sure that your system will work in the worst case.

And then that guarantees that it’ll work

in all of the messy average cases that happen

when you go out into a really randomized world.

Yeah, with driving with autonomous vehicles,

there seems to be a desire to just look for,

think adversarially,

try to figure out how to mess up the system.

And if you can be robust to all those difficult cases,

then you can, it’s a hand wavy empirical way

to show your system is safe.

Today, most adversarial example research

isn’t really focused on a particular use case,

but there are a lot of different use cases

where you’d like to make sure that the adversary

can’t interfere with the operation of your system.

Like in finance,

if you have an algorithm making trades for you,

people go to a lot of an effort

to obfuscate their algorithm.

That’s both to protect their IP

because you don’t want to research

and develop a profitable trading algorithm

then have somebody else capture the gains.

But it’s at least partly

because you don’t want people to make adversarial examples

that fool your algorithm into making bad trades.

Or I guess one area that’s been popular

in the academic literature is speech recognition.

If you use speech recognition to hear an audio wave form

and then turn that into a command

that a phone executes for you,

you don’t want a malicious adversary

to be able to produce audio

that gets interpreted as malicious commands,

especially if a human in the room doesn’t realize

that something like that is happening.

And speech recognition,

has there been much success

in being able to create adversarial examples

that fool the system?

Yeah, actually.

I guess the first work that I’m aware of

is a paper called Hidden Voice Commands

that came out in 2016, I believe.

And they were able to show that they could make sounds

that are not understandable by a human

but are recognized as the target phrase

that the attacker wants the phone to recognize it as.

Since then, things have gotten a little bit better

on the attacker’s side

when worse on the defender’s side.

It’s become possible to make sounds

that sound like normal speech

but are actually interpreted as a different sentence

than the human hears.

The level of perceptibility

of the adversarial perturbation is still kind of high.

When you listen to the recording,

it sounds like there’s some noise in the background,

just like rustling sounds.

But those rustling sounds

are actually the adversarial perturbation

that makes the phone hear a completely different sentence.

Yeah, that’s so fascinating.

Peter Norvig mentioned

that you’re writing the deep learning chapter

for the fourth edition

of the Artificial Intelligence, A Modern Approach book.

So how do you even begin summarizing

the field of deep learning in a chapter?

Well, in my case, I waited like a year

before I actually wrote anything.

Even having written a full length textbook before,

it’s still pretty intimidating

to try to start writing just one chapter

that covers everything.

One thing that helped me make that plan

was actually the experience

of having written the full book before

and then watching how the field changed

after the book came out.

I’ve realized there’s a lot of topics

that were maybe extraneous in the first book

and just seeing what stood the test

of a few years of being published

and what seems a little bit less important

to have included now helped me pare down the topics

I wanted to cover for the book.

It’s also really nice now

that the field is kind of stabilized

to the point where some core ideas from the 1980s

are still used today.

When I first started studying machine learning,

almost everything from the 1980s had been rejected

and now some of it has come back.

So that stuff that’s really stood the test of time

is what I focused on putting into the book.

There’s also, I guess, two different philosophies

about how you might write a book.

One philosophy is you try to write a reference

that covers everything.

The other philosophy is you try to provide

a high level summary that gives people the language

to understand a field

and tells them what the most important concepts are.

The first deep learning book that I wrote

with Joshua and Aaron was somewhere

between the two philosophies,

that it’s trying to be both a reference

and an introductory guide.

Writing this chapter for Russell Norvig’s book,

I was able to focus more on just a concise introduction

of the key concepts and the language

you need to read about them more.

In a lot of cases, I actually just wrote paragraphs

that said, here’s a rapidly evolving area

that you should pay attention to.

It’s pointless to try to tell you what the latest

and best version of a learn to learn model is.

I can point you to a paper that’s recent right now,

but there isn’t a whole lot of a reason to delve

into exactly what’s going on

with the latest learning to learn approach

or the latest module produced

by a learning to learn algorithm.

You should know that learning to learn is a thing

and that it may very well be the source of the latest

and greatest convolutional net or recurrent net module

that you would want to use in your latest project.

But there isn’t a lot of point in trying to summarize

exactly which architecture and which learning approach

got to which level of performance.

So you maybe focus more on the basics of the methodology.

So from back propagation to feed forward

to recurrent neural networks, convolutional,

that kind of thing?

Yeah, yeah.

So if I were to ask you, I remember I took algorithms

and data structures algorithms course.

I remember the professor asked, what is an algorithm?

And yelled at everybody in a good way

that nobody was answering it correctly.

Everybody knew what the algorithm, it was graduate course.

Everybody knew what an algorithm was,

but they weren’t able to answer it well.

So let me ask you in that same spirit,

what is deep learning?

I would say deep learning is any kind of machine learning

that involves learning parameters of more than one

consecutive step.

So that, I mean, shallow learning is things

where you learn a lot of operations that happen in parallel.

You might have a system that makes multiple steps.

Like you might have hand designed feature extractors,

but really only one step is learned.

Deep learning is anything where you have multiple operations

in sequence, and that includes the things

that are really popular today,

like convolutional networks and recurrent networks.

But it also includes some of the things that have died out

like Bolton machines,

where we weren’t using back propagation.

Today, I hear a lot of people define deep learning

as gradient descent applied

to these differentiable functions.

And I think that’s a legitimate usage of the term.

It’s just different from the way that I use the term myself.

So what’s an example of deep learning

that is not gradient descent and differentiable functions?

In your, I mean, not specifically perhaps,

but more even looking into the future,

what’s your thought about that space of approaches?

Yeah, so I tend to think of machine learning algorithms

as decomposed into really three different pieces.

There’s the model, which can be something like a neural net

or a Bolton machine or a recurrent model.

And that basically just describes how do you take data

and how do you take parameters?

And what function do you use to make a prediction

given the data and the parameters?

Another piece of the learning algorithm

is the optimization algorithm.

Or not every algorithm can be really described

in terms of optimization,

but what’s the algorithm for updating the parameters

or updating whatever the state of the network is?

And then the last part is the data set,

like how do you actually represent the world

as it comes into your machine learning system?

So I think of deep learning as telling us something about

what does the model look like?

And basically to qualify as deep,

I say that it just has to have multiple layers.

That can be multiple steps

in a feed forward differentiable computation.

That can be multiple layers in a graphical model.

There’s a lot of ways that you could satisfy me

that something has multiple steps

that are each parameterized separately.

I think of gradient descent

as being all about that other piece,

the how do you actually update the parameters piece?

So you could imagine having a deep model

like a convolutional net

and training it with something like evolution

or a genetic algorithm.

And I would say that still qualifies as deep learning.

And then in terms of models

that aren’t necessarily differentiable,

I guess Bolton machines are probably

the main example of something

where you can’t really take a derivative

and use that for the learning process.

But you can still argue that the model

has many steps of processing that it applies

when you run inference in the model.

So it’s the steps of processing that’s key.

So Jeff Hinton suggests that we need to throw away

back propagation and start all over.

What do you think about that?

What could an alternative direction

of training neural networks look like?

I don’t know that back propagation

is gonna go away entirely.

Most of the time when we decide

that a machine learning algorithm

isn’t on the critical path to research for improving AI,

the algorithm doesn’t die.

It just becomes used for some specialized set of things.

A lot of algorithms like logistic regression

don’t seem that exciting to AI researchers

who are working on things like speech recognition

or autonomous cars today.

But there’s still a lot of use for logistic regression

and things like analyzing really noisy data

in medicine and finance

or making really rapid predictions

in really time limited contexts.

So I think back propagation and gradient descent

are around to stay, but they may not end up being

everything that we need to get to real human level

or super human AI.

Are you optimistic about us discovering

back propagation has been around for a few decades?

So are you optimistic about us as a community

being able to discover something better?

Yeah, I am.

I think we likely will find something that works better.

You could imagine things like having stacks of models

where some of the lower level models

predict parameters of the higher level models.

And so at the top level,

you’re not learning in terms of literally

calculating gradients,

but just predicting how different values will perform.

You can kind of see that already in some areas

like Bayesian optimization,

where you have a Gaussian process

that predicts how well different parameter values

will perform.

We already use those kinds of algorithms

for things like hyper parameter optimization.

And in general, we know a lot of things other than back prop

that work really well for specific problems.

The main thing we haven’t found is

a way of taking one of these other

non back prop based algorithms

and having it really advanced the state of the art

on an AI level problem.

Right.

But I wouldn’t be surprised if eventually

we find that some of these algorithms

that even the ones that already exist,

not even necessarily new one,

we might find some way of customizing

one of these algorithms to do something really interesting

at the level of cognition or the level of,

I think one system that we really don’t have working

quite right yet is like short term memory.

We have things like LSTMs,

they’re called long short term memory.

They still don’t do quite what a human does

with short term memory.

Like gradient descent to learn a specific fact

has to do multiple steps on that fact.

Like if I tell you the meeting today is at 3 p.m.,

I don’t need to say over and over again,

it’s at 3 p.m., it’s at 3 p.m., it’s at 3 p.m.,

it’s at 3 p.m.

for you to do a gradient step on each one.

You just hear it once and you remember it.

There’s been some work on things like self attention

and attention like mechanisms,

like the neural Turing machine

that can write to memory cells

and update themselves with facts like that right away.

But I don’t think we’ve really nailed it yet.

And that’s one area where I’d imagine

that new optimization algorithms

or different ways of applying

existing optimization algorithms

could give us a way of just lightning fast

updating the state of a machine learning system

to contain a specific fact like that

without needing to have it presented

over and over and over again.

So some of the success of symbolic systems in the 80s

is they were able to assemble these kinds of facts better.

But there’s a lot of expert input required

and it’s very limited in that sense.

Do you ever look back to that

as something that we’ll have to return to eventually?

Sort of dust off the book from the shelf

and think about how we build knowledge,

representation, knowledge base.

Like will we have to use graph searches?

Graph searches, right.

And like first order logic and entailment

and things like that.

That kind of thing, yeah, exactly.

In my particular line of work,

which has mostly been machine learning security

and also generative modeling,

I haven’t usually found myself moving in that direction.

For generative models, I could see a little bit of,

it could be useful if you had something

like a differentiable knowledge base

or some other kind of knowledge base

where it’s possible for some of our

fuzzier machine learning algorithms

to interact with a knowledge base.

I mean, your network is kind of like that.

It’s a differentiable knowledge base of sorts.

Yeah.

But.

If we had a really easy way of giving feedback

to machine learning models,

that would clearly help a lot with generative models.

And so you could imagine one way of getting there

would be get a lot better at natural language processing.

But another way of getting there would be

take some kind of knowledge base

and figure out a way for it to actually

interact with a neural network.

Being able to have a chat with a neural network.

Yeah.

So like one thing in generative models we see a lot today

is you’ll get things like faces that are not symmetrical,

like people that have two eyes that are different colors.

I mean, there are people with eyes

that are different colors in real life,

but not nearly as many of them as you tend to see

in the machine learning generated data.

So if you had either a knowledge base

that could contain the fact,

people’s faces are generally approximately symmetric

and eye color is especially likely

to be the same on both sides.

Being able to just inject that hint

into the machine learning model

without it having to discover that itself

after studying a lot of data

would be a really useful feature.

I could see a lot of ways of getting there

without bringing back some of the 1980s technology,

but I also see some ways that you could imagine

extending the 1980s technology to play nice with neural nets

and have it help get there.

Awesome.

So you talked about the story of you coming up

with the idea of GANs at a bar with some friends.

You were arguing that this, you know, GANs would work,

generative adversarial networks,

and the others didn’t think so.

Then you went home at midnight, coded it up, and it worked.

So if I was a friend of yours at the bar,

I would also have doubts.

It’s a really nice idea,

but I’m very skeptical that it would work.

What was the basis of their skepticism?

What was the basis of your intuition why it should work?

I don’t want to be someone who goes around

promoting alcohol for the purposes of science,

but in this case,

I do actually think that drinking helped a little bit.

When your inhibitions are lowered,

you’re more willing to try out things

that you wouldn’t try out otherwise.

So I have noticed in general

that I’m less prone to shooting down some of my own ideas

when I have had a little bit to drink.

I think if I had had that idea at lunchtime,

I probably would have thought,

it’s hard enough to train one neural net,

you can’t train a second neural net

in the inner loop of the outer neural net.

That was basically my friend’s objection,

was that trying to train two neural nets at the same time

would be too hard.

So it was more about the training process,

unless, so my skepticism would be,

you know, I’m sure you could train it,

but the thing it would converge to

would not be able to generate anything reasonable,

any kind of reasonable realism.

Yeah, so part of what all of us were thinking about

when we had this conversation was deep Bolton machines,

which a lot of us in the lab, including me,

were a big fan of deep Bolton machines at the time.

They involved two separate processes

running at the same time.

One of them is called the positive phase,

where you load data into the model

and tell the model to make the data more likely.

The other one is called the negative phase,

where you draw samples from the model

and tell the model to make those samples less likely.

In a deep Bolton machine,

it’s not trivial to generate a sample.

You have to actually run an iterative process

that gets better and better samples

coming closer and closer to the distribution

the model represents.

So during the training process,

you’re always running these two systems at the same time,

one that’s updating the parameters of the model

and another one that’s trying to generate samples

from the model.

And they worked really well in things like MNIST,

but a lot of us in the lab, including me,

had tried to get deep Bolton machines

to scale past MNIST to things like generating color photos,

and we just couldn’t get the two processes

to stay synchronized.

So when I had the idea for GANs,

a lot of people thought that the discriminator

would have more or less the same problem

as the negative phase in the Bolton machine,

that trying to train the discriminator in the inner loop,

you just couldn’t get it to keep up

with the generator in the outer loop,

and that would prevent it from converging

to anything useful.

Yeah, I share that intuition.

Yeah.

But turns out to not be the case.

A lot of the time with machine learning algorithms,

it’s really hard to predict ahead of time

how well they’ll actually perform.

You have to just run the experiment and see what happens.

And I would say I still today don’t have

like one factor I can put my finger on and say,

this is why GANs worked for photo generation

and deep Bolton machines don’t.

There are a lot of theory papers

showing that under some theoretical settings,

the GAN algorithm does actually converge,

but those settings are restricted enough

that they don’t necessarily explain the whole picture

in terms of all the results that we see in practice.

So taking a step back,

can you, in the same way as we talked about deep learning,

can you tell me what generative adversarial networks are?

Yeah, so generative adversarial networks

are a particular kind of generative model.

A generative model is a machine learning model

that can train on some set of data.

Like, so you have a collection of photos of cats

and you want to generate more photos of cats,

or you want to estimate a probability distribution over cats.

So you can ask how likely it is

that some new image is a photo of a cat.

GANs are one way of doing this.

Some generative models are good at creating new data.

Other generative models are good at estimating

that density function and telling you how likely

particular pieces of data are to come

from the same distribution as the training data.

GANs are more focused on generating samples

rather than estimating the density function.

There are some kinds of GANs like FlowGAN that can do both,

but mostly GANs are about generating samples,

generating new photos of cats that look realistic.

And they do that completely from scratch.

It’s analogous to human imagination.

When a GAN creates a new image of a cat,

it’s using a neural network to produce a cat

that has not existed before.

It isn’t doing something like compositing photos together.

You’re not literally taking the eye off of one cat

and the ear off of another cat.

It’s more of this digestive process

where the neural net trains in a lot of data

and comes up with some representation

of the probability distribution

and generates entirely new cats.

There are a lot of different ways

of building a generative model.

What’s specific to GANs is that we have a two player game

in the game theoretic sense.

And as the players in this game compete,

one of them becomes able to generate realistic data.

The first player is called the generator.

It produces output data such as just images, for example.

And at the start of the learning process,

it’ll just produce completely random images.

The other player is called the discriminator.

The discriminator takes images as input

and guesses whether they’re real or fake.

You train it both on real data,

so photos that come from your training set,

actual photos of cats,

and you train it to say that those are real.

You also train it on images

that come from the generator network

and you train it to say that those are fake.

As the two players compete in this game,

the discriminator tries to become better

at recognizing whether images are real or fake.

And the generator becomes better

at fooling the discriminator into thinking

that its outputs are real.

And you can analyze this through the language of game theory

and find that there’s a Nash equilibrium

where the generator has captured

the correct probability distribution.

So in the cat example,

it makes perfectly realistic cat photos.

And the discriminator is unable to do better

than random guessing

because all the samples coming from both the data

and the generator look equally likely

to have come from either source.

So do you ever sit back

and does it just blow your mind that this thing works?

So from very,

so it’s able to estimate that density function

enough to generate realistic images.

I mean, does it, yeah.

Do you ever sit back and think how does this even,

why, this is quite incredible,

especially where GANs have gone in terms of realism.

Yeah, and not just to flatter my own work,

but generative models,

all of them have this property that

if they really did what we ask them to do,

they would do nothing but memorize the training data.

Right, exactly.

Models that are based on maximizing the likelihood,

the way that you obtain the maximum likelihood

for a specific training set

is you assign all of your probability mass

to the training examples and nowhere else.

For GANs, the game is played using a training set.

So the way that you become unbeatable in the game

is you literally memorize training examples.

One of my former interns wrote a paper,

his name is Vaishnav Nagarajan,

and he showed that it’s actually hard for the generator

to memorize the training data,

hard in a statistical learning theory sense,

that you can actually create reasons

for why it would require quite a lot of learning steps

and a lot of observations of different latent variables

before you could memorize the training data.

That still doesn’t really explain why

when you produce samples that are new,

why do you get compelling images

rather than just garbage

that’s different from the training set.

And I don’t think we really have a good answer for that,

especially if you think about

how many possible images are out there

and how few images the generative model sees

during training.

It seems just unreasonable

that generative models create new images as well as they do,

especially considering that we’re basically

training them to memorize rather than generalize.

I think part of the answer is

there’s a paper called Deep Image Prior

where they show that you can take a convolutional net

and you don’t even need to learn

the parameters of it at all,

you just use the model architecture.

And it’s already useful for things like inpainting images.

I think that shows us

that the convolutional network architecture

captures something really important

about the structure of images.

And we don’t need to actually use the learning

to capture all the information

coming out of the convolutional net.

That would imply that it would be much harder

to make generative models in other domains.

So far, we’re able to make reasonable speech models

and things like that.

But to be honest, we haven’t actually explored

a whole lot of different data sets all that much.

We don’t, for example, see a lot of deep learning models

of like biology data sets

where you have lots of microarrays measuring

the amount of different enzymes and things like that.

So we may find that some of the progress

that we’ve seen for images and speech

turns out to really rely heavily on the model architecture.

And we were able to do what we did for vision

by trying to reverse engineer the human visual system.

And maybe it’ll turn out that we can’t just use

that same trick for arbitrary kinds of data.

Right, so there’s aspect to the human vision system,

the hardware of it, that makes it without learning,

without cognition, just makes it really effective

at detecting the patterns we see in the visual world.

Yeah.

Yeah, that’s really interesting.

What, in a big, quick overview,

in your view, what types of GANs are there

and what other generative models besides GANs are there?

Yeah, so it’s maybe a little bit easier to start

with what kinds of generative models are there

other than GANs.

So most generative models are likelihood based

where to train them, you have a model that tells you

how much probability it assigns to a particular example

and you just maximize the probability assigned

to all the training examples.

It turns out that it’s hard to design a model

that can create really complicated images

or really complicated audio waveforms

and still have it be possible to estimate

the likelihood function from a computational point of view.

Most interesting models that you would just write down

intuitively, it turns out that it’s almost impossible

to calculate the amount of probability they assign

to a particular point.

So there’s a few different schools of generative models

in the likelihood family.

One approach is to very carefully design the model

so that it is computationally tractable

to measure the density it assigns to a particular point.

So there are things like autoregressive models,

like PixelCNN, those basically break down

the probability distribution into a product

over every single feature.

So for an image, you estimate the probability

of each pixel given all of the pixels that came before it.

There’s tricks where if you want to measure

the density function, you can actually calculate

the density for all these pixels more or less in parallel.

Generating the image still tends to require you

to go one pixel at a time, and that can be very slow.

But there are, again, tricks for doing this

in a hierarchical pattern where you can keep

the runtime under control.

Are the quality of the images it generates,

putting runtime aside, pretty good?

They’re reasonable, yeah.

I would say a lot of the best results

are from GANs these days, but it can be hard to tell

how much of that is based on who’s studying

which type of algorithm, if that makes sense.

The amount of effort invested in a particular.

Yeah, or like the kind of expertise.

So a lot of people who’ve traditionally been excited

about graphics or art and things like that

have gotten interested in GANs.

And to some extent, it’s hard to tell

are GANs doing better because they have a lot

of graphics and art experts behind them,

or are GANs doing better because they’re more

computationally efficient, or are GANs doing better

because they prioritize the realism of samples

over the accuracy of the density function.

I think all of those are potentially valid explanations,

and it’s hard to tell.

So can you give a brief history of GANs from 2014?

Were you paper 13?

Yeah, so a few highlights.

In the first paper, we just showed

that GANs basically work.

If you look back at the samples we had now,

they look terrible.

On the CIFAR 10 data set,

you can’t even recognize objects in them.

Your paper, sorry, you used CIFAR 10?

We used MNIST, which is little handwritten digits.

We used the Toronto Face database,

which is small grayscale photos of faces.

We did have recognizable faces.

My colleague Bing Xu put together

the first GAN face model for that paper.

We also had the CIFAR 10 data set,

which is things like very small 32 by 32 pixels

of cars and cats and dogs.

For that, we didn’t get recognizable objects,

but all the deep learning people back then

were really used to looking at these failed samples

and kind of reading them like tea leaves.

And people who are used to reading the tea leaves

recognize that our tea leaves at least look different.

Maybe not necessarily better,

but there was something unusual about them.

And that got a lot of us excited.

One of the next really big steps was LAPGAN

by Emily Denton and Sumit Chintala at Facebook AI Research,

where they actually got really good high resolution photos

working with GANs for the first time.

They had a complicated system

where they generated the image starting at low res

and then scaling up to high res,

but they were able to get it to work.

And then in 2015, I believe later that same year,

Alec Radford and Sumit Chintala and Luke Metz

published the DCGAN paper,

which it stands for deep convolutional GAN.

It’s kind of a non unique name

because these days basically all GANs

and even some before that were deep and convolutional,

but they just kind of picked a name

for a really great recipe

where they were able to actually using only one model

instead of a multi step process,

actually generate realistic images of faces

and things like that.

That was sort of like the beginning

of the Cambrian explosion of GANs.

Like once you had animals that had a backbone,

you suddenly got lots of different versions of fish

and four legged animals and things like that.

So DCGAN became kind of the backbone

for many different models that came out.

It’s used as a baseline even still.

Yeah, yeah.

And so from there,

I would say some interesting things we’ve seen

are there’s a lot you can say

about how just the quality

of standard image generation GANs has increased,

but what’s also maybe more interesting

on an intellectual level

is how the things you can use GANs for has also changed.

One thing is that you can use them to learn classifiers

without having to have class labels

for every example in your training set.

So that’s called semi supervised learning.

My colleague at OpenAI, Tim Solomons,

who’s at Brain now,

wrote a paper called Improve Techniques for Training GANs.

I’m a coauthor on this paper,

but I can’t claim any credit for this particular part.

One thing he showed in the paper

is that you can take the GAN discriminator

and use it as a classifier that actually tells you,

this image is a cat, this image is a dog,

this image is a car, this image is a truck, and so on.

Not just to say whether the image is real or fake,

but if it is real to say specifically

what kind of object it is.

And he found that you can train these classifiers

with far fewer labeled examples

than traditional classifiers.

So if you supervise based on also

not just your discrimination ability,

but your ability to classify,

you’re going to do much,

you’re going to converge much faster

to being effective at being a discriminator.

Yeah.

So for example, for the MNIST dataset,

you want to look at an image of a handwritten digit

and say whether it’s a zero, a one, or a two, and so on.

To get down to less than 1% accuracy

required around 60,000 examples

until maybe about 2014 or so.

In 2016 with this semi supervised GAN project,

Tim was able to get below 1% error

using only 100 labeled examples.

So that was about a 600X decrease

in the amount of labels that he needed.

He’s still using more images than that,

but he doesn’t need to have each of them labeled

as this one’s a one, this one’s a two,

this one’s a zero, and so on.

Then to be able to,

for GANs to be able to generate recognizable objects,

so objects from a particular class,

you still need labeled data

because you need to know what it means

to be a particular class cat, dog.

How do you think we can move away from that?

Yeah, some researchers at Brain Zurich

actually just released a really great paper

on semi supervised GANs

where their goal isn’t to classify,

it’s to make recognizable objects

despite not having a lot of labeled data.

They were working off of DeepMind’s BigGAN project

and they showed that they can match the performance

of BigGAN using only 10%, I believe,

of the labels.

BigGAN was trained on the ImageNet data set,

which is about 1.2 million images

and had all of them labeled.

This latest project from Brain Zurich

shows that they’re able to get away

with only having about 10% of the images labeled.

And they do that essentially using a clustering algorithm

where the discriminator learns

to assign the objects to groups

and then this understanding that objects can be grouped

into similar types helps it to form more realistic ideas

of what should be appearing in the image

because it knows that every image it creates

has to come from one of these archetypal groups

rather than just being some arbitrary image.

If you train a GAN with no class labels,

you tend to get things that look sort of like grass

or water or brick or dirt,

but without necessarily a lot going on in them.

And I think that’s partly because

if you look at a large ImageNet image,

the object doesn’t necessarily occupy the whole image.

And so you learn to create realistic sets of pixels,

but you don’t necessarily learn

that the object is the star of the show

and you want it to be in every image you make.

Yeah, I’ve heard you talk about the horse,

the zebra cycle GAN mapping

and how it turns out, again, thought provoking

that horses are usually on grass

and zebras are usually on drier terrain.

So when you’re doing that kind of generation,

you’re going to end up generating greener horses

or whatever, so those are connected together.

It’s not just, you’re not able to segment,

be able to generate in a segment away.

So are there other types of games you come across

in your mind that neural networks can play

with each other to be able to solve problems?

Yeah, the one that I spend most of my time on

is in security.

You can model most interactions as a game

where there’s attackers trying to break your system

and you’re the defender trying to build a resilient system.

There’s also domain adversarial learning,

which is an approach to domain adaptation

that looks really a lot like GANs.

The authors had the idea before the GAN paper came out,

their paper came out a little bit later

and they’re very nice and cited the GAN paper,

but I know that they actually had the idea

before it came out.

Domain adaptation is when you want to train

a machine learning model in one setting called a domain

and then deploy it in another domain later.

And you would like it to perform well in the new domain,

even though the new domain is different

from how it was trained.

So for example, you might want to train

on a really clean image data set like ImageNet,

but then deploy on users phones

where the user is taking pictures in the dark

and pictures while moving quickly

and just pictures that aren’t really centered

or composed all that well.

When you take a normal machine learning model,

it often degrades really badly

when you move to the new domain

because it looks so different

from what the model was trained on.

Domain adaptation algorithms try to smooth out that gap

and the domain adversarial approach

is based on training a feature extractor

where the features have the same statistics

regardless of which domain you extracted them on.

So in the domain adversarial game,

you have one player that’s a feature extractor

and another player that’s a domain recognizer.

The domain recognizer wants to look at the output

of the feature extractor

and guess which of the two domains the features came from.

So it’s a lot like the real versus fake discriminator

in GANs and then the feature extractor,

you can think of as loosely analogous

to the generator in GANs,

except what it’s trying to do here

is both fool the domain recognizer

into not knowing which domain the data came from

and also extract features that are good for classification.

So at the end of the day,

in the cases where it works out,

you can actually get features

that work about the same in both domains.

Sometimes this has a drawback

where in order to make things work the same in both domains,

it just gets worse at the first one.

But there are a lot of cases

where it actually works out well on both.

So do you think of GANs being useful

in the context of data augmentation?

Yeah, one thing you could hope for with GANs

is you could imagine I’ve got a limited training set

and I’d like to make more training data

to train something else like a classifier.

You could train the GAN on the training set

and then create more data

and then maybe the classifier

would perform better on the test set

after training on this bigger GAN generated data set.

So that’s the simplest version

of something you might hope would work.

I’ve never heard of that particular approach working,

but I think there’s some closely related things

that I think could work in the future

and some that actually already have worked.

So if we think a little bit about what we’d be hoping for

if we use the GAN to make more training data,

we’re hoping that the GAN will generalize to new examples

better than the classifier would have generalized

if it was trained on the same data.

And I don’t know of any reason to believe

that the GAN would generalize better

than the classifier would,

but what we might hope for

is that the GAN could generalize differently

from a specific classifier.

So one thing I think is worth trying

that I haven’t personally tried but someone could try is

what if you trained a whole lot of different

generative models on the same training set,

create samples from all of them

and then train a classifier on that?

Because each of the generative models

might generalize in a slightly different way.

They might capture many different axes of variation

that one individual model wouldn’t

and then the classifier can capture all of those ideas

by training in all of their data.

So it’d be a little bit like making

an ensemble of classifiers.

And I think that…

Ensemble of GANs in a way.

I think that could generalize better.

The other thing that GANs are really good for

is not necessarily generating new data

that’s exactly like what you already have,

but by generating new data that has different properties

from the data you already had.

One thing that you can do is you can create

differentially private data.

So suppose that you have something like medical records

and you don’t want to train a classifier

on the medical records and then publish the classifier

because someone might be able to reverse engineer

some of the medical records you trained on.

There’s a paper from Casey Green’s lab

that shows how you can train a GAN

using differential privacy.

And then the samples from the GAN

still have the same differential privacy guarantees

as the parameters of the GAN.

So you can make fake patient data

for other researchers to use.

And they can do almost anything they want with that data

because it doesn’t come from real people.

And the differential privacy mechanism

gives you clear guarantees

on how much the original people’s data has been protected.

That’s really interesting, actually.

I haven’t heard you talk about that before.

In terms of fairness, I’ve seen from AAAI,

your talk, how can adversarial machine learning

help models be more fair with respect to sensitive variables?

Yeah, so there’s a paper from Amos Starkey’s lab

about how to learn machine learning models

that are incapable of using specific variables.

So say, for example, you wanted to make predictions

that are not affected by gender.

It isn’t enough to just leave gender

out of the input to the model.

You can often infer gender

from a lot of other characteristics.

Like say that you have the person’s name,

but you’re not told their gender.

Well, if their name is Ian, they’re kind of obviously a man.

So what you’d like to do is make a machine learning model

that can still take in a lot of different attributes

and make a really accurate informed prediction,

but be confident that it isn’t reverse engineering gender

or another sensitive variable internally.

You can do that using something very similar

to the domain adversarial approach,

where you have one player that’s a feature extractor

and another player that’s a feature analyzer.

And you want to make sure that the feature analyzer

is not able to guess the value of the sensitive variable

that you’re trying to keep private.

Right, that’s, yeah, I love this approach.

So yeah, with the feature,

you’re not able to infer the sensitive variables.

Brilliant, that’s quite brilliant and simple actually.

Another way I think that GANs in particular

could be used for fairness

would be to make something like a CycleGAN,

where you can take data from one domain

and convert it into another.

We’ve seen CycleGAN turning horses into zebras.

We’ve seen other unsupervised GANs made by Mingyu Liu

doing things like turning day photos into night photos.

I think for fairness,

you could imagine taking records for people in one group

and transforming them into analogous people in another group

and testing to see if they’re treated equitably

across those two groups.

There’s a lot of things that’d be hard to get right

to make sure that the conversion process itself is fair.

And I don’t think it’s anywhere near

something that we could actually use yet,

but if you could design that conversion process

very carefully, it might give you a way of doing audits

where you say, what if we took people from this group,

converted them into equivalent people in another group,

does the system actually treat them how it ought to?

That’s also really interesting.

You know, in popular press and in general,

in our imagination, you think,

well, GANs are able to generate data

and you start to think about deep fakes

or being able to sort of maliciously generate data

that fakes the identity of other people.

Is this something of a concern to you?

Is this something, if you look 10, 20 years into the future,

is that something that pops up in your work,

in the work of the community

that’s working on generating models?

I’m a lot less concerned about 20 years from now

than the next few years.

I think there’ll be a kind of bumpy cultural transition

as people encounter this idea

that there can be very realistic videos

and audio that aren’t real.

I think 20 years from now,

people will mostly understand

that you shouldn’t believe something is real

just because you saw a video of it.

People will expect to see

that it’s been cryptographically signed

or have some other mechanism to make them believe

that the content is real.

There’s already people working on this.

Like there’s a startup called Truepick

that provides a lot of mechanisms

for authenticating that an image is real.

They’re maybe not quite up to having a state actor

try to evade their verification techniques,

but it’s something that people are already working on

and I think we’ll get right eventually.

So you think authentication will eventually win out.

So being able to authenticate that this is real

and this is not.

Yeah.

As opposed to GANs just getting better and better

or generative models being able to get better and better

to where the nature of what is real is normal.

I don’t think we’ll ever be able

to look at the pixels of a photo

and tell you for sure that it’s real or not real.

And I think it would actually be somewhat dangerous

to rely on that approach too much.

If you make a really good fake detector

and then someone’s able to fool your fake detector

and your fake detector says this image is not fake,

then it’s even more credible

than if you’ve never made a fake detector

in the first place.

What I do think we’ll get to is systems

that we can kind of use behind the scenes

to make estimates of what’s going on

and maybe not like use them in court

for a definitive analysis.

I also think we will likely get better authentication systems

where, imagine that every phone cryptographically signs

everything that comes out of it.

You wouldn’t be able to conclusively tell

that an image was real,

but you would be able to tell somebody

who knew the appropriate private key for this phone

was actually able to sign this image

and upload it to this server at this timestamp.

Okay, so you could imagine maybe you make phones

that have the private keys hardware embedded in them.

If like a state security agency

really wants to infiltrate the company,

they could probably plant a private key of their choice

or break open the chip and learn the private key

or something like that.

But it would make it a lot harder

for an adversary with fewer resources to fake things.

For most of us it would be okay.

So you mentioned the beer and the bar and the new ideas.

You were able to implement this

or come up with this new idea pretty quickly

and implement it pretty quickly.

Do you think there’s still many such groundbreaking ideas

in deep learning that could be developed so quickly?

Yeah, I do think that there are a lot of ideas

that can be developed really quickly.

GANs were probably a little bit of an outlier

on the whole like one hour timescale.

But just in terms of like low resource ideas

where you do something really different

on the algorithm scale and get a big payback.

I think it’s not as likely that you’ll see that

in terms of things like core machine learning technologies

like a better classifier

or a better reinforcement learning algorithm

or a better generative model.

If I had the GAN idea today,

it would be a lot harder to prove that it was useful

than it was back in 2014

because I would need to get it running

on something like ImageNet or Celeb A at high resolution.

You know, those take a while to train.

You couldn’t train it in an hour

and know that it was something really new and exciting.

Back in 2014, training on MNIST was enough.

But there are other areas of machine learning

where I think a new idea

could actually be developed really quickly

with low resources.

What’s your intuition about what areas

of machine learning are ripe for this?

Yeah, so I think fairness and interpretability

are areas where we just really don’t have any idea

how anything should be done yet.

Like for interpretability,

I don’t think we even have the right definitions.

And even just defining a really useful concept,

you don’t even need to run any experiments,

could have a huge impact on the field.

We’ve seen that, for example, in differential privacy

that Cynthia Dwork and her collaborators

made this technical definition of privacy

where before a lot of things were really mushy.

And then with that definition,

you could actually design randomized algorithms

for accessing databases and guarantee

that they preserved individual people’s privacy

in like a mathematical quantitative sense.

Right now, we all talk a lot about

how interpretable different machine learning algorithms are,

but it’s really just people’s opinion.

And everybody probably has a different idea

of what interpretability means in their head.

If we could define some concept related to interpretability

that’s actually measurable,

that would be a huge leap forward

even without a new algorithm that increases that quantity.

And also once we had the definition of differential privacy,

it was fast to get the algorithms that guaranteed it.

So you could imagine once we have definitions

of good concepts and interpretability,

we might be able to provide the algorithms

that have the interpretability guarantees quickly too.

So what do you think it takes to build a system

with human level intelligence

as we quickly venture into the philosophical?

So artificial general intelligence, what do you think it takes?

I think that it definitely takes better environments

than we currently have for training agents

that we want them to have

a really wide diversity of experiences.

I also think it’s gonna take really a lot of computation.

It’s hard to imagine exactly how much.

So you’re optimistic about simulation,

simulating a variety of environments as the path forward?

I think it’s a necessary ingredient.

Yeah, I don’t think that we’re going to get

to artificial general intelligence

by training on fixed data sets

or by thinking really hard about the problem.

I think that the agent really needs to interact

and have a variety of experiences within the same lifespan.

And today we have many different models

that can each do one thing.

And we tend to train them on one data set

or one RL environment.

Sometimes there are actually papers

about getting one set of parameters to perform well

in many different RL environments.

But we don’t really have anything like an agent

that goes seamlessly from one type of experience to another

and really integrates all the different things

that it does over the course of its life.

When we do see multi agent environments,

they tend to be,

or so many multi environment agents,

they tend to be similar environments.

Like all of them are playing like an action based video game.

We don’t really have an agent that goes from

playing a video game to like reading the Wall Street Journal

to predicting how effective a molecule will be as a drug

or something like that.

What do you think is a good test for intelligence

in your view?

There’s been a lot of benchmarks started with the,

with Alan Turing,

natural conversation being a good benchmark for intelligence.

What would Ian Goodfellow sit back

and be really damn impressed

if a system was able to accomplish?

Something that doesn’t take a lot of glue

from human engineers.

So imagine that instead of having to

go to the CIFAR website and download CIFAR 10

and then write a Python script to parse it and all that,

you could just point an agent at the CIFAR 10 problem

and it downloads and extracts the data

and trains a model and starts giving you predictions.

I feel like something that doesn’t need to have

every step of the pipeline assembled for it,

definitely understands what it’s doing.

Is AutoML moving into that direction

or are you thinking way even bigger?

AutoML has mostly been moving toward,

once we’ve built all the glue,

can the machine learning system

design the architecture really well?

And so I’m more of saying like,

if something knows how to pre process the data

so that it successfully accomplishes the task,

then it would be very hard to argue

that it doesn’t truly understand the task

in some fundamental sense.

And I don’t necessarily know that that’s like

the philosophical definition of intelligence,

but that’s something that would be really cool to build

that would be really useful and would impress me

and would convince me that we’ve made a step forward

in real AI.

So you give it like the URL for Wikipedia

and then next day expect it to be able to solve CIFAR 10.

Or like you type in a paragraph

explaining what you want it to do

and it figures out what web searches it should run

and downloads all the necessary ingredients.

So you have a very clear, calm way of speaking,

no ums, easy to edit.

I’ve seen comments for both you and I

have been identified as both potentially being robots.

If you have to prove to the world that you are indeed human,

how would you do it?

I can understand thinking that I’m a robot.

It’s the flip side of the Turing test, I think.

Yeah, yeah, the prove your human test.

Intellectually, so you have to…

Is there something that’s truly unique in your mind?

Does it go back to just natural language again?

Just being able to talk the way out of it.

Proving that I’m not a robot with today’s technology.

Yeah, that’s pretty straightforward.

Like my conversation today hasn’t veered off

into talking about the stock market or something

because of my training data.

But I guess more generally trying to prove

that something is real from the content alone

is incredibly hard.

That’s one of the main things I’ve gotten

out of my GAN research,

that you can simulate almost anything.

And so you have to really step back to a separate channel

to prove that something is real.

So like, I guess I should have had myself

stamped on a blockchain when I was born or something,

but I didn’t do that.

So according to my own research methodology,

there’s just no way to know at this point.

So what, last question, problem stands out for you

that you’re really excited about challenging

in the near future?

So I think resistance to adversarial examples,

figuring out how to make machine learning secure

against an adversary who wants to interfere

and control it, that is one of the most important things

researchers today could solve.

In all domains, image, language, driving, and everything.

I guess I’m most concerned about domains

we haven’t really encountered yet.

Like imagine 20 years from now,

when we’re using advanced AIs to do things

we haven’t even thought of yet.

Like if you ask people,

what are the important problems in security of phones

in like 2002?

I don’t think we would have anticipated

that we’re using them for nearly as many things

as we’re using them for today.

I think it’s gonna be like that with AI

that you can kind of try to speculate

about where it’s going,

but really the business opportunities

that end up taking off would be hard

to predict ahead of time.

What you can predict ahead of time

is that almost anything you can do with machine learning,

you would like to make sure

that people can’t get it to do what they want

rather than what you want,

just by showing it a funny QR code

or a funny input pattern.

And you think that the set of methodology to do that

can be bigger than any one domain?

I think so, yeah.

Yeah, like one methodology that I think is,

not a specific methodology,

but like a category of solutions

that I’m excited about today is making dynamic models

that change every time they make a prediction.

So right now we tend to train models

and then after they’re trained, we freeze them

and we just use the same rule

to classify everything that comes in from then on.

That’s really a sitting duck from a security point of view.

If you always output the same answer for the same input,

then people can just run inputs through

until they find a mistake that benefits them.

And then they use the same mistake

over and over and over again.

I think having a model that updates its predictions

so that it’s harder to predict what you’re gonna get

will make it harder for an adversary

to really take control of the system

and make it do what they want it to do.

Yeah, models that maintain a bit of a sense of mystery

about them, because they always keep changing.

Ian, thanks so much for talking today, it was awesome.

Thank you for coming in, it’s great to see you.