AI transcript
This is an iHeart podcast.
If I were going to pick one paper from the past decade that had the biggest impact on the world, I would choose one called Attention is All You Need, published in 2017.
That paper basically invented transformer models.
You’ve almost certainly used a transformer model if you have used ChatGPT or Gemini or Claude or DeepSeek.
In fact, the T in ChatGPT stands for transformer.
And transformer models have turned out to be wildly useful, not just at generating language, but also at everything from generating images to predicting what proteins will look like.
In fact, transformers are so ubiquitous and so powerful that it’s easy to forget that some guy just thought them up.
But in fact, some guy did just think up transformers, and I’m talking to him today on the show.
I’m Jacob Goldstein, and this is What’s Your Problem, the show where I talk to people who are trying to make technological progress.
My guest today is Jakob Uskorej.
And just to be clear, Jakob was one of several co-authors on that transformer paper.
And on top of that, lots of other researchers were working on related things at the same time.
So a lot of people were working on this.
But the key idea did seem to come from Jakob.
Today, Jakob is the CEO of Inceptive.
That’s a company that he co-founded to use AI to develop new kinds of medicine.
And the company is particularly focused on RNA.
We talked about his work at Inceptive in the second part of our conversation.
In the first part, we talked about his work on transformer models.
At the time he started working on the idea for transformers—this is around a decade ago now—there were a couple of big problems with existing language models.
For one thing, they were slow.
They were, in fact, so slow that they could not even keep up with all the new training data that was becoming available.
A second problem?
They struggled with what are called long-range dependencies.
Basically, in language, that’s relationships between words that are far apart from each other in a sentence.
So to start, I asked Jakob for an example we could use to discuss these problems, and also how he came up with his big idea for how to solve them.
So pick a sentence that’s going to be a good object lesson for us.
Okay, so we could have—the frog didn’t cross the road because it was too tired.
Okay, so we got our sentence.
Yep.
How would the sort of big, powerful, but slow-to-train algorithm in 2015 have processed that sentence?
So basically, it would have walked through that sentence word by word.
And so it would walk through the sentence left to right.
The frog did not cross the road because it was too tired.
Which is logical, which is how I would think a system would work.
It’s more or less how we read, right?
It’s how we read, but it’s not necessarily how we understand.
Uh-huh.
That is actually one of the integral, I would say, for what we then—how we then went about trying to speed this all up.
I love that.
I want you to say more about it.
When you say it’s not how we understand, what do you mean?
So, on one hand, right, linearity of time forces us to almost always feel that we’re communicating language in order and just linearly.
It actually turns out that that’s not really how we read, not even in terms of our saccades, in terms of our eye movements.
We actually do jump back and forth quite a bit while reading.
Uh-huh.
And if you look at conversations, you also have highly nonlinear elements where there’s repetition, there’s reference, there’s basically different flavors of interruption.
But sure, by and large, right, we would say we certainly write them left to right, right?
So, if you write a proper text, you don’t write it as you would read it, and you also don’t write it as you would talk about it.
You do write it in one linear order.
Now, as we read this and as we understand this, we actually form groups of words that then form meaning, right?
So, an example of that is, you know, adjective noun, right?
It’s—or say, in this case, an article noun.
It’s not a frog, it’s the frog, right?
We could have also said it’s the green frog or the lazy frog.
Right.
Language has a structure, right?
And there are—things can modify other things, and things can modify the modifiers.
Exactly, exactly.
But the interesting thing now is that structure, as a tree-structured, clean hierarchy, only tells you half the story.
There’s so many exceptions where statistical dependencies, where modification actually happens at a distance.
So, okay, so just to bring this back to your sample sentence, the frog didn’t cross the road because it was too tired.
That word it is actually quite far from the word frog.
And if you’re an AI going from left to right, you may well get confused there, right?
You may think it refers to road instead of to frog.
So this is one of the problems you were trying to solve.
And then the other one you were mentioning before, which is these models were just slow, because after each word, the model just recalculates what everything means.
And that just takes a long time.
They can’t go fast enough.
Exactly.
It takes a long time, and it doesn’t play to the strengths of the computers, of the accelerators that we’re using there.
And when you say accelerators, I know Google has their own chips, but basically we mean GPUs now, right?
We mean GPUs.
We mean the chips that NVIDIA sells.
What is the nature of those particular chips?
Exactly.
So the nature of those particular chips is that instead of doing a broad variety of complex computations in sequence, they are incredibly good.
They excel at performing many, many, many simple computations in parallel.
And so what this hierarchical or semi-hierarchical nature of language enables you to do is instead of having, so to speak, one place where you read the current word, you could now imagine you actually read every, you look at everything at the same time, and you apply many simple operations at the same time to each position in your sentence.
Uh-huh. So this is the big idea.
I just want to pause here because this is it, right? This is the breakthrough happening.
Yes.
It’s basically, what if instead of reading the sentence one word at a time from left to right, we read the whole thing all at once?
All at once. And now the problem is, clearly something’s got to give, right? So there’s no free lunch in that sense.
You have to now simplify what you can do at every position when you do this all in parallel.
Uh-huh.
But you can now afford to do this a bunch of times after another and revise it over time or over these steps.
And so instead of walking through the sentence from beginning to end, rather, an average sentence has like 20 words or so, average sentence in prose, instead of walking those 20 positions, what you’re doing is you’re looking at every word at the same time, but in a simpler way.
But now you can do that maybe five or six times, revising your understanding. And that turns out is faster, way faster on GPUs. And because of this hierarchical nature of language, it’s also better.
So you have this idea. And as I read the little note on the paper, it was in fact your idea. I know you were working with a team, but the paper credits you with the idea. So let’s take this idea, this basic idea of look at the whole input sentence all at once a few times and apply it to our frog sentence. Give me that frog sentence again.
The frog did not cross the road because it was too tired.
Good. Tired is good because that’s unambiguous. Hot could be either one. It could be the road or the frog, right?
Hot could be either one, exactly, yes. In fact, hot could actually be either one.
And non-referential, and non-referential because it was too hot outside.
Outside, it could be any of three things, the weather or the frog or the road.
Exactly.
I love that. Tired solves the problem. So your model, this new way of doing things, how does it parse that sentence? What does it do?
So basically, let’s look at the word it and look at it in every single step of these, you know, say a handful of times repeated operation.
Imagine you’re looking at this word it, that’s the one that you are now trying to understand better, and you now compare it to every other word in the sentence.
So you compare it to the, to frog, to did not cross the road because too and tired, was too and tired.
And initially, in the first pass already, a very simple insight the model can fairly easily learn is that it could be strongly informed by frog, by road.
by nothing, but not so, by to or by the, or maybe only to a certain extent by was.
But if you want to know more about what it denotes, then it could be, you know, it could be informed by all of these.
And just to be clear, that sort of understanding arises because it has trained in this way on lots of data.
It’s encountering a new sentence after reading lots of other sentences with lots of pronouns with different possible antecedents, yeah.
Exactly, exactly.
So now, the interesting thing is that which of the two it actually refers to doesn’t depend only on what those other two words are.
And this is why you need these subsequent steps because, so let’s start with the first step.
So what now happens is that, say the model identifies frog and road could have a lot to do with the word it.
So now you basically copy some information from both frog and road over to it.
And you don’t just copy it, you kind of transform it also on the way, but you refine your understanding of it.
And this is all learned.
This is not given by rules or, you know, in any way pre-specified.
Right, just by training on lots of libraries.
Just by training, this emerges, exactly.
And so that sort of the meaning of it after this first step is kind of influenced by both frog and road.
Yes, both frog and road.
Okay, so now we repeat this operation again.
And we now know that it is unsure, or the model basically now has this kind of superposition, right?
It could be road, it could be frog.
But now, in the next step, it also looks at tired.
And somehow the model has learned that when it means something inanimate, that tired is not the thing.
And so maybe in context of tired, it is more likely to refer to frog.
And now you know, well, it is more likely, or now maybe the model has figured out already, maybe it needs a bit more, a few more iterations,
that it is most likely to refer to frog because of the presence of tired.
So it has solved the problem.
But it has solved the problem.
So you have this idea, you try it out.
There’s a detail that you mentioned that’s kind of fun, and we kind of skipped it.
But you mentioned that another one of the co-authors, who has also gone on to do very big things,
was about to leave Google when you sort of want to test this idea.
And that fact that he was about to leave Google was actually important to the history of this idea.
Tell me about that.
It was important.
So this Ilya Abdullah-Sushin, he was, at the time that this started to gain any kind of speed,
Ilya was managing a good chunk of my organization.
And the moment he really made the decision to leave the company, he had to wait, ultimately, for his co-founder
and for them to then actually get going together in earnest.
And so he had a few months where he knew, and I also knew, that he was about to leave.
And where, you know, the right thing would, of course, be to transition his team to another manager,
which we did immediately, but where he then suddenly was in a position of having nothing to lose.
And yet, quite some time left to play with Google’s resources and do cool stuff with interesting people.
And so that’s one of those moments where suddenly your appetite for risk as a researcher just spikes, right?
Because you have, for a few more months, you have these resources at your disposal,
you’ve transitioned your responsibilities.
At that stage, you’re just like, okay, let’s try this crazy shit.
And that’s literally, in so many ways, was one of the integral catalysts.
Because that also enabled this kind of mindset of, we’re going for this now.
Whatever the reason, it still, you know, affects other people.
And so there were others who joined that collaboration really, really early on,
who I feel were much more excited and, as a result, much more likely to really work on this
and to really give it their all.
Because of his, you know, nothing left to lose, I’m going to go for this attitude at this point, right?
Was there a moment when you realized it worked?
There were actually a few moments.
And it’s interesting because, on one hand, right, it’s a very gradual thing, right?
And initially, actually, it took us many months to get to the point where we saw significant first signs of life,
of this not just being a curiosity, but really being something that would end up being competitive.
So there certainly was a moment when that started.
There was another moment when we, for the first time, had one machine translation challenge,
one language pair of the W&T task, as it’s called,
where our score, our model performed better than any other single model.
The point in time when I think all of us realized this is special
was when we not only had the best one in one of these tasks, but in multiple.
And we didn’t just have the best number.
We also, at that point, were able to establish that we’ve gotten there
with about 10 times less energy or training compute spend.
Wow.
So you do one-tenth the work, and you get a better result.
One-tenth the work, and you get a better result,
not just across one specific challenge, but across multiple,
including the hardest, or one of the harder ones, right?
And then, at that stage, we were still improving rapidly.
And then you realize, okay, this is for real.
Because it wasn’t that we had to squeeze those last little bits and pieces of gain out of it.
It was still improving fairly rapidly,
to the point where actually, by the time we actually published the paper,
we, again, reduced the compute requirements,
not quite by an entire order of magnitude, but almost, right?
So it still was getting faster and better at a pretty rapid rate.
So we had, in the paper, we had some results that were those roughly 10x faster on eight GPUs.
And what we demonstrated, in terms of quality, on those eight GPUs,
by the time we actually published the paper properly, we were able to do with one GPU.
One GPU, meaning one chip of the kind that people buy 100,000 of now?
To build a data center?
Exactly.
So the paper, actually, at the end, mentions other possible uses beyond language for this technology.
It mentions images, audio, and video, I think, explicitly.
How much were you thinking about that at the time?
Was that just like an afterthought, or were you like, hey, wait a minute, it’s not just language?
By the time it was actually published at a conference, not just the preprint, by December,
we had initial models on other modalities, on generating images.
We had the first, at that time, they were not performing that well yet, but they were rapidly
getting better.
We had the first prototypes, actually, of models working on genomic data, working on protein
structure.
That’s good foreshadowing.
Good foreshadowing as well, exactly.
But then we ended up, for a variety of reasons, we ended up, at first, focusing on applications
in computer vision.
The paper comes out, you’re working on these other applications, you’re presenting the paper,
it’s published in various forms.
What’s the response like?
It was interesting because the response built in deep learning AI circles, basically, between
the preprint that I think came out in, I want to say, June 2017, and then the actual publication,
to the extent that by the time the poster session happened at the conference, there was quite
a crowd at the poster, so we had to be shoved out of the hall in which the poster session
happened by the security, and had very hoarse voices by the end of the evening.
You guys were like the Beatles of the AI conference?
I wouldn’t say that because we weren’t the Beatles, because it was really, it was still
very specific.
You were more the cool hipster band.
You were the it hipster band.
Certainly more the cool hipster band.
But it was an interesting experience because there were some folks, including some greats
in the field, who came by and said, wow, this is cool.
What has happened since has been wild, it seems.
Wild, to say the least, yes.
Is it surprising to you?
Of course, many aspects are surprising, for sure.
We definitely saw pretty early on, already back in 2018, 2019, that something really exciting
was happening here.
Now, I’m still surprised by it.
With the advent of ChatGPT, something that didn’t go way beyond those language models that
we had already seen a few years before was suddenly the world’s fastest growing consumer product.
Ever, right?
I think ever.
Ever.
Yes.
And by the way, GBT stands for Generative Pre-Train Transformer, right?
Transformer is your word.
That’s right.
So there’s an interesting, I don’t know, business side to this, right?
Which is, you were working for Google when you came up with this.
Google presumably owned the idea.
Yep.
Had intellectual property around the idea.
Has filed many a patent.
Was it just a choice Google made to let everybody use it?
Like, when you see the fastest growing consumer product in the history of the world, not only
built on this idea, but using the name, like, and it’s a different company.
That was five years later.
Five years later.
But a patent’s good for more than five years.
Is that a choice?
Is that a strategic choice?
What’s going on there?
So the choice to do it in the first place, to publish it in the first place, is really
based on and rooted in a deep conviction of Google at the time, and I’m actually pretty
sure it still is the case, that it is actually, these developments are the tide that floats all
boats, that lifts all boats.
Uh-huh.
Like a belief in progress.
A belief in progress.
Exactly.
Like a good old-fashioned belief in…
Now, it’s also the case that at the time, organizationally, that specific research arm was unusually separated
from the product organizations.
And the reason why brain, or in general, the deep learning groups, were more separated was
in part historical.
Namely, that when they started out, there were no applications, and the technology was not ready
for being applied.
And so it’s completely understandable and just, you know, a consequence of organic developments
that when this technology suddenly is on the cusp of being incredibly impactful, you’re probably
still underutilizing it internally and potentially also not yet treating it in the same way as you
would have maybe otherwise treated previous trade secrets, for example.
Because it feels like this out there research project, not like what’s going to be this consumer
product.
And to be fair, it took OpenAI, in this case, a fair amount of time to then turn this into this
product.
And most of that time, it also, from their vantage point, wasn’t a product, right?
So up until all the way through ChatGPT, OpenAI published all of their GPT developments, maybe
not all, but, you know, a very large fraction of their work on this.
Yeah, their early models, the whole models were open.
Exactly.
They were more true to their name, really, really also believing in the same thing.
And it was only really after ChatGPT and after this, to them also surprise, to a certain extent,
success, that they started to become more closed as well when it comes to scientific developments
in this space.
We’ll be back in just a minute.
Run a business and not thinking about podcasting?
Think again.
More Americans listen to podcasts than ad-supported streaming music from Spotify and Pandora.
And as the number one podcaster, iHeart’s twice as large as the next two combined.
So whatever your customers listen to, they’ll hear your message.
Plus, only iHeart can extend your message to audiences across broadcast radio.
Think podcasting can help your business?
Think iHeart.
Streaming, radio, and podcasting.
Call 844-844-iHeart to get started.
That’s 844-844-iHeart.
Let’s talk about your company.
When did you decide to start Inceptive?
The decision took a while and was influenced by events that happened over the course of
about three months, two to three months in late 2020, starting with the birth of my first
child.
So when Amre was born, two things happened.
Number one, witnessing a pregnancy and a birth during a pandemic where there’s a pathogen that’s
rapidly spreading.
And so all of that was a pretty daunting experience.
And everything went great.
But having this new human in my arms also really made me question if I couldn’t more directly
affect people’s lives positively with my work.
And so I was at the time quite confident that indirectly it would have effect also on things
like medicine, biology, et cetera.
But I was wondering, couldn’t this happen more directly if I focused more on it?
The next thing that happened was that AlphaFold 2 results at CASP-14 were published.
CASP-14 is this biannual challenge for protein structure prediction and some other related problems.
This is the protein folding problem.
And this is the protein folding problem, exactly.
So machine learning solving the protein folding problem, which had been a problem for decades,
given a chain of amino acids to predict the 3D structure of a protein.
Precisely.
And humans failed and machine learning succeeded.
Just amazing.
Yes.
It’s a great example.
Humans failed despite the fact that we actually understand the physics fundamentally, but we
still couldn’t create models that were good enough using our conceptual understanding
of the processes involved.
Yeah, you would think an algorithm would work on that one, right?
You would just think an old school set of rules.
Like, we know what the molecules look like.
We know the laws of physics.
It’s amazing that we couldn’t predict it that way, right?
All you want to know is what shape is the protein going to be?
You know all of the constituent parts.
You know every atom in it.
And you still couldn’t predict it with a set of rules, but AI, machine learning, could.
Amazing.
Yes.
And it is amazing.
Actually, when you put it like this, it’s important to point out that when we say we
understand it, we make massive oversimplifying assumptions.
Because we ignore all the other players that are present when the protein folds.
We ignore a lot of the kinetics of it because we say we know the structure, but the truth
is we don’t know all the wiggling and all the shenanigans that happen on the way there,
right?
And we don’t know about, you know, chaperone proteins that are there to influence the folding.
We don’t know around all sorts of other aspects.
I’m doing the physics one.
I’m doing the assume a frictionless plane version of protein.
Precisely.
Which is why it didn’t work.
And the beauty is that deep learning doesn’t need to make this assumption.
AI doesn’t need to make this assumption.
AI just looks at data.
And it can look at more data than any human or even humanity eventually could look at together.
It’s such a good example problem to demonstrate that these models are ready for prime time in
this field and ready for lots of applications, not just one or two, but many.
And so that happens.
They’ll be sold, exactly.
And then the third thing was that these COVID mRNA vaccines came out with astonishing 90 plus
percent efficacy.
So fast also.
Out of the gate.
How fast and how good they were is still so underrated.
Underrated.
At the beginning of the pandemic, people were like, it’ll be two or three years, and if they’re
60 percent effective, that’ll be great.
Exactly.
And so-
Everybody forgets that.
Everybody forgets it.
And when you look at it, this is a molecule family that was for, you know, most of the
most of the time that we’ve known about it, since the 60s, I suppose, we’ve treated it
like an addicted stepchild of molecular biology.
Because we’ve massed-
You’re talking about RNA in general?
RNA.
RNA in general.
Yeah.
Everybody loves DNA, right?
DNA is the movie star.
Exactly.
Exactly, exactly.
Even though now, looking back, DNA is merely, you know, the place where life takes its notes,
maybe the hard drive and the memory.
It’s the book, right?
It’s the book, right?
It’s the book.
So, but at the end of the day, it’s this molecule family that was about to save, you know, depending
on the estimate, tens of millions of lives, and in rapid time.
So all these things hold, but we have no training data to apply anything like alpha-fold to this
specific molecule family.
No training data to speak of.
We had 200,000 known protein structures at the time.
I believe, maybe optimistically, we had maybe 1,200 known RNA structures.
And on top of that, it was also fairly clear that for RNA, going directly to function would
be much, much more important because it’s, in a certain sense, a weak, less strongly structured
molecule and other aspects of the molecule might play a bigger role.
And then, on top of that, the attention that generative AI was receiving overall, also now
in the field of pharma or of medicine, was building.
And so I ended up finding myself in a conversation where a very, I’d say, wise, long-time mentor
of mine pointed out that, you know, maybe 10 years from now or so, somebody could tell my
daughter that there was this perfect storm where this macromolecule with no training data was
about to save the world and could do so much more in the direction of positively impacting
people’s lives.
We didn’t have training data.
It would be very expensive to create it.
But using the technology that I’ve been, or technologies that I’ve been working on for
the last, I don’t know, 10 plus years, and the ability, because of the attention that
people were now giving to AI in this field, the ability to raise quite a bit of money,
I, in that position, chose to stay back at my cushy dream job in big tech and not actually
take this opportunity to really positively impact people’s lives.
And that idea was not one I was willing to entertain.
You couldn’t just coast it out at Google and let somebody else go figure out RNA.
Yeah.
And it’s not just RNA.
I think RNA is a great starting point at the end of the day.
But building models that learn from, first of all, all the publicly available data that we
can possibly get our hands on, but also from data that we can reasonably effectively create
in our own lab, how to design molecules for specific functions, is something that now is
within reach and that will, in the next years and in the years to come, have completely transformational
impact on how we even think about what medicines are.
That any opportunity to speed this up, to make this happen, even just a day sooner than it could
have otherwise happened, is incredibly valuable, in my opinion.
As you’re talking about this idea that the absence of training data seems to be at the
center of it, right?
It seems to be the core problem, which makes sense, right?
Like, the reason language works so well is basically because of the internet.
I know now we’re going beyond it, but it just happened to be that there was this incredibly
giant set of natural language that became available.
We don’t have anything like that for RNA.
So are you, I mean, it’s kind of step one at Inceptive, creating the data?
Is that kind of what’s happening?
So step one at Inceptive is learning to use all the data, or was, I think we’ve made a lot
of progress in that direction, learning to use all the data that is available already, and
identify what other data we’re missing.
And then see how far we can get with just the publicly available data, and at the same
time scale up generating our own data.
And it turns out that actually, because of the nature of evolution, because of how evolution
isn’t actually incentivized to really explore the entire space of possibilities, it is almost
always a given that if you are trying to design exceptional molecules, especially ones that
are not, say, you know, natural formats, you are basically guaranteed to need novel training
data.
Yeah.
Basically, you’re saying you build RNAs that don’t exist in the world, that have therapeutic
uses, and there’s no, kind of definitionally, no training data for that, because they don’t
exist.
The funny thing is, we have a few of them, and so we have existence proofs of RNA molecules,
for example, RNA viruses, that actually exhibit incredibly complex, different functions in ourselves
that do all sorts of things that we don’t usually like, but if we could use those, you know, for
good, if we could use those, you know, in ways that would actually be aimed at fighting disease
rather than creating them, those kinds of functions, even just a small subset of them, would really
transform medicine already.
And so we know it’s possible.
What are you dreaming of when you say that?
What are you thinking of, specifically?
Okay.
So, for example, right, one estimate is that in order for COVID to infect you, you would need
potentially as few as five COVID genomes inside your organism.
That’s already it.
Five viral particles?
Five viral particles.
Yeah.
You inhale those.
You wouldn’t have to inject it.
You wouldn’t even have to swallow it.
You inhaled them.
What if we could have a medicine that worked as well as a disease, is a version of your dream.
Exactly.
Exactly.
So, at the end of the day, right, this medicine is able to spread in your body only into certain
types of organs and tissues and cells.
It does certain things there that are really quite complex, right?
Changing the cells’ behavior.
Yeah.
Again, not usually in this case in favorable ways, but still in ways that wouldn’t have
to be modified that much in order to potentially be exactly what you would need for a complex
multifactorial medicine.
And if you could make all of that happen by just inhaling five of those molecules, then,
again, that would completely change how you think about medicine, right?
You have viruses that aren’t immediately active, but that are inactive for long periods of time
in your organism.
And only under certain conditions, say, under certain immune conditions, really start being
reactivated.
Why can’t we have medicines that work in a similar way, where you actually, not only in
a vaccination sense, but where you take a medicine for a genetic predisposition for a
certain disease, that you are able to design a medicine that you can take and that waits until
the disease actually starts to develop.
And only then, and only where that disease then starts to develop, becomes active and
actually facts it.
And potentially also then alarms the doctor through a blood test.
Like for cancer cells or something.
So you have some kind of prophylactic medicine in your body, and it is encoded in such a way
that it just hangs out there like herpes, to take a pathological example.
For example, yes.
And only in certain settings does it do anything.
And those settings are, if you see a cancer cell, destroy it.
Otherwise, just sit there.
Precisely.
And if you can design those also in ways where you can just make them all go away when you
take a, say, a completely harmless small molecule, and that’s, again, entirely feasible.
Sure.
So, I mean, you’re dreaming big.
These are wonderful big, you know, science fiction-y dreams, and I hope you figure them
out.
On a practical level, what’s happening at the company right now?
How many people work there?
What are they doing?
And what have they figured out so far?
We’re around 40.
What we’re doing is really exactly what we just talked about.
We’re basically scaling data generation experiments in our lab that allow us to assess a variety
of different functions of different, mostly RNA molecules, actually mostly mRNA molecules at the
moment, that are relevant to a pretty broad variety of different diseases.
And so, this ranges from things like infectious disease vaccines to cell therapies that can be applied
in oncology or against autoimmune disease.
We have mRNAs that we hope will eventually be effective in enzyme replacement as enzyme replacement
therapies for families of, or a large family of rare diseases.
And the list goes on.
And so, we’re creating this, or growing this training data set that eventually, on top of
foundation and models that we pre-trained on all publicly available data, allow us to tune
those foundation models towards designing exceptional molecules for exactly those applications and many
more sharing similar properties.
So, you basically build new mRNA molecules and test them, and then you give that data to
your model, and presumably it tells you what to build next, or it helps you figure out what to
build next.
It’s sort of a loop in that way?
The models are definitely one interesting source for proposals, if you wish, for what to synthesize
and test next.
They’re not the only such source.
So, we basically also explore kind of in maybe less guided or heuristically guided ways.
But, exactly.
So, in some of the cases, it’s really quite iterative.
For some of those functions and for some of those modalities and diseases or disease targets,
we’re actually already at a point where our models can spit out entirely novel molecules that
really are unlike anything they’ve ever seen or we’ve ever seen in nature, that very consistently
perform quite favorably compared to pretty strong baselines by incumbents in the field.
When you say perform quite favorably compared to baselines by incumbents in the field, I mean,
does that on some level mean better than what experts would think up?
Better than what experts can think of and also better than more traditional machine learning
tools can easily produce.
It’s like that famous moment in the Go match when AlphaGo made some move that, like, no human
being ever would have thought of.
Move 37.
Yes.
So, I would say we’ve long passed the Move 37 in the sense that our understanding of the
underlying biological phenomena is so incomplete that for most of the things that we’re able
to design for, we don’t really understand why they happen.
Huh.
When you say we, do you mean at Inceptive or do you mean just medicine in general?
I would say just medicine in general.
Okay.
So, Inceptive is doing this very kind of high-level work, right?
I mean, building what will hopefully be the foundation.
What’s the right amount of time in the future to ask about?
When will we know if it works?
Do you think five years?
So, the general idea of using generative AI and similar techniques to generate therapeutics,
there are some things in clinical trials that were largely designed with AI.
As far as I know, we’re still, maybe now we have the first trials just now starting for
molecules that were truly entirely designed by AI.
As opposed to sort of selected from a library?
Selected, influenced, exactly.
Selected, adjusted, tuned, tweaked, et cetera, right?
So, that’s really still only happening just now.
Okay.
But we will see, I believe, the first success or a first success of such molecules, certainly
within the next five years.
What about more narrowly the project at Inceptive?
It’s a similar timeframe.
We should be able to get molecules into the clinic in the next few years, certainly in the
next handful of years.
Now, these will not be molecules with, where the objective that we used in their design is,
you know, even remotely as complex or the, you know, kind of the different functions that
we’re designing for are not going to be even remotely as diverse as, say, what you would find in,
because we used this example earlier in RNA virus.
These will really be more, you know, simpler.
Those will be molecules that don’t do things that we couldn’t possibly have done before,
but that do them much better in ways that are more accessible, in ways that come with less side
effects.
What biotech largely is, is they make protein drugs.
And so if you could make an mRNA drug where you put the mRNA into the body and the body makes the protein,
it wouldn’t be some crazy sleeper cell that sits in your body for 20 years or whatever.
But it might be a more practical alternative to today’s biotech drugs.
Absolutely.
So you’ve had a kind of crash course in biology in the last few years.
Yes.
And I’m curious, like, what is, what is something that has been particularly compelling or surprising
or interesting to you that you have learned about biology?
There are countless things.
The biggest one or the red thread across many of them is really just how effective life is
at finding solutions to problems that, on one hand, are incredibly robust, surprisingly robust,
and on the other hand, are so different from how we would design solutions to similar problems.
Aha.
That really, this comes back to this idea that we might just not be particularly well-equipped
in terms of cognitive capabilities to understand biology, that basically, you know, we are,
we would never think to do it this way.
And how we think to do it is oftentimes much more brittle.
Aha.
Brittle is an interesting world.
Less resilient, less able to persist under different conditions.
Exactly.
Exactly.
I mean, you know, we still haven’t built machines that can fix themselves, for one.
Which is fundamentally the miracle of being a human being.
Which is fundamentally the miracle of life.
I’m still here after going through all this.
Exactly.
Exactly.
Exactly.
And so, and of course, this is true across the scales, right?
From, you know, single cells all the way to complex organisms like ourselves.
And really just how many also very different kinds of solutions life has found and or constantly
is finding.
And you see this all over the place.
And it’s both daunting, humbling, but also incredibly inspiring when it comes to applying
AI in this area.
Because again, I think that at least so far, it’s the best tool and maybe actually the only
tool we have so far in face of this kind of complexity, really design interventions that go way beyond
what we were able to do or are able to do just based on our own conceptual understanding.
We’ll be back in a minute with the lightning round.
Run a business and not thinking about podcasting?
Think again.
More Americans listen to podcasts than ad-supported streaming music from Spotify and Pandora.
And as the number one podcaster, iHeart’s twice as large as the next two combined.
So whatever your customers listen to, they’ll hear your message.
Plus, only iHeart can extend your message to audiences across broadcast radio.
Think podcasting can help your business?
Think iHeart.
Streaming, radio, and podcasting.
Call 844-844-IHEART to get started.
That’s 844-844-IHEART.
Let’s finish with the lightning round.
As an inventor of the Transformer model, are there particular possible uses of it that
worry you slash make you sad?
I am quite concerned about the P-Doom, Doomerism, whatever you want to call it, existential fear
instilling rhetoric that is in some cases actually also promoted by people, by entities in the space.
So just to be clear, you’re not worried about the existential risk.
You’re worried about people talking about the existential risk.
I’m worried about the existential risk being inflated, or the perception being inflated to the extent
that we actually don’t look enough at some of the much more concrete and much more immediate risks.
I’m not going to say that the existential risk is zero, but that would be silly.
What is a concrete and immediate risk that is, you think, under-discussed?
These large-scale models are such effective tools in manipulating people in large numbers already
today, and it’s happening everywhere for many, many different purposes by, in some cases, benevolent,
and in many cases, malevolent actors that I really firmly believe we need to look much more
at things like enabling cryptographic certification of human-generated content, because doing that
with the machine-generated content is not going to work, but we definitely can cryptographically
certify human-generated content as such.
Basically, watermarking or something, some way to say, a human made this.
Exactly.
What would you be working on if you were not working in biology, on drug development?
Education.
Using artificial intelligence to democratize access to education.
What have you seen that has been impressive or compelling to you in that regard?
There are lots of little examples so far, and really countless.
It’s what’s happening at the Khan Academy, there are many examples of AI applied to education
problems in places like China, for example.
You have a bunch of very compelling examples in fiction, a book I really like by a guy named
Neil Stevenson, The Diamond Age, or Young Lady’s Illustrated Primer, that I recommend if
you just want to…
Everybody in AI talks about that.
Well, now they do, yeah.
Yeah, well, now they do.
You liked it before it was cool, I’m sure.
At one point, I thought it was really, really important to ensure that Neil Stevenson knows
that we are about to be able to build the primer, and so I ended up having coffee with
him to tell him.
Oh, that’s great.
So, at the end of the day, maybe the biggest inspiration there is my daughter.
She’s four and a half now, and I think she could, today, read, she can read okay, but she could
read, you know, grade school level if she had access to, you know, an AI tutor teaching her
how to read.
Does your daughter use AI?
Use, you know, AI chatbots?
Not directly without me, but we’ve actually used ChatGPT to implement an AI reading tutor
that works reasonably well.
I mean, we basically, you know, kind of as they call it now, vibe coding.
We vibe coded, and Amway wasn’t there for all of it.
It took some time, but she was there for some of it.
Oh, you vibe coded it with her?
Yeah, well, I mean, she was there, she, you know, she witnessed a good chunk of it, yes.
Although she was more interested in the image generation parts.
But yeah, we have a sketch of one that she quite enjoys.
So, that’s kind of like the extent of her at this age using AI directly.
Jakob Uskoreit is the CEO and co-founder of Inceptive, and the co-author of the paper,
attention is all you need.
Just a quick note, this is our last episode before a break of a couple of weeks,
and then we’ll be back with more episodes.
Please email us at problematpushkin.fm.
We are always looking for new guests for the show.
Today’s show was produced by Trina Menino and Gabriel Hunter-Chang.
It was edited by Alexandra Gerritsen and engineered by Sarah Brugger.
This is an iHeart Podcast.
If I were going to pick one paper from the past decade that had the biggest impact on the world, I would choose one called Attention is All You Need, published in 2017.
That paper basically invented transformer models.
You’ve almost certainly used a transformer model if you have used ChatGPT or Gemini or Claude or DeepSeek.
In fact, the T in ChatGPT stands for transformer.
And transformer models have turned out to be wildly useful, not just at generating language, but also at everything from generating images to predicting what proteins will look like.
In fact, transformers are so ubiquitous and so powerful that it’s easy to forget that some guy just thought them up.
But in fact, some guy did just think up transformers, and I’m talking to him today on the show.
I’m Jacob Goldstein, and this is What’s Your Problem, the show where I talk to people who are trying to make technological progress.
My guest today is Jakob Uskorej.
And just to be clear, Jakob was one of several co-authors on that transformer paper.
And on top of that, lots of other researchers were working on related things at the same time.
So a lot of people were working on this.
But the key idea did seem to come from Jakob.
Today, Jakob is the CEO of Inceptive.
That’s a company that he co-founded to use AI to develop new kinds of medicine.
And the company is particularly focused on RNA.
We talked about his work at Inceptive in the second part of our conversation.
In the first part, we talked about his work on transformer models.
At the time he started working on the idea for transformers—this is around a decade ago now—there were a couple of big problems with existing language models.
For one thing, they were slow.
They were, in fact, so slow that they could not even keep up with all the new training data that was becoming available.
A second problem?
They struggled with what are called long-range dependencies.
Basically, in language, that’s relationships between words that are far apart from each other in a sentence.
So to start, I asked Jakob for an example we could use to discuss these problems, and also how he came up with his big idea for how to solve them.
So pick a sentence that’s going to be a good object lesson for us.
Okay, so we could have—the frog didn’t cross the road because it was too tired.
Okay, so we got our sentence.
Yep.
How would the sort of big, powerful, but slow-to-train algorithm in 2015 have processed that sentence?
So basically, it would have walked through that sentence word by word.
And so it would walk through the sentence left to right.
The frog did not cross the road because it was too tired.
Which is logical, which is how I would think a system would work.
It’s more or less how we read, right?
It’s how we read, but it’s not necessarily how we understand.
Uh-huh.
That is actually one of the integral, I would say, for what we then—how we then went about trying to speed this all up.
I love that.
I want you to say more about it.
When you say it’s not how we understand, what do you mean?
So, on one hand, right, linearity of time forces us to almost always feel that we’re communicating language in order and just linearly.
It actually turns out that that’s not really how we read, not even in terms of our saccades, in terms of our eye movements.
We actually do jump back and forth quite a bit while reading.
Uh-huh.
And if you look at conversations, you also have highly nonlinear elements where there’s repetition, there’s reference, there’s basically different flavors of interruption.
But sure, by and large, right, we would say we certainly write them left to right, right?
So, if you write a proper text, you don’t write it as you would read it, and you also don’t write it as you would talk about it.
You do write it in one linear order.
Now, as we read this and as we understand this, we actually form groups of words that then form meaning, right?
So, an example of that is, you know, adjective noun, right?
It’s—or say, in this case, an article noun.
It’s not a frog, it’s the frog, right?
We could have also said it’s the green frog or the lazy frog.
Right.
Language has a structure, right?
And there are—things can modify other things, and things can modify the modifiers.
Exactly, exactly.
But the interesting thing now is that structure, as a tree-structured, clean hierarchy, only tells you half the story.
There’s so many exceptions where statistical dependencies, where modification actually happens at a distance.
So, okay, so just to bring this back to your sample sentence, the frog didn’t cross the road because it was too tired.
That word it is actually quite far from the word frog.
And if you’re an AI going from left to right, you may well get confused there, right?
You may think it refers to road instead of to frog.
So this is one of the problems you were trying to solve.
And then the other one you were mentioning before, which is these models were just slow, because after each word, the model just recalculates what everything means.
And that just takes a long time.
They can’t go fast enough.
Exactly.
It takes a long time, and it doesn’t play to the strengths of the computers, of the accelerators that we’re using there.
And when you say accelerators, I know Google has their own chips, but basically we mean GPUs now, right?
We mean GPUs.
We mean the chips that NVIDIA sells.
What is the nature of those particular chips?
Exactly.
So the nature of those particular chips is that instead of doing a broad variety of complex computations in sequence, they are incredibly good.
They excel at performing many, many, many simple computations in parallel.
And so what this hierarchical or semi-hierarchical nature of language enables you to do is instead of having, so to speak, one place where you read the current word, you could now imagine you actually read every, you look at everything at the same time, and you apply many simple operations at the same time to each position in your sentence.
Uh-huh. So this is the big idea.
I just want to pause here because this is it, right? This is the breakthrough happening.
Yes.
It’s basically, what if instead of reading the sentence one word at a time from left to right, we read the whole thing all at once?
All at once. And now the problem is, clearly something’s got to give, right? So there’s no free lunch in that sense.
You have to now simplify what you can do at every position when you do this all in parallel.
Uh-huh.
But you can now afford to do this a bunch of times after another and revise it over time or over these steps.
And so instead of walking through the sentence from beginning to end, rather, an average sentence has like 20 words or so, average sentence in prose, instead of walking those 20 positions, what you’re doing is you’re looking at every word at the same time, but in a simpler way.
But now you can do that maybe five or six times, revising your understanding. And that turns out is faster, way faster on GPUs. And because of this hierarchical nature of language, it’s also better.
So you have this idea. And as I read the little note on the paper, it was in fact your idea. I know you were working with a team, but the paper credits you with the idea. So let’s take this idea, this basic idea of look at the whole input sentence all at once a few times and apply it to our frog sentence. Give me that frog sentence again.
The frog did not cross the road because it was too tired.
Good. Tired is good because that’s unambiguous. Hot could be either one. It could be the road or the frog, right?
Hot could be either one, exactly, yes. In fact, hot could actually be either one.
And non-referential, and non-referential because it was too hot outside.
Outside, it could be any of three things, the weather or the frog or the road.
Exactly.
I love that. Tired solves the problem. So your model, this new way of doing things, how does it parse that sentence? What does it do?
So basically, let’s look at the word it and look at it in every single step of these, you know, say a handful of times repeated operation.
Imagine you’re looking at this word it, that’s the one that you are now trying to understand better, and you now compare it to every other word in the sentence.
So you compare it to the, to frog, to did not cross the road because too and tired, was too and tired.
And initially, in the first pass already, a very simple insight the model can fairly easily learn is that it could be strongly informed by frog, by road.
by nothing, but not so, by to or by the, or maybe only to a certain extent by was.
But if you want to know more about what it denotes, then it could be, you know, it could be informed by all of these.
And just to be clear, that sort of understanding arises because it has trained in this way on lots of data.
It’s encountering a new sentence after reading lots of other sentences with lots of pronouns with different possible antecedents, yeah.
Exactly, exactly.
So now, the interesting thing is that which of the two it actually refers to doesn’t depend only on what those other two words are.
And this is why you need these subsequent steps because, so let’s start with the first step.
So what now happens is that, say the model identifies frog and road could have a lot to do with the word it.
So now you basically copy some information from both frog and road over to it.
And you don’t just copy it, you kind of transform it also on the way, but you refine your understanding of it.
And this is all learned.
This is not given by rules or, you know, in any way pre-specified.
Right, just by training on lots of libraries.
Just by training, this emerges, exactly.
And so that sort of the meaning of it after this first step is kind of influenced by both frog and road.
Yes, both frog and road.
Okay, so now we repeat this operation again.
And we now know that it is unsure, or the model basically now has this kind of superposition, right?
It could be road, it could be frog.
But now, in the next step, it also looks at tired.
And somehow the model has learned that when it means something inanimate, that tired is not the thing.
And so maybe in context of tired, it is more likely to refer to frog.
And now you know, well, it is more likely, or now maybe the model has figured out already, maybe it needs a bit more, a few more iterations,
that it is most likely to refer to frog because of the presence of tired.
So it has solved the problem.
But it has solved the problem.
So you have this idea, you try it out.
There’s a detail that you mentioned that’s kind of fun, and we kind of skipped it.
But you mentioned that another one of the co-authors, who has also gone on to do very big things,
was about to leave Google when you sort of want to test this idea.
And that fact that he was about to leave Google was actually important to the history of this idea.
Tell me about that.
It was important.
So this Ilya Abdullah-Sushin, he was, at the time that this started to gain any kind of speed,
Ilya was managing a good chunk of my organization.
And the moment he really made the decision to leave the company, he had to wait, ultimately, for his co-founder
and for them to then actually get going together in earnest.
And so he had a few months where he knew, and I also knew, that he was about to leave.
And where, you know, the right thing would, of course, be to transition his team to another manager,
which we did immediately, but where he then suddenly was in a position of having nothing to lose.
And yet, quite some time left to play with Google’s resources and do cool stuff with interesting people.
And so that’s one of those moments where suddenly your appetite for risk as a researcher just spikes, right?
Because you have, for a few more months, you have these resources at your disposal,
you’ve transitioned your responsibilities.
At that stage, you’re just like, okay, let’s try this crazy shit.
And that’s literally, in so many ways, was one of the integral catalysts.
Because that also enabled this kind of mindset of, we’re going for this now.
Whatever the reason, it still, you know, affects other people.
And so there were others who joined that collaboration really, really early on,
who I feel were much more excited and, as a result, much more likely to really work on this
and to really give it their all.
Because of his, you know, nothing left to lose, I’m going to go for this attitude at this point, right?
Was there a moment when you realized it worked?
There were actually a few moments.
And it’s interesting because, on one hand, right, it’s a very gradual thing, right?
And initially, actually, it took us many months to get to the point where we saw significant first signs of life,
of this not just being a curiosity, but really being something that would end up being competitive.
So there certainly was a moment when that started.
There was another moment when we, for the first time, had one machine translation challenge,
one language pair of the W&T task, as it’s called,
where our score, our model performed better than any other single model.
The point in time when I think all of us realized this is special
was when we not only had the best one in one of these tasks, but in multiple.
And we didn’t just have the best number.
We also, at that point, were able to establish that we’ve gotten there
with about 10 times less energy or training compute spend.
Wow.
So you do one-tenth the work, and you get a better result.
One-tenth the work, and you get a better result,
not just across one specific challenge, but across multiple,
including the hardest, or one of the harder ones, right?
And then, at that stage, we were still improving rapidly.
And then you realize, okay, this is for real.
Because it wasn’t that we had to squeeze those last little bits and pieces of gain out of it.
It was still improving fairly rapidly,
to the point where actually, by the time we actually published the paper,
we, again, reduced the compute requirements,
not quite by an entire order of magnitude, but almost, right?
So it still was getting faster and better at a pretty rapid rate.
So we had, in the paper, we had some results that were those roughly 10x faster on eight GPUs.
And what we demonstrated, in terms of quality, on those eight GPUs,
by the time we actually published the paper properly, we were able to do with one GPU.
One GPU, meaning one chip of the kind that people buy 100,000 of now?
To build a data center?
Exactly.
So the paper, actually, at the end, mentions other possible uses beyond language for this technology.
It mentions images, audio, and video, I think, explicitly.
How much were you thinking about that at the time?
Was that just like an afterthought, or were you like, hey, wait a minute, it’s not just language?
By the time it was actually published at a conference, not just the preprint, by December,
we had initial models on other modalities, on generating images.
We had the first, at that time, they were not performing that well yet, but they were rapidly
getting better.
We had the first prototypes, actually, of models working on genomic data, working on protein
structure.
That’s good foreshadowing.
Good foreshadowing as well, exactly.
But then we ended up, for a variety of reasons, we ended up, at first, focusing on applications
in computer vision.
The paper comes out, you’re working on these other applications, you’re presenting the paper,
it’s published in various forms.
What’s the response like?
It was interesting because the response built in deep learning AI circles, basically, between
the preprint that I think came out in, I want to say, June 2017, and then the actual publication,
to the extent that by the time the poster session happened at the conference, there was quite
a crowd at the poster, so we had to be shoved out of the hall in which the poster session
happened by the security, and had very hoarse voices by the end of the evening.
You guys were like the Beatles of the AI conference?
I wouldn’t say that because we weren’t the Beatles, because it was really, it was still
very specific.
You were more the cool hipster band.
You were the it hipster band.
Certainly more the cool hipster band.
But it was an interesting experience because there were some folks, including some greats
in the field, who came by and said, wow, this is cool.
What has happened since has been wild, it seems.
Wild, to say the least, yes.
Is it surprising to you?
Of course, many aspects are surprising, for sure.
We definitely saw pretty early on, already back in 2018, 2019, that something really exciting
was happening here.
Now, I’m still surprised by it.
With the advent of ChatGPT, something that didn’t go way beyond those language models that
we had already seen a few years before was suddenly the world’s fastest growing consumer product.
Ever, right?
I think ever.
Ever.
Yes.
And by the way, GBT stands for Generative Pre-Train Transformer, right?
Transformer is your word.
That’s right.
So there’s an interesting, I don’t know, business side to this, right?
Which is, you were working for Google when you came up with this.
Google presumably owned the idea.
Yep.
Had intellectual property around the idea.
Has filed many a patent.
Was it just a choice Google made to let everybody use it?
Like, when you see the fastest growing consumer product in the history of the world, not only
built on this idea, but using the name, like, and it’s a different company.
That was five years later.
Five years later.
But a patent’s good for more than five years.
Is that a choice?
Is that a strategic choice?
What’s going on there?
So the choice to do it in the first place, to publish it in the first place, is really
based on and rooted in a deep conviction of Google at the time, and I’m actually pretty
sure it still is the case, that it is actually, these developments are the tide that floats all
boats, that lifts all boats.
Uh-huh.
Like a belief in progress.
A belief in progress.
Exactly.
Like a good old-fashioned belief in…
Now, it’s also the case that at the time, organizationally, that specific research arm was unusually separated
from the product organizations.
And the reason why brain, or in general, the deep learning groups, were more separated was
in part historical.
Namely, that when they started out, there were no applications, and the technology was not ready
for being applied.
And so it’s completely understandable and just, you know, a consequence of organic developments
that when this technology suddenly is on the cusp of being incredibly impactful, you’re probably
still underutilizing it internally and potentially also not yet treating it in the same way as you
would have maybe otherwise treated previous trade secrets, for example.
Because it feels like this out there research project, not like what’s going to be this consumer
product.
And to be fair, it took OpenAI, in this case, a fair amount of time to then turn this into this
product.
And most of that time, it also, from their vantage point, wasn’t a product, right?
So up until all the way through ChatGPT, OpenAI published all of their GPT developments, maybe
not all, but, you know, a very large fraction of their work on this.
Yeah, their early models, the whole models were open.
Exactly.
They were more true to their name, really, really also believing in the same thing.
And it was only really after ChatGPT and after this, to them also surprise, to a certain extent,
success, that they started to become more closed as well when it comes to scientific developments
in this space.
We’ll be back in just a minute.
Run a business and not thinking about podcasting?
Think again.
More Americans listen to podcasts than ad-supported streaming music from Spotify and Pandora.
And as the number one podcaster, iHeart’s twice as large as the next two combined.
So whatever your customers listen to, they’ll hear your message.
Plus, only iHeart can extend your message to audiences across broadcast radio.
Think podcasting can help your business?
Think iHeart.
Streaming, radio, and podcasting.
Call 844-844-iHeart to get started.
That’s 844-844-iHeart.
Let’s talk about your company.
When did you decide to start Inceptive?
The decision took a while and was influenced by events that happened over the course of
about three months, two to three months in late 2020, starting with the birth of my first
child.
So when Amre was born, two things happened.
Number one, witnessing a pregnancy and a birth during a pandemic where there’s a pathogen that’s
rapidly spreading.
And so all of that was a pretty daunting experience.
And everything went great.
But having this new human in my arms also really made me question if I couldn’t more directly
affect people’s lives positively with my work.
And so I was at the time quite confident that indirectly it would have effect also on things
like medicine, biology, et cetera.
But I was wondering, couldn’t this happen more directly if I focused more on it?
The next thing that happened was that AlphaFold 2 results at CASP-14 were published.
CASP-14 is this biannual challenge for protein structure prediction and some other related problems.
This is the protein folding problem.
And this is the protein folding problem, exactly.
So machine learning solving the protein folding problem, which had been a problem for decades,
given a chain of amino acids to predict the 3D structure of a protein.
Precisely.
And humans failed and machine learning succeeded.
Just amazing.
Yes.
It’s a great example.
Humans failed despite the fact that we actually understand the physics fundamentally, but we
still couldn’t create models that were good enough using our conceptual understanding
of the processes involved.
Yeah, you would think an algorithm would work on that one, right?
You would just think an old school set of rules.
Like, we know what the molecules look like.
We know the laws of physics.
It’s amazing that we couldn’t predict it that way, right?
All you want to know is what shape is the protein going to be?
You know all of the constituent parts.
You know every atom in it.
And you still couldn’t predict it with a set of rules, but AI, machine learning, could.
Amazing.
Yes.
And it is amazing.
Actually, when you put it like this, it’s important to point out that when we say we
understand it, we make massive oversimplifying assumptions.
Because we ignore all the other players that are present when the protein folds.
We ignore a lot of the kinetics of it because we say we know the structure, but the truth
is we don’t know all the wiggling and all the shenanigans that happen on the way there,
right?
And we don’t know about, you know, chaperone proteins that are there to influence the folding.
We don’t know around all sorts of other aspects.
I’m doing the physics one.
I’m doing the assume a frictionless plane version of protein.
Precisely.
Which is why it didn’t work.
And the beauty is that deep learning doesn’t need to make this assumption.
AI doesn’t need to make this assumption.
AI just looks at data.
And it can look at more data than any human or even humanity eventually could look at together.
It’s such a good example problem to demonstrate that these models are ready for prime time in
this field and ready for lots of applications, not just one or two, but many.
And so that happens.
They’ll be sold, exactly.
And then the third thing was that these COVID mRNA vaccines came out with astonishing 90 plus
percent efficacy.
So fast also.
Out of the gate.
How fast and how good they were is still so underrated.
Underrated.
At the beginning of the pandemic, people were like, it’ll be two or three years, and if they’re
60 percent effective, that’ll be great.
Exactly.
And so-
Everybody forgets that.
Everybody forgets it.
And when you look at it, this is a molecule family that was for, you know, most of the
most of the time that we’ve known about it, since the 60s, I suppose, we’ve treated it
like an addicted stepchild of molecular biology.
Because we’ve massed-
You’re talking about RNA in general?
RNA.
RNA in general.
Yeah.
Everybody loves DNA, right?
DNA is the movie star.
Exactly.
Exactly, exactly.
Even though now, looking back, DNA is merely, you know, the place where life takes its notes,
maybe the hard drive and the memory.
It’s the book, right?
It’s the book, right?
It’s the book.
So, but at the end of the day, it’s this molecule family that was about to save, you know, depending
on the estimate, tens of millions of lives, and in rapid time.
So all these things hold, but we have no training data to apply anything like alpha-fold to this
specific molecule family.
No training data to speak of.
We had 200,000 known protein structures at the time.
I believe, maybe optimistically, we had maybe 1,200 known RNA structures.
And on top of that, it was also fairly clear that for RNA, going directly to function would
be much, much more important because it’s, in a certain sense, a weak, less strongly structured
molecule and other aspects of the molecule might play a bigger role.
And then, on top of that, the attention that generative AI was receiving overall, also now
in the field of pharma or of medicine, was building.
And so I ended up finding myself in a conversation where a very, I’d say, wise, long-time mentor
of mine pointed out that, you know, maybe 10 years from now or so, somebody could tell my
daughter that there was this perfect storm where this macromolecule with no training data was
about to save the world and could do so much more in the direction of positively impacting
people’s lives.
We didn’t have training data.
It would be very expensive to create it.
But using the technology that I’ve been, or technologies that I’ve been working on for
the last, I don’t know, 10 plus years, and the ability, because of the attention that
people were now giving to AI in this field, the ability to raise quite a bit of money,
I, in that position, chose to stay back at my cushy dream job in big tech and not actually
take this opportunity to really positively impact people’s lives.
And that idea was not one I was willing to entertain.
You couldn’t just coast it out at Google and let somebody else go figure out RNA.
Yeah.
And it’s not just RNA.
I think RNA is a great starting point at the end of the day.
But building models that learn from, first of all, all the publicly available data that we
can possibly get our hands on, but also from data that we can reasonably effectively create
in our own lab, how to design molecules for specific functions, is something that now is
within reach and that will, in the next years and in the years to come, have completely transformational
impact on how we even think about what medicines are.
That any opportunity to speed this up, to make this happen, even just a day sooner than it could
have otherwise happened, is incredibly valuable, in my opinion.
As you’re talking about this idea that the absence of training data seems to be at the
center of it, right?
It seems to be the core problem, which makes sense, right?
Like, the reason language works so well is basically because of the internet.
I know now we’re going beyond it, but it just happened to be that there was this incredibly
giant set of natural language that became available.
We don’t have anything like that for RNA.
So are you, I mean, it’s kind of step one at Inceptive, creating the data?
Is that kind of what’s happening?
So step one at Inceptive is learning to use all the data, or was, I think we’ve made a lot
of progress in that direction, learning to use all the data that is available already, and
identify what other data we’re missing.
And then see how far we can get with just the publicly available data, and at the same
time scale up generating our own data.
And it turns out that actually, because of the nature of evolution, because of how evolution
isn’t actually incentivized to really explore the entire space of possibilities, it is almost
always a given that if you are trying to design exceptional molecules, especially ones that
are not, say, you know, natural formats, you are basically guaranteed to need novel training
data.
Yeah.
Basically, you’re saying you build RNAs that don’t exist in the world, that have therapeutic
uses, and there’s no, kind of definitionally, no training data for that, because they don’t
exist.
The funny thing is, we have a few of them, and so we have existence proofs of RNA molecules,
for example, RNA viruses, that actually exhibit incredibly complex, different functions in ourselves
that do all sorts of things that we don’t usually like, but if we could use those, you know, for
good, if we could use those, you know, in ways that would actually be aimed at fighting disease
rather than creating them, those kinds of functions, even just a small subset of them, would really
transform medicine already.
And so we know it’s possible.
What are you dreaming of when you say that?
What are you thinking of, specifically?
Okay.
So, for example, right, one estimate is that in order for COVID to infect you, you would need
potentially as few as five COVID genomes inside your organism.
That’s already it.
Five viral particles?
Five viral particles.
Yeah.
You inhale those.
You wouldn’t have to inject it.
You wouldn’t even have to swallow it.
You inhaled them.
What if we could have a medicine that worked as well as a disease, is a version of your dream.
Exactly.
Exactly.
So, at the end of the day, right, this medicine is able to spread in your body only into certain
types of organs and tissues and cells.
It does certain things there that are really quite complex, right?
Changing the cells’ behavior.
Yeah.
Again, not usually in this case in favorable ways, but still in ways that wouldn’t have
to be modified that much in order to potentially be exactly what you would need for a complex
multifactorial medicine.
And if you could make all of that happen by just inhaling five of those molecules, then,
again, that would completely change how you think about medicine, right?
You have viruses that aren’t immediately active, but that are inactive for long periods of time
in your organism.
And only under certain conditions, say, under certain immune conditions, really start being
reactivated.
Why can’t we have medicines that work in a similar way, where you actually, not only in
a vaccination sense, but where you take a medicine for a genetic predisposition for a
certain disease, that you are able to design a medicine that you can take and that waits until
the disease actually starts to develop.
And only then, and only where that disease then starts to develop, becomes active and
actually facts it.
And potentially also then alarms the doctor through a blood test.
Like for cancer cells or something.
So you have some kind of prophylactic medicine in your body, and it is encoded in such a way
that it just hangs out there like herpes, to take a pathological example.
For example, yes.
And only in certain settings does it do anything.
And those settings are, if you see a cancer cell, destroy it.
Otherwise, just sit there.
Precisely.
And if you can design those also in ways where you can just make them all go away when you
take a, say, a completely harmless small molecule, and that’s, again, entirely feasible.
Sure.
So, I mean, you’re dreaming big.
These are wonderful big, you know, science fiction-y dreams, and I hope you figure them
out.
On a practical level, what’s happening at the company right now?
How many people work there?
What are they doing?
And what have they figured out so far?
We’re around 40.
What we’re doing is really exactly what we just talked about.
We’re basically scaling data generation experiments in our lab that allow us to assess a variety
of different functions of different, mostly RNA molecules, actually mostly mRNA molecules at the
moment, that are relevant to a pretty broad variety of different diseases.
And so, this ranges from things like infectious disease vaccines to cell therapies that can be applied
in oncology or against autoimmune disease.
We have mRNAs that we hope will eventually be effective in enzyme replacement as enzyme replacement
therapies for families of, or a large family of rare diseases.
And the list goes on.
And so, we’re creating this, or growing this training data set that eventually, on top of
foundation and models that we pre-trained on all publicly available data, allow us to tune
those foundation models towards designing exceptional molecules for exactly those applications and many
more sharing similar properties.
So, you basically build new mRNA molecules and test them, and then you give that data to
your model, and presumably it tells you what to build next, or it helps you figure out what to
build next.
It’s sort of a loop in that way?
The models are definitely one interesting source for proposals, if you wish, for what to synthesize
and test next.
They’re not the only such source.
So, we basically also explore kind of in maybe less guided or heuristically guided ways.
But, exactly.
So, in some of the cases, it’s really quite iterative.
For some of those functions and for some of those modalities and diseases or disease targets,
we’re actually already at a point where our models can spit out entirely novel molecules that
really are unlike anything they’ve ever seen or we’ve ever seen in nature, that very consistently
perform quite favorably compared to pretty strong baselines by incumbents in the field.
When you say perform quite favorably compared to baselines by incumbents in the field, I mean,
does that on some level mean better than what experts would think up?
Better than what experts can think of and also better than more traditional machine learning
tools can easily produce.
It’s like that famous moment in the Go match when AlphaGo made some move that, like, no human
being ever would have thought of.
Move 37.
Yes.
So, I would say we’ve long passed the Move 37 in the sense that our understanding of the
underlying biological phenomena is so incomplete that for most of the things that we’re able
to design for, we don’t really understand why they happen.
Huh.
When you say we, do you mean at Inceptive or do you mean just medicine in general?
I would say just medicine in general.
Okay.
So, Inceptive is doing this very kind of high-level work, right?
I mean, building what will hopefully be the foundation.
What’s the right amount of time in the future to ask about?
When will we know if it works?
Do you think five years?
So, the general idea of using generative AI and similar techniques to generate therapeutics,
there are some things in clinical trials that were largely designed with AI.
As far as I know, we’re still, maybe now we have the first trials just now starting for
molecules that were truly entirely designed by AI.
As opposed to sort of selected from a library?
Selected, influenced, exactly.
Selected, adjusted, tuned, tweaked, et cetera, right?
So, that’s really still only happening just now.
Okay.
But we will see, I believe, the first success or a first success of such molecules, certainly
within the next five years.
What about more narrowly the project at Inceptive?
It’s a similar timeframe.
We should be able to get molecules into the clinic in the next few years, certainly in the
next handful of years.
Now, these will not be molecules with, where the objective that we used in their design is,
you know, even remotely as complex or the, you know, kind of the different functions that
we’re designing for are not going to be even remotely as diverse as, say, what you would find in,
because we used this example earlier in RNA virus.
These will really be more, you know, simpler.
Those will be molecules that don’t do things that we couldn’t possibly have done before,
but that do them much better in ways that are more accessible, in ways that come with less side
effects.
What biotech largely is, is they make protein drugs.
And so if you could make an mRNA drug where you put the mRNA into the body and the body makes the protein,
it wouldn’t be some crazy sleeper cell that sits in your body for 20 years or whatever.
But it might be a more practical alternative to today’s biotech drugs.
Absolutely.
So you’ve had a kind of crash course in biology in the last few years.
Yes.
And I’m curious, like, what is, what is something that has been particularly compelling or surprising
or interesting to you that you have learned about biology?
There are countless things.
The biggest one or the red thread across many of them is really just how effective life is
at finding solutions to problems that, on one hand, are incredibly robust, surprisingly robust,
and on the other hand, are so different from how we would design solutions to similar problems.
Aha.
That really, this comes back to this idea that we might just not be particularly well-equipped
in terms of cognitive capabilities to understand biology, that basically, you know, we are,
we would never think to do it this way.
And how we think to do it is oftentimes much more brittle.
Aha.
Brittle is an interesting world.
Less resilient, less able to persist under different conditions.
Exactly.
Exactly.
I mean, you know, we still haven’t built machines that can fix themselves, for one.
Which is fundamentally the miracle of being a human being.
Which is fundamentally the miracle of life.
I’m still here after going through all this.
Exactly.
Exactly.
Exactly.
And so, and of course, this is true across the scales, right?
From, you know, single cells all the way to complex organisms like ourselves.
And really just how many also very different kinds of solutions life has found and or constantly
is finding.
And you see this all over the place.
And it’s both daunting, humbling, but also incredibly inspiring when it comes to applying
AI in this area.
Because again, I think that at least so far, it’s the best tool and maybe actually the only
tool we have so far in face of this kind of complexity, really design interventions that go way beyond
what we were able to do or are able to do just based on our own conceptual understanding.
We’ll be back in a minute with the lightning round.
Run a business and not thinking about podcasting?
Think again.
More Americans listen to podcasts than ad-supported streaming music from Spotify and Pandora.
And as the number one podcaster, iHeart’s twice as large as the next two combined.
So whatever your customers listen to, they’ll hear your message.
Plus, only iHeart can extend your message to audiences across broadcast radio.
Think podcasting can help your business?
Think iHeart.
Streaming, radio, and podcasting.
Call 844-844-IHEART to get started.
That’s 844-844-IHEART.
Let’s finish with the lightning round.
As an inventor of the Transformer model, are there particular possible uses of it that
worry you slash make you sad?
I am quite concerned about the P-Doom, Doomerism, whatever you want to call it, existential fear
instilling rhetoric that is in some cases actually also promoted by people, by entities in the space.
So just to be clear, you’re not worried about the existential risk.
You’re worried about people talking about the existential risk.
I’m worried about the existential risk being inflated, or the perception being inflated to the extent
that we actually don’t look enough at some of the much more concrete and much more immediate risks.
I’m not going to say that the existential risk is zero, but that would be silly.
What is a concrete and immediate risk that is, you think, under-discussed?
These large-scale models are such effective tools in manipulating people in large numbers already
today, and it’s happening everywhere for many, many different purposes by, in some cases, benevolent,
and in many cases, malevolent actors that I really firmly believe we need to look much more
at things like enabling cryptographic certification of human-generated content, because doing that
with the machine-generated content is not going to work, but we definitely can cryptographically
certify human-generated content as such.
Basically, watermarking or something, some way to say, a human made this.
Exactly.
What would you be working on if you were not working in biology, on drug development?
Education.
Using artificial intelligence to democratize access to education.
What have you seen that has been impressive or compelling to you in that regard?
There are lots of little examples so far, and really countless.
It’s what’s happening at the Khan Academy, there are many examples of AI applied to education
problems in places like China, for example.
You have a bunch of very compelling examples in fiction, a book I really like by a guy named
Neil Stevenson, The Diamond Age, or Young Lady’s Illustrated Primer, that I recommend if
you just want to…
Everybody in AI talks about that.
Well, now they do, yeah.
Yeah, well, now they do.
You liked it before it was cool, I’m sure.
At one point, I thought it was really, really important to ensure that Neil Stevenson knows
that we are about to be able to build the primer, and so I ended up having coffee with
him to tell him.
Oh, that’s great.
So, at the end of the day, maybe the biggest inspiration there is my daughter.
She’s four and a half now, and I think she could, today, read, she can read okay, but she could
read, you know, grade school level if she had access to, you know, an AI tutor teaching her
how to read.
Does your daughter use AI?
Use, you know, AI chatbots?
Not directly without me, but we’ve actually used ChatGPT to implement an AI reading tutor
that works reasonably well.
I mean, we basically, you know, kind of as they call it now, vibe coding.
We vibe coded, and Amway wasn’t there for all of it.
It took some time, but she was there for some of it.
Oh, you vibe coded it with her?
Yeah, well, I mean, she was there, she, you know, she witnessed a good chunk of it, yes.
Although she was more interested in the image generation parts.
But yeah, we have a sketch of one that she quite enjoys.
So, that’s kind of like the extent of her at this age using AI directly.
Jakob Uskoreit is the CEO and co-founder of Inceptive, and the co-author of the paper,
attention is all you need.
Just a quick note, this is our last episode before a break of a couple of weeks,
and then we’ll be back with more episodes.
Please email us at problematpushkin.fm.
We are always looking for new guests for the show.
Today’s show was produced by Trina Menino and Gabriel Hunter-Chang.
It was edited by Alexandra Gerritsen and engineered by Sarah Brugger.
This is an iHeart Podcast.
Jakob Uszkoreit is the CEO and co-founder of Inceptive, a biotech start-up. He’s also a co-author of “Attention is All You Need,” the paper that created transformer models. Today, transformers power chatbots like ChatGPT and Claude. They’ve also led to breakthroughs in everything from generating images to predicting the structure of proteins.
On today’s show, Jakob talks about the invention of transformer models. And he discusses how he’s using those models to try to invent new kinds of medicine, with a particular focus on RNA.
See omnystudio.com/listener for privacy information.
