AI transcript
0:00:05 The content here is for informational purposes only, should not be taken as legal business
0:00:10 tax or investment advice, or be used to evaluate any investment or security and is not directed
0:00:14 at any investors or potential investors in any A16Z fund.
0:00:17 For more details, please see a16z.com/disclosures.
0:00:21 Hi, and welcome to the A16Z podcast.
0:00:22 I’m Frank Chen.
0:00:28 Today, I’m here with Carnegie Mellon’s Professor Tom Mitchell, who has been involved with machine
0:00:30 learning basically his entire career.
0:00:34 So I’m super excited to have this conversation with Tom, where he can tell us a little bit
0:00:37 about the history and where all of our techniques came from.
0:00:40 And we’ll spend time talking about the future, where the field is going.
0:00:45 So Carnegie Mellon’s been involved in sort of standing up the fundamental teaching institutions
0:00:50 and research institutions of the big areas, computer science, artificial intelligence
0:00:52 and machine learning.
0:00:56 So take us back to the early days, the you and Newell and Jeff Hinton are teaching this
0:00:57 class.
0:00:58 What was your curriculum like?
0:00:59 Like what were you teaching?
0:01:02 Pretty different, I imagine, than what we teach undergrads today.
0:01:03 That’s right.
0:01:05 Well, at the time, this was the 1980s.
0:01:13 So artificial intelligence at that point was dominated by what we would call symbolic methods.
0:01:19 Where things like formal logic would be used to do inference.
0:01:24 And much of machine learning was really about learning symbolic structures, symbolic representations
0:01:25 of knowledge.
0:01:31 But there was this kind of young whipper snapper, Jeff Hinton, who had a different idea.
0:01:41 And so he was working on a book with Rommelhart McClellan that became a very well-known parallel
0:01:47 data processing book that kind of launched the field of neural nets.
0:01:49 And they were, if I remember, psychologists, right?
0:01:58 Yeah. Jay McClellan is a psychologist here at CMU, Rommelhart, kind of a neuroscientist.
0:02:01 And more, he was a very broad person.
0:02:07 And Jeff, so the three of them were kind of the rebels who were taking things off in a
0:02:09 different paradigm.
0:02:14 The empire wanted us to do research on knowledge representation and inference and first-order
0:02:15 logic.
0:02:20 I remember as an undergrad, I took this computer aided class that John H.M.M.D. wrote called
0:02:24 Tarski’s World, where we learned all about first-world logic.
0:02:25 What could you prove?
0:02:26 What could you not prove?
0:02:30 And so that’s what the establishment, quote, unquote, was teaching.
0:02:37 And then Jeff was the rebel off in neural network land, and he gets his reprise later.
0:02:40 So take us back to the world of knowledge representation, because I’m actually seeing
0:02:44 a lot of startups these days who are trying to bring back some of these techniques to
0:02:49 complement deep learning, because there are well-known challenges with deep learning.
0:02:52 Like, we’re not encoding any priors, we’re learning everything for the first time, we
0:02:56 need tons of labeled data sets to make progress.
0:03:00 And so take us back to the days of knowledge representation.
0:03:04 What were we trying to solve with those set of techniques, and how might we use them today?
0:03:13 So back in the ’80s and the ’90s, and I have to say that some of the really senior people
0:03:20 in the field were totally devoted to this paradigm of logical inference, logical representations.
0:03:27 People like John McCarthy, for example, were very strong proponents of this, and really
0:03:34 essentially just saw that reasoning is theorem-proving, and therefore if we’re going to get computers
0:03:38 to do it, that’s what we have to do.
0:03:42 There were some problems with that, and there still are.
0:03:48 One thing that I remember from back then that was an example was the banana in the tailpipe
0:03:50 problem.
0:03:55 These logical systems were used to reason to do things like, how would you plan a sequence
0:03:57 of actions to achieve a goal?
0:03:59 Like, how would you get from here to the airport?
0:04:05 Well, you’d walk to your car, you’d put the key in, turn the car on, you’d drive out of
0:04:10 the parking lot, get on the interstate, go to the airport exit, et cetera.
0:04:12 But what if there’s a banana in the tailpipe?
0:04:17 Even back then, before it became a meme in Beverly Hills Cop, we were worried about
0:04:19 the banana in the tailpipe.
0:04:20 That’s right.
0:04:25 And the point of the banana in the tailpipe is there are an infinite number of other things
0:04:31 that you don’t say when you spin out a plan like that.
0:04:39 And any proof, if it’s a proof, really, is going to have to cover all those conditions,
0:04:43 and that’s kind of an infinitely intractable problem.
0:04:48 You couldn’t encode enough to do the inference you needed for your plans to be successful.
0:04:49 Right.
0:04:56 And so one of the big changes between the ’80s and 2019 is that we no longer really think
0:05:03 in the field of AI that inference is proving things, instead it’s building a plausible
0:05:05 chain of argument.
0:05:10 And it might be wrong, and if it goes wrong, if there is a banana in the tailpipe, you’ll
0:05:12 deal with it when it happens and when you figure it out.
0:05:13 Right.
0:05:18 So we move from certainty and proof to sort of probabilistic reasoning, right, Bayesian
0:05:20 techniques started becoming popular.
0:05:21 Right.
0:05:25 And so around 19– in the late ’90s, in fact.
0:05:31 So if you look at the history of machine learning, there’s an interesting trajectory where in
0:05:38 maybe up to the mid ’80s, things were pretty much focused on symbolic representations.
0:05:43 Actually, if you go back to the ’60s, there were the perceptron, but then it got swallowed
0:05:50 up by the end of the ’60s by symbolic representations and trying to reason that way and trying to
0:05:52 learn those kind of symbolic structures.
0:05:59 Then when the neural net wave came in around the late ’80s, early ’90s, that started competing
0:06:02 with the idea of symbolic representations.
0:06:09 But then in the late ’90s, the statisticians moved in and probabilistic methods became
0:06:11 very popular.
0:06:16 And at the time, there was this– if you look at this history, you can’t help but realize
0:06:22 what a social phenomenon technology advances in sciences and technology.
0:06:24 People influencing each other at conferences.
0:06:25 Right.
0:06:29 They were shaming them into adopting a new paradigm.
0:06:36 And so one of the slogans, or one of the phrases you kept hearing when people started working
0:06:41 on probabilistic, statistical probabilistic methods, they would never call them that.
0:06:48 They would have called them, instead, principled probabilistic methods, just to kind of shine
0:06:53 the light on the distinction between neural nets, which are just somehow tuning a gazillion
0:06:57 parameters in the principled methods that were being used.
0:07:03 And so that became really the dominant paradigm in the late ’90s and kind of remained in charge
0:07:12 of the field up through till about 2009, 2010, when now, as everybody kind of knows, deep
0:07:18 networks made a very serious revolution showing that they could do all kinds of amazing things
0:07:20 that hadn’t been done before.
0:07:21 Yeah.
0:07:26 We really are living in a golden age here in deep learning in neural network land.
0:07:29 But let’s go back to the original sort of rebel group.
0:07:35 This is Jeff Hinton hanging out in the shadow of sort of first order logic and saying, “No,
0:07:36 this is going to work.”
0:07:40 I think they were loosely inspired by the architecture of the brain.
0:07:41 Is that–
0:07:42 Definitely.
0:07:43 Definitely.
0:07:49 The kinds of arguments Jerry Feldman was one of the people who gave some of these arguments.
0:07:55 She said, “Look, you recognize your mother in about 100 milliseconds.
0:08:03 Your neurons can’t switch state in faster than a few milliseconds.
0:08:10 And so it looks like at most the chain of inference that you’re doing to go from your
0:08:16 retina to recognize your mother can only be about 10 deep, just from the timing.”
0:08:17 Oh, fascinating.
0:08:21 So it was an argument of sort of how long it took to recognize your mother.
0:08:22 Right.
0:08:23 And then how slow your neurons are, right?
0:08:26 Because they’re basically, these are biochemical processes, right?
0:08:27 Right.
0:08:28 Fascinating.
0:08:31 So really a computational efficiency argument.
0:08:37 And therefore, Jerry would say, “There must be a lot of stuff happening in parallel.
0:08:41 It must be a very wide chain of inference if it’s only 10 layers deep.”
0:08:45 And then he says, “Look at the brain, look at visual cortex.”
0:08:46 Yeah.
0:08:47 You got it.
0:08:51 And so neuroscientists at this time were making progress in understanding the structure of
0:08:55 neurons and how they connected to each other and how they formed connections, and those
0:08:58 connections could change strength over time, right?
0:09:02 All mediated by chemical interactions in the computer science community was inspired by
0:09:03 this.
0:09:04 Definitely.
0:09:12 And the level of abstraction at which the computational neural nets met up with the
0:09:19 real biological neural nets was not a very detailed level, but where they kind of became
0:09:28 the same was this idea of distributed representations, that in fact it might be a collection of hundreds
0:09:35 or thousands or millions of neurons that simultaneously were firing that represent your mother instead
0:09:37 of a symbol.
0:09:38 Right.
0:09:39 Right.
0:09:46 It’s a completely different notion of what it even means to represent knowledge.
0:09:51 And really, one of the most exciting things that has come out of the last decade of research
0:10:00 in neural and deep networks is a better understanding, although we still don’t fully understand of
0:10:07 how these artificial neural networks can learn very, very useful representations.
0:10:14 And for me, a simple example of that, that in a sentence summarizes it, is we have neural
0:10:22 networks now that can take as input an image of photograph and output a text caption for
0:10:23 that photograph.
0:10:28 What kind of representation must be in the middle of that neural network in order to
0:10:35 actually capture the meaning well enough that you can go from a visual stimulus to the equivalent
0:10:43 textual content, that it’s really, it must be capturing a very basic core representation
0:10:45 of the meaning of that photograph.
0:10:46 Yeah.
0:10:50 And one of my favorite things about the brain, which is otherwise this very sort of slow
0:10:54 computer, right, if you just look at neuron speeds, is that not only can they do this,
0:10:59 but they can actually use this, the representation they’re deriving to actually inform our actions
0:11:01 and our plans and our goals, right?
0:11:05 So not only is it like this picture has a chair in it, but like I can sit in that chair.
0:11:06 I can simulate sitting in that chair.
0:11:10 I think like that chair is going to support my weight, and all of these things happen
0:11:14 in like milliseconds, despite the fact that the basic components of the brain are very
0:11:15 slow.
0:11:16 Yeah.
0:11:17 It’s an amazing thing.
0:11:21 In fact, now that you mentioned it, I have to tell you, a half of my research life these
0:11:29 days is in studying how the human brain represents meaning of language.
0:11:32 We use brain imaging methods to do this.
0:11:39 And in one set of studies, we put people in a fMRI scanner, and we showed them just common
0:11:49 nouns like automobile, airplane, a knife, a chair, and so forth.
0:11:56 And we would get a picture literally with about three millimeter resolution of the three-dimensional
0:12:02 neural activity in their brain as they think about these different words.
0:12:06 And we’re interested in the question of all kinds of fundamental questions like what do
0:12:08 these representations look like?
0:12:12 Are they the same in your brain and my brain?
0:12:17 Given that they don’t appear instantaneously, by the way, it takes you about 400 milliseconds
0:12:22 to understand a word, if I put it on the screen in front of you.
0:12:24 What happens during that 400 milliseconds?
0:12:30 How do these representations evolve and come to be?
0:12:36 And one of the most interesting things we found, we studied this question by training
0:12:46 a machine learning system to take as input an arbitrary noun and to predict the brain
0:12:49 image that we will see if a person reads that noun.
0:12:56 Now, we only had data for 60 nouns at that time, so we didn’t train it on every noun
0:12:57 in the world.
0:13:00 We trained it on 60.
0:13:05 In fact, what we did was we trained it only on 58, so we could hold out two nouns that
0:13:06 we hadn’t seen.
0:13:12 And then we would test how well it could extrapolate to new nouns that you had never seen by showing
0:13:18 it the two held out nouns and having it predict the images.
0:13:21 Then we’d show it two images and we’d say, “Well, which of those is strawberry and which
0:13:23 of those is airplane?”
0:13:26 And it was right 80% of the time.
0:13:29 So, you could actually predict essentially brain state, right?
0:13:31 I’m going to show you a strawberry.
0:13:35 Let me predict the configuration of your neurons and who’s lighting up and who’s not.
0:13:42 So then we had a model that we trained with machine learning that captured something about
0:13:44 representations in the brain.
0:13:50 We used that to discover that the representations are almost identical in your brain and mind.
0:13:56 We could train on one set of people and decode what other people were thinking about.
0:14:02 And we also found that the representations themselves are grounded in parts of the brain
0:14:05 that are associated with perception.
0:14:10 So, if I give you a word like “peach,” the parts of your brain that code the meaning
0:14:16 of that are the ones associated with the sense of taste and manipulation because sometimes
0:14:19 you pick up a peach in visual color.
0:14:21 Yeah, that is fascinating.
0:14:25 Well, it’s so exciting to think that the brain structures are identical across people
0:14:30 because what everybody wants is sort of that, remember that scene in the matrix where you
0:14:34 sort of like you’re jacked straight into your brain and you’re like, “Oh, now I know Kung-Fu.”
0:14:35 Right?
0:14:36 Like this is what we want, right?
0:14:42 We want to learn new skills and sort of new facts and new inferences just like loading
0:14:43 an SD card, right?
0:14:48 And so, the fact that we’re sort of converging to the same structures in the brain at least
0:14:50 makes that theoretically possible.
0:14:51 We’re a ways away from that.
0:14:52 We’re a ways away from that.
0:14:53 But I’m with you.
0:14:54 Yeah, awesome.
0:15:01 So, another area that interests you is finding biases and why don’t we start by distinguishing
0:15:06 sort of two types of biases because when you hear the word “bias” today in machine learning,
0:15:10 you’re mostly thinking about things like, “Gee, let me make sure my data set is representative
0:15:13 so I don’t draw the wrong conclusion from that,” right?
0:15:18 So the classic example being here that I don’t do good recognition on people with darker skin
0:15:23 because I didn’t have enough of those samples in my data set and so the bias here is you’ve
0:15:29 selected a very small subset of the target data set that you want to cover and make predictions
0:15:31 on and therefore your predictions are poor, right?
0:15:33 So, that’s one sense of bias.
0:15:37 But there’s another sense of bias, that statistical bias, which is kind of what you want out of
0:15:38 algorithms.
0:15:39 So, maybe talk about this notion.
0:15:40 Yeah, sure.
0:15:45 And this is really a very important issue right now because now that machine learning
0:15:52 is being used in practice in many different ways, the issue of bias really is very important
0:15:54 to deal with.
0:16:00 You gave an example, another example would be, for instance, you have some historical
0:16:06 loan applications in which ones were approved, but maybe there’s some bias that say people
0:16:12 of one gender receive fewer loan approvals just because of their gender and if that’s
0:16:18 inherent in the data and you train a machine learning system that’s successful, well, it’s
0:16:22 probably going to learn the patterns that are in that data.
0:16:29 So, the notion of what I’ll call social bias, socially unacceptable bias, is really this
0:16:36 idea that you want the data set to reflect the kind of decision making that you want
0:16:39 the program to make if you’re going to train the program.
0:16:44 And that’s kind of the common sense notion of bias that most people talk about.
0:16:50 But there’s a lot of confusion in the field right now because bias is also used in statistical
0:16:55 machine learning to really with a very different meaning.
0:17:02 We’ll say that an algorithm is unbiased if the patterns that it learns, the decision
0:17:10 rules that it learns for approving loans, for example, reflect correctly the patterns that
0:17:12 are in the data.
0:17:18 Also that notion of statistically unbiased just means the algorithm’s doing its job of
0:17:21 recapitulating the decisions that are in the data.
0:17:28 The notion of the data itself being biased is really an orthogonal notion.
0:17:32 And there’s some interesting research going on now.
0:17:39 So for example, typically when we train a machine learning system, say to do loan approval,
0:17:45 a typical thing would be you can think of these machine learning algorithms as optimization
0:17:46 algorithms.
0:17:54 They’re going to tune maybe the parameters of your deep network so that they maximize
0:18:00 the number of decisions that they make that agree with the training examples.
0:18:06 But if your training examples have this kind of bias that maybe females receive fewer loan
0:18:09 approvals than males.
0:18:13 There’s some new work where people say, well, let’s change that objective that we’re trying
0:18:22 to optimize in addition to fitting the decisions that are in the training data as well as possible.
0:18:29 Let’s put another constraint that the probability of a female being approved for a loan has
0:18:33 to be equal to the probability of a male being approved.
0:18:38 And then subject to that constraint, we’ll try to match as many decisions as possible.
0:18:44 So there’s a lot of work right now in really technical work trying to understand if there
0:18:51 are ways of thinking more creatively, more imaginatively about how to even frame the
0:18:58 machine learning problem so that we can take what might be biased datasets but impose constraints
0:19:01 on the decision rules that we want to learn from those.
0:19:03 Yeah, that’s super interesting.
0:19:08 We’re sort of envisioning the world we want rather than the data of the world that we
0:19:13 came from because we might not be happy with the representation of the representedness
0:19:16 I guess, of the data that we came from.
0:19:20 And it’s causing people to look a lot more carefully at even the very notion of what
0:19:25 it means to be biased and what it means to be fair.
0:19:29 Are there good measures for fairness that the community is driving towards or do we
0:19:34 not really have a sort of an objective measure of fairness?
0:19:35 We don’t.
0:19:41 We don’t have an objective measure and there’s a lot of activity right now to discussing
0:19:48 that, including people like our philosophy professor, David Danks, who is very much part
0:19:54 of this discussion and social scientists, technology people, all getting together.
0:20:01 And in fact, there are now a couple of conferences centered around how to introduce fairness and
0:20:07 explainability and trust in AI systems.
0:20:11 This is a very important issue, but it’s not only technical.
0:20:16 It’s partly getting our philosophical, social, trying to get our heads around what it is
0:20:17 that we really want.
0:20:21 That’s a beautiful thing about AI and about computers in general.
0:20:27 It forces you to be way more precise when you are getting a computer to do it about
0:20:29 what you want.
0:20:36 And so even if you just think about self-driving cars, we have, when I was 16, I took a test
0:20:39 and I was approved to be a human driver.
0:20:43 They never asked me questions about whether I would swerve to hit the old lady or swerve
0:20:44 to hit the baby carriage.
0:20:45 Right.
0:20:47 Charlie’s problem was not on the DMV test.
0:20:48 Exactly.
0:20:50 But it’s on the test for the computers.
0:20:51 Right.
0:20:53 Yeah, it’s really interesting that we sort of hold computers to a different standard
0:20:55 because we’re programming them, right?
0:20:59 We can be explicit and we can have them sort of hit goals or not, right?
0:21:04 And those are design decisions rather than sort of, you know, bundled into a brain, right,
0:21:05 of a person.
0:21:06 Yeah.
0:21:12 And so I think of, you know, look, banks historically have hired loan officers.
0:21:16 Those loan officers may or may not be fair, right, according to the definitions that we’re
0:21:21 sort of talking about now, but we kind of hold those humans, those human loan officers to
0:21:24 a different standard than we would hold the algorithms.
0:21:25 That’s true.
0:21:27 I mean, who knows which way it will go in the future.
0:21:33 If we continue to have human loan officers and some computer loan officers, will we
0:21:39 up the constraints on the humans so that they pass the same qualifications?
0:21:45 Or will we drop the constraints on the computers so that they’re no more titrated than the
0:21:46 people?
0:21:47 Yeah.
0:21:48 That’s fascinating, right?
0:21:49 Who is master?
0:21:50 Who is the student, right?
0:21:55 The intuitive thing is let’s make the humans the models that we train our systems to approach,
0:21:56 right?
0:21:58 Like human competence being the goal.
0:22:00 The other way to think about it is no, right?
0:22:04 We can actually introduce constraints like, you know, equal number of men and women or
0:22:07 equal number of this ethnicity versus another ethnicity.
0:22:12 And our algorithms as a result of those constraints could be more fair than humans.
0:22:14 And so we invert it, right?
0:22:16 Let’s get humans up to that level of impartiality.
0:22:17 Right.
0:22:21 Like maybe the algorithm can end up teaching the human how to make those decisions in a
0:22:26 different way so that the fairness outcome you want is really achieved.
0:22:27 Yeah.
0:22:28 That’s fascinating.
0:22:32 And it’s great that sort of not just computer scientists are involved in this conversation
0:22:36 about the ethicists and the social scientists who are weighing in.
0:22:40 So that gives me hope that, you know, sort of smart people across disciplines are really
0:22:41 grappling with this.
0:22:43 So we get the outcomes that we want.
0:22:49 Well, sort of a related topic to this, right, sort of social impact of AI.
0:22:56 You recently co-wrote a paper with MIT’s Eric Bernholzen about the workplace implications.
0:23:00 And I think you also testified on Capitol Hill about what AI is going to do to jobs.
0:23:04 So why don’t you talk a little bit about what you guys found in the paper?
0:23:10 Well, this actually started with Eric and I co-chairing a National Academy study on
0:23:18 automation in the workforce, which was a two-year affair with a committee of about 15 experts
0:23:24 from around the country who were economists, social scientists, labor experts, technologists.
0:23:30 And in that study, I think we learned so much.
0:23:35 It turns out when you really dig into the question of what’s going to be the impact
0:23:39 of AI and automation on jobs.
0:23:46 You can’t escape noticing that there are many different forces that automation and technology
0:23:48 is exerting on the workforce.
0:23:50 One of them, of course, is automation.
0:23:55 Toll booth operators are going away, do not sign up to be a toll booth operator.
0:24:02 But in other kinds of jobs, instead of the job going away, there will be a shift, a redistribution
0:24:05 of the tasks.
0:24:07 So take, for example, a doctor.
0:24:12 A doctor has multiple tasks, for instance, they have to diagnose the patient, they have
0:24:17 to generate some possible therapies, they have to have a heart-to-heart discussion with
0:24:25 the patient about which of those therapies the patient elects to follow, and they have
0:24:26 to bill the patient.
0:24:31 Now computers are getting, computers are pretty good at billing, but they’re getting better
0:24:35 at diagnosis and they’re getting better at suggesting therapies.
0:24:41 For example, in just in the last couple of years, we’ve seen computers that are at the
0:24:49 same level, if not a little better than doctors at things like diagnosing skin cancer and
0:24:50 other kinds of diseases.
0:24:55 The radiologists, the tissue biopsies, all of these things, we’re using these computer
0:24:59 vision techniques to get very good performance.
0:25:02 So what does this mean about the future of doctors?
0:25:07 Well, I think what it means is automation happens at the level of the individual tasks,
0:25:09 not at the job level.
0:25:16 If a job is a bundle of tasks, like diagnosis, therapy, heart-to-heart chat, what’s going
0:25:24 to happen is computers will provide future doctors with more assistance, in some degrees,
0:25:30 hopefully automating billing, but some amount of automation or advice giving.
0:25:36 But for other tasks, like having that heart-to-heart chat, we’re very, very far from when computers
0:25:40 are going to be able to do anything close to that.
0:25:44 Good bedside manner is not going to be a future of your RoboDoc anytime soon.
0:25:53 And so what you find, if you look into this, and Eric and I recently had a paper in Science
0:25:59 with a more detailed study of this, but what you find is that the majority of jobs are
0:26:04 not like toll-booth operators, where there’s just one task, and if that gets automated,
0:26:05 that’s the end of the job.
0:26:15 The majority of jobs, like podcast interviewer, or computer, or professor, or doctor, really
0:26:16 are a bundle of tasks.
0:26:23 And so what’s going to happen is that, according to our study, the majority, more than half
0:26:29 of jobs, are going to be influenced, impacted by automation, but the impact won’t be a
0:26:30 elimination.
0:26:34 It’ll be a redistribution of the time that you spend on different tasks.
0:26:43 And we even conjecture that successful businesses in the future will, to some degree, be redefining
0:26:49 what the collection of jobs is that they’re hiring for.
0:26:53 Because they still have to cover the tasks through some combination of automation and
0:27:01 manual work, but the current bundles of tasks that form jobs today might shift dramatically.
0:27:06 So the key insight is to think of a job as a bundle of tasks, and that bundle might change
0:27:13 over time as AI enters and says, well, look, this specific task I’m very good at in algorithm
0:27:16 land, and so let’s get humans to focus on other things.
0:27:19 We just need to think of them as differently bundled.
0:27:24 Well, the last topic I wanted to talk with you about, Tom, is around whether this is
0:27:27 the best time ever for AI research.
0:27:29 So we started the grand campaign.
0:27:32 Some would argue summer of 1956 with the Dartmouth conference.
0:27:35 And we’ve had several winters and summers.
0:27:36 Where are we now?
0:27:39 And then what are you most excited about looking into the future?
0:27:45 I think we’re absolutely at the best time ever for the field of artificial intelligence.
0:27:48 And there have been, as you say, ups and downs over the year.
0:27:55 And for example, in the late ’80s, AI was very hot, and there was great expectation
0:27:57 of the things it would be able to do.
0:28:02 There was also great fear, by the way, of what Japan was going to do.
0:28:03 Yeah.
0:28:08 This is the fifth generation supercomputer and the entire national policy of Japan, right?
0:28:10 Focusing on this area.
0:28:16 And so in the U.S., there was great concern that this would have a big impact, Japan would
0:28:18 take over the economy.
0:28:21 So there are some parallels here.
0:28:24 Now again, AI, it’s very popular.
0:28:27 People have great expectations.
0:28:31 And there’s a great amount of fear, I have to say, about what China and other countries
0:28:34 might be doing in AI.
0:28:44 But one really, really important difference is that, unlike in the 1980s, right now, there’s
0:28:48 a huge record of accomplishment over the last 10 years.
0:28:57 We already have AI and machine learning being used across many, many different, really economically
0:29:00 valuable tasks.
0:29:06 And therefore, I think, really, there’s very little chance that we’ll have a crash, although
0:29:11 I completely agree with my friends who say, “But isn’t AI overhyped?”
0:29:13 Absolutely, it’s overhyped.
0:29:20 But there is enough reality there to keep the field progressing and to keep commercial
0:29:25 interest and to keep economic investment going for a long time to come.
0:29:27 So you would argue, this time it really is different.
0:29:28 It really is different.
0:29:31 Because we have real working stuff to point to.
0:29:36 And over the next 10 years, we’ll have a whole lot more real working stuff that influences
0:29:38 our lives daily.
0:29:44 So as a university researcher, I look at this and I say, “Where is this going and what should
0:29:47 we be doing in the university?”
0:29:51 If you want to think about that, you have to realize just how much progress there was
0:29:52 in the last 10 years.
0:29:59 When the iPhone came out, I guess that’s 11 years ago, computers were deaf and blind.
0:30:03 When the iPhone came out, you could not talk to your iPhone.
0:30:09 This is such a weird idea, but you could not talk to your iPhone because speech recognition
0:30:11 didn’t work.
0:30:19 And now computers can transcribe voice to text just as well as people.
0:30:26 Similarly, when you pointed your camera at a scene, it couldn’t recognize with any accuracy
0:30:29 the things that were on the table in the scene.
0:30:34 And now it can do that with about the same accuracy comparable to humans.
0:30:38 And in some visual tasks, like skin cancer detection, even better than you know.
0:30:40 Even better than trained doctors, yeah.
0:30:41 Better than trained.
0:30:44 So it’s hard to remember that it’s only been 10 years.
0:30:47 And that’s the thing about progress in AI.
0:30:53 You forget because it becomes so familiar, just how dramatic the improvement has been.
0:30:55 Now think about what that means.
0:31:00 That means we’re really in the first five years of having computers that are not deaf
0:31:01 and blind.
0:31:08 And now think about what are the kinds of intelligence that you could exhibit if you
0:31:13 were deaf and blind, well, you could do game playing and inventory control.
0:31:17 You could do things that don’t involve perception.
0:31:24 But once you can perceive the world and converse in the world, there’s an explosion of new
0:31:25 applications you can do.
0:31:30 So we’re going to have garage door openers that open for you because they recognize that
0:31:32 your car coming down the driveway.
0:31:37 We’re going to have many, many things that we haven’t even thought about that just leverage
0:31:43 off this very recent progress in perceptual AI.
0:31:49 So going forward, I think a lot about how I want to invest my own research time.
0:31:51 I’m interested still in machine learning.
0:31:53 I’m very proud of the field of machine learning.
0:31:55 It’s come a long way.
0:32:01 But I’m also somebody who thinks we’re only at the beginning.
0:32:05 I think if you want to know the future of machine learning, all you need to do is look
0:32:10 at how humans learn and computers don’t yet.
0:32:16 So we learn, for example, we do learn statistically like computers do.
0:32:24 My phone watches me over time and statistically, it eventually learns where it thinks my house
0:32:27 is and where it thinks my work is.
0:32:30 It statistically learns what my preferences are.
0:32:33 But I also have a human assistant.
0:32:39 And if she tried to figure out what I wanted her to do by statistically watching me do
0:32:43 things a thousand times, I would have fired her so long ago.
0:32:46 A lot of false positives and false negatives, right?
0:32:47 Right.
0:32:48 So she doesn’t learn that way.
0:32:51 She learns by having a conversation with me.
0:32:56 I go into the office and I say, “Hey, this semester I’m team teaching a course with
0:33:00 Katarina on deep reinforcement learning.
0:33:02 Here’s what I want you to do.
0:33:04 Whenever this happens, you do this.
0:33:10 Whenever we’re preparing to hand out a homework assignment, if it hasn’t been pretested by
0:33:16 the teaching assistants two days before handout, you send a note saying, “Get that thing pretested.”
0:33:22 So what I do is I teach her and we have a conversation, she clarifies.
0:33:32 So one of the new paradigms for machine learning that I predict we will see in the coming decade
0:33:35 is what I’ll call conversational learning.
0:33:42 Use the kind of conversational interfaces that we have, say, with our phones to allow
0:33:52 people to literally teach their devices what they want them to do instead of have the device
0:33:54 statistically learn it.
0:34:01 And if you go down that road, here’s a really interesting angle on it.
0:34:07 It becomes kind of like replacing computer programming with natural language instruction.
0:34:15 So I’ll give you an example of a prototype system that we’ve been working on together
0:34:19 with Brad Myers, one of our faculty in HCI.
0:34:25 It allows you to say to your phone something like, “Whenever it snows at night, I want
0:34:27 you to wake me up 30 minutes earlier.”
0:34:33 If you live in Pittsburgh, this is a useful app, and none of the California engineers
0:34:34 have created that app.
0:34:40 And today, I could create that app if I took the trouble of learning the computer language
0:34:43 of the phone, I could program it.
0:34:49 But only far less than 1% of phone users can actually have taken the time to learn the
0:34:51 language of the computer.
0:34:56 We’re giving the phone the chance to learn the language of the person.
0:35:03 So with our phone prototype, if you say, “Whenever it snows at night, wake me up 30 minutes
0:35:08 earlier,” it says, “I don’t understand, do you want to teach me?”
0:35:12 And you can say, “Yes, here’s how you find out if it’s snowing at night.”
0:35:17 You open up this weather app right here, and where it says current conditions, if that
0:35:20 says SNOW, it’s snowing.
0:35:25 Here’s how you wake me up 30 minutes earlier, you open up that alarm app, and this number
0:35:27 you subtract 30 from it.
0:35:36 So with a combination of showing, demonstrating, and telling voice, we’re trying to give users
0:35:44 the opportunity to create their own apps, their own programs with the same kind of instruction,
0:35:49 voice, and demonstration that you would use if you were trying to teach me how to do it.
0:35:50 I love that.
0:35:53 It’s sort of a natural language for an end to, we have an investment in a company called
0:35:57 ift, if this, then that, which is you can program those things, but you have to be a
0:35:59 little sophisticated.
0:36:03 You’d like to just be able to talk to your phone and have it figured out how do I fill
0:36:05 the slots into ift that ift wants.
0:36:07 Exactly, and if this, then that is a wonderful thing.
0:36:14 It has a huge library of these apps that you can download, but as you say, you still have
0:36:17 to learn the language of the computer to create those.
0:36:21 We’re trying to have the computer learn the language of the person.
0:36:27 If that line of research plays out, and I believe it will this decade, we’ll be in a very different
0:36:34 world because we’ll be in a world where instead of the elite few, less than 1% of phone users
0:36:39 being able to program, it’ll be 99% of phone users who can do this.
0:36:45 Now, think about what that does for the whole conception of how we think about human-computer
0:36:46 interaction.
0:36:47 Yeah.
0:36:49 That’s a profound shift in society, right?
0:36:55 Just like everybody became literate, not just the priests, and look what happened to society.
0:36:56 Exactly.
0:36:57 Yeah.
0:36:59 Think about what it means for the future of jobs.
0:37:04 Right now, if you have a computer introduced as your teammate, you, the human, and the
0:37:11 computer are a team, well, the computer is frozen, and the teammate who gets to do the
0:37:16 adapting is the human because the computer is a fixed functionality.
0:37:21 What if in that team, the human could just teach the computer how they want the computer
0:37:22 to help them do their job.
0:37:25 It would be a completely different dynamic.
0:37:27 It would change the future of work.
0:37:28 Yeah.
0:37:29 That’s fascinating.
0:37:33 And then I think another thread that you’re super interested in on the future of machine
0:37:37 learning is something around never-ending learning, so tell us about that.
0:37:38 Sure.
0:37:44 Again, I just go back to what do humans do that computers don’t yet, and computers are
0:37:49 very good at, say, learning to diagnose skin cancer.
0:37:54 You give it some very specific tasks and some data, but if you look at people, people learn
0:37:57 so many things.
0:37:59 We learn to all kinds of things.
0:38:00 You can tap the hands.
0:38:02 You can do double entry bookkeeping.
0:38:03 Right?
0:38:04 Right.
0:38:05 You can add numbers.
0:38:11 You can play music, all kinds of things, and a lot of those things we learn over time
0:38:17 in a kind of synergistic way, in a staged sequence.
0:38:21 First you learn to crawl, then you learn to walk, then you learn to run, then you learn
0:38:28 to ride a bike, and it wouldn’t make any sense to do them in the other sequence, because
0:38:31 you’re actually learning to learn.
0:38:37 When you acquire one skill, it puts you in a position that you now are capable of learning
0:38:38 the next skill.
0:38:44 So I’m very interested in what it would mean to give a computer that kind of capability
0:38:51 to do learning for days and weeks and years and decades.
0:38:59 We have a project we call our Never-Ending Language Learner, which started in 2010, running
0:39:03 24 hours a day trying to learn to read the web.
0:39:04 Fascinating.
0:39:07 And there’s something about sort of the longitudinal, right?
0:39:09 We started it in 2010 and it just keeps on going.
0:39:13 So it’s not just like transfer learning from one model to another.
0:39:15 It’s like long running.
0:39:20 It’s long running and it has many different learning tasks.
0:39:25 It’s building up a knowledge base of knowledge about the world.
0:39:33 But to keep it short, I’ll just say we’ve learned so much from that project about how
0:39:39 to organize the architecture of a system so that it can invent new learning tasks as it
0:39:46 goes so that it can get synergy once it learns one thing to become better at learning another
0:39:47 thing.
0:39:53 How, in fact, very importantly, it can use unlabeled data to train itself instead of
0:39:57 requiring an army of data labellers.
0:40:04 So I just think this is an area that’s relatively untouched in the machine learning field.
0:40:11 But looking forward, we’re already seeing an increasing number of embedded machine learning
0:40:14 systems in continuous use.
0:40:22 And as we see more and more of those in the internet of things and elsewhere, the opportunity
0:40:29 for learning continuously for days and weeks and months and years and decades is increasingly
0:40:30 there.
0:40:36 We ought to be developing the ideas, the concepts of how to organize those systems to take advantage
0:40:37 of that.
0:40:38 Yeah.
0:40:41 I love both of these design approaches in that they’re sort of inspired by humans and
0:40:50 sort of humans are mysteriously good at learning and adapting and they sort of shine a spotlight
0:40:53 on where machine learning algorithms are not yet.
0:40:56 So it’s such a fertile area to look for inspiration.
0:41:00 Well, Tom, it’s been a great delight having you on the podcast.
0:41:03 Thanks for sharing about the history and the future of machine learning.
0:41:06 We can tell you’re still fired up after all of these decades.
0:41:11 And so that’s a great delight just to see somebody who is committed, basically, their
0:41:17 life to understanding the mysteries of learning and wishing many good decades to come as you
0:41:19 continue working on it.
0:41:20 Thanks.
0:41:21 Thanks for doing this podcast.
0:41:25 I think it’s a great thing to get a conversation going and it’s a great contribution to do
0:41:25 that.
0:00:10 tax or investment advice, or be used to evaluate any investment or security and is not directed
0:00:14 at any investors or potential investors in any A16Z fund.
0:00:17 For more details, please see a16z.com/disclosures.
0:00:21 Hi, and welcome to the A16Z podcast.
0:00:22 I’m Frank Chen.
0:00:28 Today, I’m here with Carnegie Mellon’s Professor Tom Mitchell, who has been involved with machine
0:00:30 learning basically his entire career.
0:00:34 So I’m super excited to have this conversation with Tom, where he can tell us a little bit
0:00:37 about the history and where all of our techniques came from.
0:00:40 And we’ll spend time talking about the future, where the field is going.
0:00:45 So Carnegie Mellon’s been involved in sort of standing up the fundamental teaching institutions
0:00:50 and research institutions of the big areas, computer science, artificial intelligence
0:00:52 and machine learning.
0:00:56 So take us back to the early days, the you and Newell and Jeff Hinton are teaching this
0:00:57 class.
0:00:58 What was your curriculum like?
0:00:59 Like what were you teaching?
0:01:02 Pretty different, I imagine, than what we teach undergrads today.
0:01:03 That’s right.
0:01:05 Well, at the time, this was the 1980s.
0:01:13 So artificial intelligence at that point was dominated by what we would call symbolic methods.
0:01:19 Where things like formal logic would be used to do inference.
0:01:24 And much of machine learning was really about learning symbolic structures, symbolic representations
0:01:25 of knowledge.
0:01:31 But there was this kind of young whipper snapper, Jeff Hinton, who had a different idea.
0:01:41 And so he was working on a book with Rommelhart McClellan that became a very well-known parallel
0:01:47 data processing book that kind of launched the field of neural nets.
0:01:49 And they were, if I remember, psychologists, right?
0:01:58 Yeah. Jay McClellan is a psychologist here at CMU, Rommelhart, kind of a neuroscientist.
0:02:01 And more, he was a very broad person.
0:02:07 And Jeff, so the three of them were kind of the rebels who were taking things off in a
0:02:09 different paradigm.
0:02:14 The empire wanted us to do research on knowledge representation and inference and first-order
0:02:15 logic.
0:02:20 I remember as an undergrad, I took this computer aided class that John H.M.M.D. wrote called
0:02:24 Tarski’s World, where we learned all about first-world logic.
0:02:25 What could you prove?
0:02:26 What could you not prove?
0:02:30 And so that’s what the establishment, quote, unquote, was teaching.
0:02:37 And then Jeff was the rebel off in neural network land, and he gets his reprise later.
0:02:40 So take us back to the world of knowledge representation, because I’m actually seeing
0:02:44 a lot of startups these days who are trying to bring back some of these techniques to
0:02:49 complement deep learning, because there are well-known challenges with deep learning.
0:02:52 Like, we’re not encoding any priors, we’re learning everything for the first time, we
0:02:56 need tons of labeled data sets to make progress.
0:03:00 And so take us back to the days of knowledge representation.
0:03:04 What were we trying to solve with those set of techniques, and how might we use them today?
0:03:13 So back in the ’80s and the ’90s, and I have to say that some of the really senior people
0:03:20 in the field were totally devoted to this paradigm of logical inference, logical representations.
0:03:27 People like John McCarthy, for example, were very strong proponents of this, and really
0:03:34 essentially just saw that reasoning is theorem-proving, and therefore if we’re going to get computers
0:03:38 to do it, that’s what we have to do.
0:03:42 There were some problems with that, and there still are.
0:03:48 One thing that I remember from back then that was an example was the banana in the tailpipe
0:03:50 problem.
0:03:55 These logical systems were used to reason to do things like, how would you plan a sequence
0:03:57 of actions to achieve a goal?
0:03:59 Like, how would you get from here to the airport?
0:04:05 Well, you’d walk to your car, you’d put the key in, turn the car on, you’d drive out of
0:04:10 the parking lot, get on the interstate, go to the airport exit, et cetera.
0:04:12 But what if there’s a banana in the tailpipe?
0:04:17 Even back then, before it became a meme in Beverly Hills Cop, we were worried about
0:04:19 the banana in the tailpipe.
0:04:20 That’s right.
0:04:25 And the point of the banana in the tailpipe is there are an infinite number of other things
0:04:31 that you don’t say when you spin out a plan like that.
0:04:39 And any proof, if it’s a proof, really, is going to have to cover all those conditions,
0:04:43 and that’s kind of an infinitely intractable problem.
0:04:48 You couldn’t encode enough to do the inference you needed for your plans to be successful.
0:04:49 Right.
0:04:56 And so one of the big changes between the ’80s and 2019 is that we no longer really think
0:05:03 in the field of AI that inference is proving things, instead it’s building a plausible
0:05:05 chain of argument.
0:05:10 And it might be wrong, and if it goes wrong, if there is a banana in the tailpipe, you’ll
0:05:12 deal with it when it happens and when you figure it out.
0:05:13 Right.
0:05:18 So we move from certainty and proof to sort of probabilistic reasoning, right, Bayesian
0:05:20 techniques started becoming popular.
0:05:21 Right.
0:05:25 And so around 19– in the late ’90s, in fact.
0:05:31 So if you look at the history of machine learning, there’s an interesting trajectory where in
0:05:38 maybe up to the mid ’80s, things were pretty much focused on symbolic representations.
0:05:43 Actually, if you go back to the ’60s, there were the perceptron, but then it got swallowed
0:05:50 up by the end of the ’60s by symbolic representations and trying to reason that way and trying to
0:05:52 learn those kind of symbolic structures.
0:05:59 Then when the neural net wave came in around the late ’80s, early ’90s, that started competing
0:06:02 with the idea of symbolic representations.
0:06:09 But then in the late ’90s, the statisticians moved in and probabilistic methods became
0:06:11 very popular.
0:06:16 And at the time, there was this– if you look at this history, you can’t help but realize
0:06:22 what a social phenomenon technology advances in sciences and technology.
0:06:24 People influencing each other at conferences.
0:06:25 Right.
0:06:29 They were shaming them into adopting a new paradigm.
0:06:36 And so one of the slogans, or one of the phrases you kept hearing when people started working
0:06:41 on probabilistic, statistical probabilistic methods, they would never call them that.
0:06:48 They would have called them, instead, principled probabilistic methods, just to kind of shine
0:06:53 the light on the distinction between neural nets, which are just somehow tuning a gazillion
0:06:57 parameters in the principled methods that were being used.
0:07:03 And so that became really the dominant paradigm in the late ’90s and kind of remained in charge
0:07:12 of the field up through till about 2009, 2010, when now, as everybody kind of knows, deep
0:07:18 networks made a very serious revolution showing that they could do all kinds of amazing things
0:07:20 that hadn’t been done before.
0:07:21 Yeah.
0:07:26 We really are living in a golden age here in deep learning in neural network land.
0:07:29 But let’s go back to the original sort of rebel group.
0:07:35 This is Jeff Hinton hanging out in the shadow of sort of first order logic and saying, “No,
0:07:36 this is going to work.”
0:07:40 I think they were loosely inspired by the architecture of the brain.
0:07:41 Is that–
0:07:42 Definitely.
0:07:43 Definitely.
0:07:49 The kinds of arguments Jerry Feldman was one of the people who gave some of these arguments.
0:07:55 She said, “Look, you recognize your mother in about 100 milliseconds.
0:08:03 Your neurons can’t switch state in faster than a few milliseconds.
0:08:10 And so it looks like at most the chain of inference that you’re doing to go from your
0:08:16 retina to recognize your mother can only be about 10 deep, just from the timing.”
0:08:17 Oh, fascinating.
0:08:21 So it was an argument of sort of how long it took to recognize your mother.
0:08:22 Right.
0:08:23 And then how slow your neurons are, right?
0:08:26 Because they’re basically, these are biochemical processes, right?
0:08:27 Right.
0:08:28 Fascinating.
0:08:31 So really a computational efficiency argument.
0:08:37 And therefore, Jerry would say, “There must be a lot of stuff happening in parallel.
0:08:41 It must be a very wide chain of inference if it’s only 10 layers deep.”
0:08:45 And then he says, “Look at the brain, look at visual cortex.”
0:08:46 Yeah.
0:08:47 You got it.
0:08:51 And so neuroscientists at this time were making progress in understanding the structure of
0:08:55 neurons and how they connected to each other and how they formed connections, and those
0:08:58 connections could change strength over time, right?
0:09:02 All mediated by chemical interactions in the computer science community was inspired by
0:09:03 this.
0:09:04 Definitely.
0:09:12 And the level of abstraction at which the computational neural nets met up with the
0:09:19 real biological neural nets was not a very detailed level, but where they kind of became
0:09:28 the same was this idea of distributed representations, that in fact it might be a collection of hundreds
0:09:35 or thousands or millions of neurons that simultaneously were firing that represent your mother instead
0:09:37 of a symbol.
0:09:38 Right.
0:09:39 Right.
0:09:46 It’s a completely different notion of what it even means to represent knowledge.
0:09:51 And really, one of the most exciting things that has come out of the last decade of research
0:10:00 in neural and deep networks is a better understanding, although we still don’t fully understand of
0:10:07 how these artificial neural networks can learn very, very useful representations.
0:10:14 And for me, a simple example of that, that in a sentence summarizes it, is we have neural
0:10:22 networks now that can take as input an image of photograph and output a text caption for
0:10:23 that photograph.
0:10:28 What kind of representation must be in the middle of that neural network in order to
0:10:35 actually capture the meaning well enough that you can go from a visual stimulus to the equivalent
0:10:43 textual content, that it’s really, it must be capturing a very basic core representation
0:10:45 of the meaning of that photograph.
0:10:46 Yeah.
0:10:50 And one of my favorite things about the brain, which is otherwise this very sort of slow
0:10:54 computer, right, if you just look at neuron speeds, is that not only can they do this,
0:10:59 but they can actually use this, the representation they’re deriving to actually inform our actions
0:11:01 and our plans and our goals, right?
0:11:05 So not only is it like this picture has a chair in it, but like I can sit in that chair.
0:11:06 I can simulate sitting in that chair.
0:11:10 I think like that chair is going to support my weight, and all of these things happen
0:11:14 in like milliseconds, despite the fact that the basic components of the brain are very
0:11:15 slow.
0:11:16 Yeah.
0:11:17 It’s an amazing thing.
0:11:21 In fact, now that you mentioned it, I have to tell you, a half of my research life these
0:11:29 days is in studying how the human brain represents meaning of language.
0:11:32 We use brain imaging methods to do this.
0:11:39 And in one set of studies, we put people in a fMRI scanner, and we showed them just common
0:11:49 nouns like automobile, airplane, a knife, a chair, and so forth.
0:11:56 And we would get a picture literally with about three millimeter resolution of the three-dimensional
0:12:02 neural activity in their brain as they think about these different words.
0:12:06 And we’re interested in the question of all kinds of fundamental questions like what do
0:12:08 these representations look like?
0:12:12 Are they the same in your brain and my brain?
0:12:17 Given that they don’t appear instantaneously, by the way, it takes you about 400 milliseconds
0:12:22 to understand a word, if I put it on the screen in front of you.
0:12:24 What happens during that 400 milliseconds?
0:12:30 How do these representations evolve and come to be?
0:12:36 And one of the most interesting things we found, we studied this question by training
0:12:46 a machine learning system to take as input an arbitrary noun and to predict the brain
0:12:49 image that we will see if a person reads that noun.
0:12:56 Now, we only had data for 60 nouns at that time, so we didn’t train it on every noun
0:12:57 in the world.
0:13:00 We trained it on 60.
0:13:05 In fact, what we did was we trained it only on 58, so we could hold out two nouns that
0:13:06 we hadn’t seen.
0:13:12 And then we would test how well it could extrapolate to new nouns that you had never seen by showing
0:13:18 it the two held out nouns and having it predict the images.
0:13:21 Then we’d show it two images and we’d say, “Well, which of those is strawberry and which
0:13:23 of those is airplane?”
0:13:26 And it was right 80% of the time.
0:13:29 So, you could actually predict essentially brain state, right?
0:13:31 I’m going to show you a strawberry.
0:13:35 Let me predict the configuration of your neurons and who’s lighting up and who’s not.
0:13:42 So then we had a model that we trained with machine learning that captured something about
0:13:44 representations in the brain.
0:13:50 We used that to discover that the representations are almost identical in your brain and mind.
0:13:56 We could train on one set of people and decode what other people were thinking about.
0:14:02 And we also found that the representations themselves are grounded in parts of the brain
0:14:05 that are associated with perception.
0:14:10 So, if I give you a word like “peach,” the parts of your brain that code the meaning
0:14:16 of that are the ones associated with the sense of taste and manipulation because sometimes
0:14:19 you pick up a peach in visual color.
0:14:21 Yeah, that is fascinating.
0:14:25 Well, it’s so exciting to think that the brain structures are identical across people
0:14:30 because what everybody wants is sort of that, remember that scene in the matrix where you
0:14:34 sort of like you’re jacked straight into your brain and you’re like, “Oh, now I know Kung-Fu.”
0:14:35 Right?
0:14:36 Like this is what we want, right?
0:14:42 We want to learn new skills and sort of new facts and new inferences just like loading
0:14:43 an SD card, right?
0:14:48 And so, the fact that we’re sort of converging to the same structures in the brain at least
0:14:50 makes that theoretically possible.
0:14:51 We’re a ways away from that.
0:14:52 We’re a ways away from that.
0:14:53 But I’m with you.
0:14:54 Yeah, awesome.
0:15:01 So, another area that interests you is finding biases and why don’t we start by distinguishing
0:15:06 sort of two types of biases because when you hear the word “bias” today in machine learning,
0:15:10 you’re mostly thinking about things like, “Gee, let me make sure my data set is representative
0:15:13 so I don’t draw the wrong conclusion from that,” right?
0:15:18 So the classic example being here that I don’t do good recognition on people with darker skin
0:15:23 because I didn’t have enough of those samples in my data set and so the bias here is you’ve
0:15:29 selected a very small subset of the target data set that you want to cover and make predictions
0:15:31 on and therefore your predictions are poor, right?
0:15:33 So, that’s one sense of bias.
0:15:37 But there’s another sense of bias, that statistical bias, which is kind of what you want out of
0:15:38 algorithms.
0:15:39 So, maybe talk about this notion.
0:15:40 Yeah, sure.
0:15:45 And this is really a very important issue right now because now that machine learning
0:15:52 is being used in practice in many different ways, the issue of bias really is very important
0:15:54 to deal with.
0:16:00 You gave an example, another example would be, for instance, you have some historical
0:16:06 loan applications in which ones were approved, but maybe there’s some bias that say people
0:16:12 of one gender receive fewer loan approvals just because of their gender and if that’s
0:16:18 inherent in the data and you train a machine learning system that’s successful, well, it’s
0:16:22 probably going to learn the patterns that are in that data.
0:16:29 So, the notion of what I’ll call social bias, socially unacceptable bias, is really this
0:16:36 idea that you want the data set to reflect the kind of decision making that you want
0:16:39 the program to make if you’re going to train the program.
0:16:44 And that’s kind of the common sense notion of bias that most people talk about.
0:16:50 But there’s a lot of confusion in the field right now because bias is also used in statistical
0:16:55 machine learning to really with a very different meaning.
0:17:02 We’ll say that an algorithm is unbiased if the patterns that it learns, the decision
0:17:10 rules that it learns for approving loans, for example, reflect correctly the patterns that
0:17:12 are in the data.
0:17:18 Also that notion of statistically unbiased just means the algorithm’s doing its job of
0:17:21 recapitulating the decisions that are in the data.
0:17:28 The notion of the data itself being biased is really an orthogonal notion.
0:17:32 And there’s some interesting research going on now.
0:17:39 So for example, typically when we train a machine learning system, say to do loan approval,
0:17:45 a typical thing would be you can think of these machine learning algorithms as optimization
0:17:46 algorithms.
0:17:54 They’re going to tune maybe the parameters of your deep network so that they maximize
0:18:00 the number of decisions that they make that agree with the training examples.
0:18:06 But if your training examples have this kind of bias that maybe females receive fewer loan
0:18:09 approvals than males.
0:18:13 There’s some new work where people say, well, let’s change that objective that we’re trying
0:18:22 to optimize in addition to fitting the decisions that are in the training data as well as possible.
0:18:29 Let’s put another constraint that the probability of a female being approved for a loan has
0:18:33 to be equal to the probability of a male being approved.
0:18:38 And then subject to that constraint, we’ll try to match as many decisions as possible.
0:18:44 So there’s a lot of work right now in really technical work trying to understand if there
0:18:51 are ways of thinking more creatively, more imaginatively about how to even frame the
0:18:58 machine learning problem so that we can take what might be biased datasets but impose constraints
0:19:01 on the decision rules that we want to learn from those.
0:19:03 Yeah, that’s super interesting.
0:19:08 We’re sort of envisioning the world we want rather than the data of the world that we
0:19:13 came from because we might not be happy with the representation of the representedness
0:19:16 I guess, of the data that we came from.
0:19:20 And it’s causing people to look a lot more carefully at even the very notion of what
0:19:25 it means to be biased and what it means to be fair.
0:19:29 Are there good measures for fairness that the community is driving towards or do we
0:19:34 not really have a sort of an objective measure of fairness?
0:19:35 We don’t.
0:19:41 We don’t have an objective measure and there’s a lot of activity right now to discussing
0:19:48 that, including people like our philosophy professor, David Danks, who is very much part
0:19:54 of this discussion and social scientists, technology people, all getting together.
0:20:01 And in fact, there are now a couple of conferences centered around how to introduce fairness and
0:20:07 explainability and trust in AI systems.
0:20:11 This is a very important issue, but it’s not only technical.
0:20:16 It’s partly getting our philosophical, social, trying to get our heads around what it is
0:20:17 that we really want.
0:20:21 That’s a beautiful thing about AI and about computers in general.
0:20:27 It forces you to be way more precise when you are getting a computer to do it about
0:20:29 what you want.
0:20:36 And so even if you just think about self-driving cars, we have, when I was 16, I took a test
0:20:39 and I was approved to be a human driver.
0:20:43 They never asked me questions about whether I would swerve to hit the old lady or swerve
0:20:44 to hit the baby carriage.
0:20:45 Right.
0:20:47 Charlie’s problem was not on the DMV test.
0:20:48 Exactly.
0:20:50 But it’s on the test for the computers.
0:20:51 Right.
0:20:53 Yeah, it’s really interesting that we sort of hold computers to a different standard
0:20:55 because we’re programming them, right?
0:20:59 We can be explicit and we can have them sort of hit goals or not, right?
0:21:04 And those are design decisions rather than sort of, you know, bundled into a brain, right,
0:21:05 of a person.
0:21:06 Yeah.
0:21:12 And so I think of, you know, look, banks historically have hired loan officers.
0:21:16 Those loan officers may or may not be fair, right, according to the definitions that we’re
0:21:21 sort of talking about now, but we kind of hold those humans, those human loan officers to
0:21:24 a different standard than we would hold the algorithms.
0:21:25 That’s true.
0:21:27 I mean, who knows which way it will go in the future.
0:21:33 If we continue to have human loan officers and some computer loan officers, will we
0:21:39 up the constraints on the humans so that they pass the same qualifications?
0:21:45 Or will we drop the constraints on the computers so that they’re no more titrated than the
0:21:46 people?
0:21:47 Yeah.
0:21:48 That’s fascinating, right?
0:21:49 Who is master?
0:21:50 Who is the student, right?
0:21:55 The intuitive thing is let’s make the humans the models that we train our systems to approach,
0:21:56 right?
0:21:58 Like human competence being the goal.
0:22:00 The other way to think about it is no, right?
0:22:04 We can actually introduce constraints like, you know, equal number of men and women or
0:22:07 equal number of this ethnicity versus another ethnicity.
0:22:12 And our algorithms as a result of those constraints could be more fair than humans.
0:22:14 And so we invert it, right?
0:22:16 Let’s get humans up to that level of impartiality.
0:22:17 Right.
0:22:21 Like maybe the algorithm can end up teaching the human how to make those decisions in a
0:22:26 different way so that the fairness outcome you want is really achieved.
0:22:27 Yeah.
0:22:28 That’s fascinating.
0:22:32 And it’s great that sort of not just computer scientists are involved in this conversation
0:22:36 about the ethicists and the social scientists who are weighing in.
0:22:40 So that gives me hope that, you know, sort of smart people across disciplines are really
0:22:41 grappling with this.
0:22:43 So we get the outcomes that we want.
0:22:49 Well, sort of a related topic to this, right, sort of social impact of AI.
0:22:56 You recently co-wrote a paper with MIT’s Eric Bernholzen about the workplace implications.
0:23:00 And I think you also testified on Capitol Hill about what AI is going to do to jobs.
0:23:04 So why don’t you talk a little bit about what you guys found in the paper?
0:23:10 Well, this actually started with Eric and I co-chairing a National Academy study on
0:23:18 automation in the workforce, which was a two-year affair with a committee of about 15 experts
0:23:24 from around the country who were economists, social scientists, labor experts, technologists.
0:23:30 And in that study, I think we learned so much.
0:23:35 It turns out when you really dig into the question of what’s going to be the impact
0:23:39 of AI and automation on jobs.
0:23:46 You can’t escape noticing that there are many different forces that automation and technology
0:23:48 is exerting on the workforce.
0:23:50 One of them, of course, is automation.
0:23:55 Toll booth operators are going away, do not sign up to be a toll booth operator.
0:24:02 But in other kinds of jobs, instead of the job going away, there will be a shift, a redistribution
0:24:05 of the tasks.
0:24:07 So take, for example, a doctor.
0:24:12 A doctor has multiple tasks, for instance, they have to diagnose the patient, they have
0:24:17 to generate some possible therapies, they have to have a heart-to-heart discussion with
0:24:25 the patient about which of those therapies the patient elects to follow, and they have
0:24:26 to bill the patient.
0:24:31 Now computers are getting, computers are pretty good at billing, but they’re getting better
0:24:35 at diagnosis and they’re getting better at suggesting therapies.
0:24:41 For example, in just in the last couple of years, we’ve seen computers that are at the
0:24:49 same level, if not a little better than doctors at things like diagnosing skin cancer and
0:24:50 other kinds of diseases.
0:24:55 The radiologists, the tissue biopsies, all of these things, we’re using these computer
0:24:59 vision techniques to get very good performance.
0:25:02 So what does this mean about the future of doctors?
0:25:07 Well, I think what it means is automation happens at the level of the individual tasks,
0:25:09 not at the job level.
0:25:16 If a job is a bundle of tasks, like diagnosis, therapy, heart-to-heart chat, what’s going
0:25:24 to happen is computers will provide future doctors with more assistance, in some degrees,
0:25:30 hopefully automating billing, but some amount of automation or advice giving.
0:25:36 But for other tasks, like having that heart-to-heart chat, we’re very, very far from when computers
0:25:40 are going to be able to do anything close to that.
0:25:44 Good bedside manner is not going to be a future of your RoboDoc anytime soon.
0:25:53 And so what you find, if you look into this, and Eric and I recently had a paper in Science
0:25:59 with a more detailed study of this, but what you find is that the majority of jobs are
0:26:04 not like toll-booth operators, where there’s just one task, and if that gets automated,
0:26:05 that’s the end of the job.
0:26:15 The majority of jobs, like podcast interviewer, or computer, or professor, or doctor, really
0:26:16 are a bundle of tasks.
0:26:23 And so what’s going to happen is that, according to our study, the majority, more than half
0:26:29 of jobs, are going to be influenced, impacted by automation, but the impact won’t be a
0:26:30 elimination.
0:26:34 It’ll be a redistribution of the time that you spend on different tasks.
0:26:43 And we even conjecture that successful businesses in the future will, to some degree, be redefining
0:26:49 what the collection of jobs is that they’re hiring for.
0:26:53 Because they still have to cover the tasks through some combination of automation and
0:27:01 manual work, but the current bundles of tasks that form jobs today might shift dramatically.
0:27:06 So the key insight is to think of a job as a bundle of tasks, and that bundle might change
0:27:13 over time as AI enters and says, well, look, this specific task I’m very good at in algorithm
0:27:16 land, and so let’s get humans to focus on other things.
0:27:19 We just need to think of them as differently bundled.
0:27:24 Well, the last topic I wanted to talk with you about, Tom, is around whether this is
0:27:27 the best time ever for AI research.
0:27:29 So we started the grand campaign.
0:27:32 Some would argue summer of 1956 with the Dartmouth conference.
0:27:35 And we’ve had several winters and summers.
0:27:36 Where are we now?
0:27:39 And then what are you most excited about looking into the future?
0:27:45 I think we’re absolutely at the best time ever for the field of artificial intelligence.
0:27:48 And there have been, as you say, ups and downs over the year.
0:27:55 And for example, in the late ’80s, AI was very hot, and there was great expectation
0:27:57 of the things it would be able to do.
0:28:02 There was also great fear, by the way, of what Japan was going to do.
0:28:03 Yeah.
0:28:08 This is the fifth generation supercomputer and the entire national policy of Japan, right?
0:28:10 Focusing on this area.
0:28:16 And so in the U.S., there was great concern that this would have a big impact, Japan would
0:28:18 take over the economy.
0:28:21 So there are some parallels here.
0:28:24 Now again, AI, it’s very popular.
0:28:27 People have great expectations.
0:28:31 And there’s a great amount of fear, I have to say, about what China and other countries
0:28:34 might be doing in AI.
0:28:44 But one really, really important difference is that, unlike in the 1980s, right now, there’s
0:28:48 a huge record of accomplishment over the last 10 years.
0:28:57 We already have AI and machine learning being used across many, many different, really economically
0:29:00 valuable tasks.
0:29:06 And therefore, I think, really, there’s very little chance that we’ll have a crash, although
0:29:11 I completely agree with my friends who say, “But isn’t AI overhyped?”
0:29:13 Absolutely, it’s overhyped.
0:29:20 But there is enough reality there to keep the field progressing and to keep commercial
0:29:25 interest and to keep economic investment going for a long time to come.
0:29:27 So you would argue, this time it really is different.
0:29:28 It really is different.
0:29:31 Because we have real working stuff to point to.
0:29:36 And over the next 10 years, we’ll have a whole lot more real working stuff that influences
0:29:38 our lives daily.
0:29:44 So as a university researcher, I look at this and I say, “Where is this going and what should
0:29:47 we be doing in the university?”
0:29:51 If you want to think about that, you have to realize just how much progress there was
0:29:52 in the last 10 years.
0:29:59 When the iPhone came out, I guess that’s 11 years ago, computers were deaf and blind.
0:30:03 When the iPhone came out, you could not talk to your iPhone.
0:30:09 This is such a weird idea, but you could not talk to your iPhone because speech recognition
0:30:11 didn’t work.
0:30:19 And now computers can transcribe voice to text just as well as people.
0:30:26 Similarly, when you pointed your camera at a scene, it couldn’t recognize with any accuracy
0:30:29 the things that were on the table in the scene.
0:30:34 And now it can do that with about the same accuracy comparable to humans.
0:30:38 And in some visual tasks, like skin cancer detection, even better than you know.
0:30:40 Even better than trained doctors, yeah.
0:30:41 Better than trained.
0:30:44 So it’s hard to remember that it’s only been 10 years.
0:30:47 And that’s the thing about progress in AI.
0:30:53 You forget because it becomes so familiar, just how dramatic the improvement has been.
0:30:55 Now think about what that means.
0:31:00 That means we’re really in the first five years of having computers that are not deaf
0:31:01 and blind.
0:31:08 And now think about what are the kinds of intelligence that you could exhibit if you
0:31:13 were deaf and blind, well, you could do game playing and inventory control.
0:31:17 You could do things that don’t involve perception.
0:31:24 But once you can perceive the world and converse in the world, there’s an explosion of new
0:31:25 applications you can do.
0:31:30 So we’re going to have garage door openers that open for you because they recognize that
0:31:32 your car coming down the driveway.
0:31:37 We’re going to have many, many things that we haven’t even thought about that just leverage
0:31:43 off this very recent progress in perceptual AI.
0:31:49 So going forward, I think a lot about how I want to invest my own research time.
0:31:51 I’m interested still in machine learning.
0:31:53 I’m very proud of the field of machine learning.
0:31:55 It’s come a long way.
0:32:01 But I’m also somebody who thinks we’re only at the beginning.
0:32:05 I think if you want to know the future of machine learning, all you need to do is look
0:32:10 at how humans learn and computers don’t yet.
0:32:16 So we learn, for example, we do learn statistically like computers do.
0:32:24 My phone watches me over time and statistically, it eventually learns where it thinks my house
0:32:27 is and where it thinks my work is.
0:32:30 It statistically learns what my preferences are.
0:32:33 But I also have a human assistant.
0:32:39 And if she tried to figure out what I wanted her to do by statistically watching me do
0:32:43 things a thousand times, I would have fired her so long ago.
0:32:46 A lot of false positives and false negatives, right?
0:32:47 Right.
0:32:48 So she doesn’t learn that way.
0:32:51 She learns by having a conversation with me.
0:32:56 I go into the office and I say, “Hey, this semester I’m team teaching a course with
0:33:00 Katarina on deep reinforcement learning.
0:33:02 Here’s what I want you to do.
0:33:04 Whenever this happens, you do this.
0:33:10 Whenever we’re preparing to hand out a homework assignment, if it hasn’t been pretested by
0:33:16 the teaching assistants two days before handout, you send a note saying, “Get that thing pretested.”
0:33:22 So what I do is I teach her and we have a conversation, she clarifies.
0:33:32 So one of the new paradigms for machine learning that I predict we will see in the coming decade
0:33:35 is what I’ll call conversational learning.
0:33:42 Use the kind of conversational interfaces that we have, say, with our phones to allow
0:33:52 people to literally teach their devices what they want them to do instead of have the device
0:33:54 statistically learn it.
0:34:01 And if you go down that road, here’s a really interesting angle on it.
0:34:07 It becomes kind of like replacing computer programming with natural language instruction.
0:34:15 So I’ll give you an example of a prototype system that we’ve been working on together
0:34:19 with Brad Myers, one of our faculty in HCI.
0:34:25 It allows you to say to your phone something like, “Whenever it snows at night, I want
0:34:27 you to wake me up 30 minutes earlier.”
0:34:33 If you live in Pittsburgh, this is a useful app, and none of the California engineers
0:34:34 have created that app.
0:34:40 And today, I could create that app if I took the trouble of learning the computer language
0:34:43 of the phone, I could program it.
0:34:49 But only far less than 1% of phone users can actually have taken the time to learn the
0:34:51 language of the computer.
0:34:56 We’re giving the phone the chance to learn the language of the person.
0:35:03 So with our phone prototype, if you say, “Whenever it snows at night, wake me up 30 minutes
0:35:08 earlier,” it says, “I don’t understand, do you want to teach me?”
0:35:12 And you can say, “Yes, here’s how you find out if it’s snowing at night.”
0:35:17 You open up this weather app right here, and where it says current conditions, if that
0:35:20 says SNOW, it’s snowing.
0:35:25 Here’s how you wake me up 30 minutes earlier, you open up that alarm app, and this number
0:35:27 you subtract 30 from it.
0:35:36 So with a combination of showing, demonstrating, and telling voice, we’re trying to give users
0:35:44 the opportunity to create their own apps, their own programs with the same kind of instruction,
0:35:49 voice, and demonstration that you would use if you were trying to teach me how to do it.
0:35:50 I love that.
0:35:53 It’s sort of a natural language for an end to, we have an investment in a company called
0:35:57 ift, if this, then that, which is you can program those things, but you have to be a
0:35:59 little sophisticated.
0:36:03 You’d like to just be able to talk to your phone and have it figured out how do I fill
0:36:05 the slots into ift that ift wants.
0:36:07 Exactly, and if this, then that is a wonderful thing.
0:36:14 It has a huge library of these apps that you can download, but as you say, you still have
0:36:17 to learn the language of the computer to create those.
0:36:21 We’re trying to have the computer learn the language of the person.
0:36:27 If that line of research plays out, and I believe it will this decade, we’ll be in a very different
0:36:34 world because we’ll be in a world where instead of the elite few, less than 1% of phone users
0:36:39 being able to program, it’ll be 99% of phone users who can do this.
0:36:45 Now, think about what that does for the whole conception of how we think about human-computer
0:36:46 interaction.
0:36:47 Yeah.
0:36:49 That’s a profound shift in society, right?
0:36:55 Just like everybody became literate, not just the priests, and look what happened to society.
0:36:56 Exactly.
0:36:57 Yeah.
0:36:59 Think about what it means for the future of jobs.
0:37:04 Right now, if you have a computer introduced as your teammate, you, the human, and the
0:37:11 computer are a team, well, the computer is frozen, and the teammate who gets to do the
0:37:16 adapting is the human because the computer is a fixed functionality.
0:37:21 What if in that team, the human could just teach the computer how they want the computer
0:37:22 to help them do their job.
0:37:25 It would be a completely different dynamic.
0:37:27 It would change the future of work.
0:37:28 Yeah.
0:37:29 That’s fascinating.
0:37:33 And then I think another thread that you’re super interested in on the future of machine
0:37:37 learning is something around never-ending learning, so tell us about that.
0:37:38 Sure.
0:37:44 Again, I just go back to what do humans do that computers don’t yet, and computers are
0:37:49 very good at, say, learning to diagnose skin cancer.
0:37:54 You give it some very specific tasks and some data, but if you look at people, people learn
0:37:57 so many things.
0:37:59 We learn to all kinds of things.
0:38:00 You can tap the hands.
0:38:02 You can do double entry bookkeeping.
0:38:03 Right?
0:38:04 Right.
0:38:05 You can add numbers.
0:38:11 You can play music, all kinds of things, and a lot of those things we learn over time
0:38:17 in a kind of synergistic way, in a staged sequence.
0:38:21 First you learn to crawl, then you learn to walk, then you learn to run, then you learn
0:38:28 to ride a bike, and it wouldn’t make any sense to do them in the other sequence, because
0:38:31 you’re actually learning to learn.
0:38:37 When you acquire one skill, it puts you in a position that you now are capable of learning
0:38:38 the next skill.
0:38:44 So I’m very interested in what it would mean to give a computer that kind of capability
0:38:51 to do learning for days and weeks and years and decades.
0:38:59 We have a project we call our Never-Ending Language Learner, which started in 2010, running
0:39:03 24 hours a day trying to learn to read the web.
0:39:04 Fascinating.
0:39:07 And there’s something about sort of the longitudinal, right?
0:39:09 We started it in 2010 and it just keeps on going.
0:39:13 So it’s not just like transfer learning from one model to another.
0:39:15 It’s like long running.
0:39:20 It’s long running and it has many different learning tasks.
0:39:25 It’s building up a knowledge base of knowledge about the world.
0:39:33 But to keep it short, I’ll just say we’ve learned so much from that project about how
0:39:39 to organize the architecture of a system so that it can invent new learning tasks as it
0:39:46 goes so that it can get synergy once it learns one thing to become better at learning another
0:39:47 thing.
0:39:53 How, in fact, very importantly, it can use unlabeled data to train itself instead of
0:39:57 requiring an army of data labellers.
0:40:04 So I just think this is an area that’s relatively untouched in the machine learning field.
0:40:11 But looking forward, we’re already seeing an increasing number of embedded machine learning
0:40:14 systems in continuous use.
0:40:22 And as we see more and more of those in the internet of things and elsewhere, the opportunity
0:40:29 for learning continuously for days and weeks and months and years and decades is increasingly
0:40:30 there.
0:40:36 We ought to be developing the ideas, the concepts of how to organize those systems to take advantage
0:40:37 of that.
0:40:38 Yeah.
0:40:41 I love both of these design approaches in that they’re sort of inspired by humans and
0:40:50 sort of humans are mysteriously good at learning and adapting and they sort of shine a spotlight
0:40:53 on where machine learning algorithms are not yet.
0:40:56 So it’s such a fertile area to look for inspiration.
0:41:00 Well, Tom, it’s been a great delight having you on the podcast.
0:41:03 Thanks for sharing about the history and the future of machine learning.
0:41:06 We can tell you’re still fired up after all of these decades.
0:41:11 And so that’s a great delight just to see somebody who is committed, basically, their
0:41:17 life to understanding the mysteries of learning and wishing many good decades to come as you
0:41:19 continue working on it.
0:41:20 Thanks.
0:41:21 Thanks for doing this podcast.
0:41:25 I think it’s a great thing to get a conversation going and it’s a great contribution to do
0:41:25 that.
How have we gotten to where were are with machine learning? Where are we going?
a16z Operating Partner Frank Chen and Carnegie Mellon professor Tom Mitchell first stroll down memory lane, visiting the major landmarks: the symbolic approach of the 1970s, the ”principled probabalistic methods” of the 1980s, and today’s deep learning phase. Then they go on to explore the frontiers of research. Along the way, they cover:
- How planning systems from the 1970s and early 1980s were stymied by the ”banana in the tailpipe” problem
- How the relatively slow neurons in our visual cortex work together to deliver very speedy and accurate recognition
- How fMRI scans of the brain reveal common neural patterns across people when they are exposed to common nouns like chair, car, knife, and so on
- How the computer science community is working with social scientists (psychologists, economists, and philosophers) on building measures for fairness and transparency for machine learning models
- How we want our self-driving cars to have reasonable answers to the Trolley Problem, but no one sitting for their DMV exam is ever asked how they would respond
- How there were inflated expectations (and great social fears) for AI in the 1980s, and how the US concerns about Japan compare to our concerns about China today
- Whether this is the best time ever for AI and ML research and what continues to fascinate and motivate Tom after decades in the field
The views expressed here are those of the individual AH Capital Management, L.L.C. (“a16z”) personnel quoted and are not the views of a16z or its affiliates. Certain information contained in here has been obtained from third-party sources, including from portfolio companies of funds managed by a16z. While taken from sources believed to be reliable, a16z has not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation.
This content is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. You should consult your own advisers as to those matters. References to any securities or digital assets are for illustrative purposes only, and do not constitute an investment recommendation or offer to provide investment advisory services. Furthermore, this content is not directed at nor intended for use by any investors or prospective investors, and may not under any circumstances be relied upon when making a decision to invest in any fund managed by a16z. (An offering to invest in an a16z fund will be made only by the private placement memorandum, subscription agreement, and other relevant documentation of any such fund and should be read in their entirety.) Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z, and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by Andreessen Horowitz (excluding investments and certain publicly traded cryptocurrencies/ digital assets for which the issuer has not provided permission for a16z to disclose publicly) is available at https://a16z.com/investments/.
Charts and graphs provided within are for informational purposes solely and should not be relied upon when making any investment decision. Past performance is not indicative of future results. The content speaks only as of the date indicated. Any projections, estimates, forecasts, targets, prospects, and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Please see https://a16z.com/disclosures for additional important information.