Summary & Insights
0:00:05 The content here is for informational purposes only, should not be taken as legal business
0:00:10 tax or investment advice, or be used to evaluate any investment or security and is not directed
0:00:14 at any investors or potential investors in any A16Z fund.
0:00:17 For more details, please see a16z.com/disclosures.
0:00:21 Hi, and welcome to the A16Z podcast.
0:00:22 I’m Frank Chen.
0:00:28 Today, I’m here with Carnegie Mellon’s Professor Tom Mitchell, who has been involved with machine
0:00:30 learning basically his entire career.
0:00:34 So I’m super excited to have this conversation with Tom, where he can tell us a little bit
0:00:37 about the history and where all of our techniques came from.
0:00:40 And we’ll spend time talking about the future, where the field is going.
0:00:45 So Carnegie Mellon’s been involved in sort of standing up the fundamental teaching institutions
0:00:50 and research institutions of the big areas, computer science, artificial intelligence
0:00:52 and machine learning.
0:00:56 So take us back to the early days, the you and Newell and Jeff Hinton are teaching this
0:00:57 class.
0:00:58 What was your curriculum like?
0:00:59 Like what were you teaching?
0:01:02 Pretty different, I imagine, than what we teach undergrads today.
0:01:03 That’s right.
0:01:05 Well, at the time, this was the 1980s.
0:01:13 So artificial intelligence at that point was dominated by what we would call symbolic methods.
0:01:19 Where things like formal logic would be used to do inference.
0:01:24 And much of machine learning was really about learning symbolic structures, symbolic representations
0:01:25 of knowledge.
0:01:31 But there was this kind of young whipper snapper, Jeff Hinton, who had a different idea.
0:01:41 And so he was working on a book with Rommelhart McClellan that became a very well-known parallel
0:01:47 data processing book that kind of launched the field of neural nets.
0:01:49 And they were, if I remember, psychologists, right?
0:01:58 Yeah. Jay McClellan is a psychologist here at CMU, Rommelhart, kind of a neuroscientist.
0:02:01 And more, he was a very broad person.
0:02:07 And Jeff, so the three of them were kind of the rebels who were taking things off in a
0:02:09 different paradigm.
0:02:14 The empire wanted us to do research on knowledge representation and inference and first-order
0:02:15 logic.
0:02:20 I remember as an undergrad, I took this computer aided class that John H.M.M.D. wrote called
0:02:24 Tarski’s World, where we learned all about first-world logic.
0:02:25 What could you prove?
0:02:26 What could you not prove?
0:02:30 And so that’s what the establishment, quote, unquote, was teaching.
0:02:37 And then Jeff was the rebel off in neural network land, and he gets his reprise later.
0:02:40 So take us back to the world of knowledge representation, because I’m actually seeing
0:02:44 a lot of startups these days who are trying to bring back some of these techniques to
0:02:49 complement deep learning, because there are well-known challenges with deep learning.
0:02:52 Like, we’re not encoding any priors, we’re learning everything for the first time, we
0:02:56 need tons of labeled data sets to make progress.
0:03:00 And so take us back to the days of knowledge representation.
0:03:04 What were we trying to solve with those set of techniques, and how might we use them today?
0:03:13 So back in the ’80s and the ’90s, and I have to say that some of the really senior people
0:03:20 in the field were totally devoted to this paradigm of logical inference, logical representations.
0:03:27 People like John McCarthy, for example, were very strong proponents of this, and really
0:03:34 essentially just saw that reasoning is theorem-proving, and therefore if we’re going to get computers
0:03:38 to do it, that’s what we have to do.
0:03:42 There were some problems with that, and there still are.
0:03:48 One thing that I remember from back then that was an example was the banana in the tailpipe
0:03:50 problem.
0:03:55 These logical systems were used to reason to do things like, how would you plan a sequence
0:03:57 of actions to achieve a goal?
0:03:59 Like, how would you get from here to the airport?
0:04:05 Well, you’d walk to your car, you’d put the key in, turn the car on, you’d drive out of
0:04:10 the parking lot, get on the interstate, go to the airport exit, et cetera.
0:04:12 But what if there’s a banana in the tailpipe?
0:04:17 Even back then, before it became a meme in Beverly Hills Cop, we were worried about
0:04:19 the banana in the tailpipe.
0:04:20 That’s right.
0:04:25 And the point of the banana in the tailpipe is there are an infinite number of other things
0:04:31 that you don’t say when you spin out a plan like that.
0:04:39 And any proof, if it’s a proof, really, is going to have to cover all those conditions,
0:04:43 and that’s kind of an infinitely intractable problem.
0:04:48 You couldn’t encode enough to do the inference you needed for your plans to be successful.
0:04:49 Right.
0:04:56 And so one of the big changes between the ’80s and 2019 is that we no longer really think
0:05:03 in the field of AI that inference is proving things, instead it’s building a plausible
0:05:05 chain of argument.
0:05:10 And it might be wrong, and if it goes wrong, if there is a banana in the tailpipe, you’ll
0:05:12 deal with it when it happens and when you figure it out.
0:05:13 Right.
0:05:18 So we move from certainty and proof to sort of probabilistic reasoning, right, Bayesian
0:05:20 techniques started becoming popular.
0:05:21 Right.
0:05:25 And so around 19– in the late ’90s, in fact.
0:05:31 So if you look at the history of machine learning, there’s an interesting trajectory where in
0:05:38 maybe up to the mid ’80s, things were pretty much focused on symbolic representations.
0:05:43 Actually, if you go back to the ’60s, there were the perceptron, but then it got swallowed
0:05:50 up by the end of the ’60s by symbolic representations and trying to reason that way and trying to
0:05:52 learn those kind of symbolic structures.
0:05:59 Then when the neural net wave came in around the late ’80s, early ’90s, that started competing
0:06:02 with the idea of symbolic representations.
0:06:09 But then in the late ’90s, the statisticians moved in and probabilistic methods became
0:06:11 very popular.
0:06:16 And at the time, there was this– if you look at this history, you can’t help but realize
0:06:22 what a social phenomenon technology advances in sciences and technology.
0:06:24 People influencing each other at conferences.
0:06:25 Right.
0:06:29 They were shaming them into adopting a new paradigm.
0:06:36 And so one of the slogans, or one of the phrases you kept hearing when people started working
0:06:41 on probabilistic, statistical probabilistic methods, they would never call them that.
0:06:48 They would have called them, instead, principled probabilistic methods, just to kind of shine
0:06:53 the light on the distinction between neural nets, which are just somehow tuning a gazillion
0:06:57 parameters in the principled methods that were being used.
0:07:03 And so that became really the dominant paradigm in the late ’90s and kind of remained in charge
0:07:12 of the field up through till about 2009, 2010, when now, as everybody kind of knows, deep
0:07:18 networks made a very serious revolution showing that they could do all kinds of amazing things
0:07:20 that hadn’t been done before.
0:07:21 Yeah.
0:07:26 We really are living in a golden age here in deep learning in neural network land.
0:07:29 But let’s go back to the original sort of rebel group.
0:07:35 This is Jeff Hinton hanging out in the shadow of sort of first order logic and saying, “No,
0:07:36 this is going to work.”
0:07:40 I think they were loosely inspired by the architecture of the brain.
0:07:41 Is that–
0:07:42 Definitely.
0:07:43 Definitely.
0:07:49 The kinds of arguments Jerry Feldman was one of the people who gave some of these arguments.
0:07:55 She said, “Look, you recognize your mother in about 100 milliseconds.
0:08:03 Your neurons can’t switch state in faster than a few milliseconds.
0:08:10 And so it looks like at most the chain of inference that you’re doing to go from your
0:08:16 retina to recognize your mother can only be about 10 deep, just from the timing.”
0:08:17 Oh, fascinating.
0:08:21 So it was an argument of sort of how long it took to recognize your mother.
0:08:22 Right.
0:08:23 And then how slow your neurons are, right?
0:08:26 Because they’re basically, these are biochemical processes, right?
0:08:27 Right.
0:08:28 Fascinating.
0:08:31 So really a computational efficiency argument.
0:08:37 And therefore, Jerry would say, “There must be a lot of stuff happening in parallel.
0:08:41 It must be a very wide chain of inference if it’s only 10 layers deep.”
0:08:45 And then he says, “Look at the brain, look at visual cortex.”
0:08:46 Yeah.
0:08:47 You got it.
0:08:51 And so neuroscientists at this time were making progress in understanding the structure of
0:08:55 neurons and how they connected to each other and how they formed connections, and those
0:08:58 connections could change strength over time, right?
0:09:02 All mediated by chemical interactions in the computer science community was inspired by
0:09:03 this.
0:09:04 Definitely.
0:09:12 And the level of abstraction at which the computational neural nets met up with the
0:09:19 real biological neural nets was not a very detailed level, but where they kind of became
0:09:28 the same was this idea of distributed representations, that in fact it might be a collection of hundreds
0:09:35 or thousands or millions of neurons that simultaneously were firing that represent your mother instead
0:09:37 of a symbol.
0:09:38 Right.
0:09:39 Right.
0:09:46 It’s a completely different notion of what it even means to represent knowledge.
0:09:51 And really, one of the most exciting things that has come out of the last decade of research
0:10:00 in neural and deep networks is a better understanding, although we still don’t fully understand of
0:10:07 how these artificial neural networks can learn very, very useful representations.
0:10:14 And for me, a simple example of that, that in a sentence summarizes it, is we have neural
0:10:22 networks now that can take as input an image of photograph and output a text caption for
0:10:23 that photograph.
0:10:28 What kind of representation must be in the middle of that neural network in order to
0:10:35 actually capture the meaning well enough that you can go from a visual stimulus to the equivalent
0:10:43 textual content, that it’s really, it must be capturing a very basic core representation
0:10:45 of the meaning of that photograph.
0:10:46 Yeah.
0:10:50 And one of my favorite things about the brain, which is otherwise this very sort of slow
0:10:54 computer, right, if you just look at neuron speeds, is that not only can they do this,
0:10:59 but they can actually use this, the representation they’re deriving to actually inform our actions
0:11:01 and our plans and our goals, right?
0:11:05 So not only is it like this picture has a chair in it, but like I can sit in that chair.
0:11:06 I can simulate sitting in that chair.
0:11:10 I think like that chair is going to support my weight, and all of these things happen
0:11:14 in like milliseconds, despite the fact that the basic components of the brain are very
0:11:15 slow.
0:11:16 Yeah.
0:11:17 It’s an amazing thing.
0:11:21 In fact, now that you mentioned it, I have to tell you, a half of my research life these
0:11:29 days is in studying how the human brain represents meaning of language.
0:11:32 We use brain imaging methods to do this.
0:11:39 And in one set of studies, we put people in a fMRI scanner, and we showed them just common
0:11:49 nouns like automobile, airplane, a knife, a chair, and so forth.
0:11:56 And we would get a picture literally with about three millimeter resolution of the three-dimensional
0:12:02 neural activity in their brain as they think about these different words.
0:12:06 And we’re interested in the question of all kinds of fundamental questions like what do
0:12:08 these representations look like?
0:12:12 Are they the same in your brain and my brain?
0:12:17 Given that they don’t appear instantaneously, by the way, it takes you about 400 milliseconds
0:12:22 to understand a word, if I put it on the screen in front of you.
0:12:24 What happens during that 400 milliseconds?
0:12:30 How do these representations evolve and come to be?
0:12:36 And one of the most interesting things we found, we studied this question by training
0:12:46 a machine learning system to take as input an arbitrary noun and to predict the brain
0:12:49 image that we will see if a person reads that noun.
0:12:56 Now, we only had data for 60 nouns at that time, so we didn’t train it on every noun
0:12:57 in the world.
0:13:00 We trained it on 60.
0:13:05 In fact, what we did was we trained it only on 58, so we could hold out two nouns that
0:13:06 we hadn’t seen.
0:13:12 And then we would test how well it could extrapolate to new nouns that you had never seen by showing
0:13:18 it the two held out nouns and having it predict the images.
0:13:21 Then we’d show it two images and we’d say, “Well, which of those is strawberry and which
0:13:23 of those is airplane?”
0:13:26 And it was right 80% of the time.
0:13:29 So, you could actually predict essentially brain state, right?
0:13:31 I’m going to show you a strawberry.
0:13:35 Let me predict the configuration of your neurons and who’s lighting up and who’s not.
0:13:42 So then we had a model that we trained with machine learning that captured something about
0:13:44 representations in the brain.
0:13:50 We used that to discover that the representations are almost identical in your brain and mind.
0:13:56 We could train on one set of people and decode what other people were thinking about.
0:14:02 And we also found that the representations themselves are grounded in parts of the brain
0:14:05 that are associated with perception.
0:14:10 So, if I give you a word like “peach,” the parts of your brain that code the meaning
0:14:16 of that are the ones associated with the sense of taste and manipulation because sometimes
0:14:19 you pick up a peach in visual color.
0:14:21 Yeah, that is fascinating.
0:14:25 Well, it’s so exciting to think that the brain structures are identical across people
0:14:30 because what everybody wants is sort of that, remember that scene in the matrix where you
0:14:34 sort of like you’re jacked straight into your brain and you’re like, “Oh, now I know Kung-Fu.”
0:14:35 Right?
0:14:36 Like this is what we want, right?
0:14:42 We want to learn new skills and sort of new facts and new inferences just like loading
0:14:43 an SD card, right?
0:14:48 And so, the fact that we’re sort of converging to the same structures in the brain at least
0:14:50 makes that theoretically possible.
0:14:51 We’re a ways away from that.
0:14:52 We’re a ways away from that.
0:14:53 But I’m with you.
0:14:54 Yeah, awesome.
0:15:01 So, another area that interests you is finding biases and why don’t we start by distinguishing
0:15:06 sort of two types of biases because when you hear the word “bias” today in machine learning,
0:15:10 you’re mostly thinking about things like, “Gee, let me make sure my data set is representative
0:15:13 so I don’t draw the wrong conclusion from that,” right?
0:15:18 So the classic example being here that I don’t do good recognition on people with darker skin
0:15:23 because I didn’t have enough of those samples in my data set and so the bias here is you’ve
0:15:29 selected a very small subset of the target data set that you want to cover and make predictions
0:15:31 on and therefore your predictions are poor, right?
0:15:33 So, that’s one sense of bias.
0:15:37 But there’s another sense of bias, that statistical bias, which is kind of what you want out of
0:15:38 algorithms.
0:15:39 So, maybe talk about this notion.
0:15:40 Yeah, sure.
0:15:45 And this is really a very important issue right now because now that machine learning
0:15:52 is being used in practice in many different ways, the issue of bias really is very important
0:15:54 to deal with.
0:16:00 You gave an example, another example would be, for instance, you have some historical
0:16:06 loan applications in which ones were approved, but maybe there’s some bias that say people
0:16:12 of one gender receive fewer loan approvals just because of their gender and if that’s
0:16:18 inherent in the data and you train a machine learning system that’s successful, well, it’s
0:16:22 probably going to learn the patterns that are in that data.
0:16:29 So, the notion of what I’ll call social bias, socially unacceptable bias, is really this
0:16:36 idea that you want the data set to reflect the kind of decision making that you want
0:16:39 the program to make if you’re going to train the program.
0:16:44 And that’s kind of the common sense notion of bias that most people talk about.
0:16:50 But there’s a lot of confusion in the field right now because bias is also used in statistical
0:16:55 machine learning to really with a very different meaning.
0:17:02 We’ll say that an algorithm is unbiased if the patterns that it learns, the decision
0:17:10 rules that it learns for approving loans, for example, reflect correctly the patterns that
0:17:12 are in the data.
0:17:18 Also that notion of statistically unbiased just means the algorithm’s doing its job of
0:17:21 recapitulating the decisions that are in the data.
0:17:28 The notion of the data itself being biased is really an orthogonal notion.
0:17:32 And there’s some interesting research going on now.
0:17:39 So for example, typically when we train a machine learning system, say to do loan approval,
0:17:45 a typical thing would be you can think of these machine learning algorithms as optimization
0:17:46 algorithms.
0:17:54 They’re going to tune maybe the parameters of your deep network so that they maximize
0:18:00 the number of decisions that they make that agree with the training examples.
0:18:06 But if your training examples have this kind of bias that maybe females receive fewer loan
0:18:09 approvals than males.
0:18:13 There’s some new work where people say, well, let’s change that objective that we’re trying
0:18:22 to optimize in addition to fitting the decisions that are in the training data as well as possible.
0:18:29 Let’s put another constraint that the probability of a female being approved for a loan has
0:18:33 to be equal to the probability of a male being approved.
0:18:38 And then subject to that constraint, we’ll try to match as many decisions as possible.
0:18:44 So there’s a lot of work right now in really technical work trying to understand if there
0:18:51 are ways of thinking more creatively, more imaginatively about how to even frame the
0:18:58 machine learning problem so that we can take what might be biased datasets but impose constraints
0:19:01 on the decision rules that we want to learn from those.
0:19:03 Yeah, that’s super interesting.
0:19:08 We’re sort of envisioning the world we want rather than the data of the world that we
0:19:13 came from because we might not be happy with the representation of the representedness
0:19:16 I guess, of the data that we came from.
0:19:20 And it’s causing people to look a lot more carefully at even the very notion of what
0:19:25 it means to be biased and what it means to be fair.
0:19:29 Are there good measures for fairness that the community is driving towards or do we
0:19:34 not really have a sort of an objective measure of fairness?
0:19:35 We don’t.
0:19:41 We don’t have an objective measure and there’s a lot of activity right now to discussing
0:19:48 that, including people like our philosophy professor, David Danks, who is very much part
0:19:54 of this discussion and social scientists, technology people, all getting together.
0:20:01 And in fact, there are now a couple of conferences centered around how to introduce fairness and
0:20:07 explainability and trust in AI systems.
0:20:11 This is a very important issue, but it’s not only technical.
0:20:16 It’s partly getting our philosophical, social, trying to get our heads around what it is
0:20:17 that we really want.
0:20:21 That’s a beautiful thing about AI and about computers in general.
0:20:27 It forces you to be way more precise when you are getting a computer to do it about
0:20:29 what you want.
0:20:36 And so even if you just think about self-driving cars, we have, when I was 16, I took a test
0:20:39 and I was approved to be a human driver.
0:20:43 They never asked me questions about whether I would swerve to hit the old lady or swerve
0:20:44 to hit the baby carriage.
0:20:45 Right.
0:20:47 Charlie’s problem was not on the DMV test.
0:20:48 Exactly.
0:20:50 But it’s on the test for the computers.
0:20:51 Right.
0:20:53 Yeah, it’s really interesting that we sort of hold computers to a different standard
0:20:55 because we’re programming them, right?
0:20:59 We can be explicit and we can have them sort of hit goals or not, right?
0:21:04 And those are design decisions rather than sort of, you know, bundled into a brain, right,
0:21:05 of a person.
0:21:06 Yeah.
0:21:12 And so I think of, you know, look, banks historically have hired loan officers.
0:21:16 Those loan officers may or may not be fair, right, according to the definitions that we’re
0:21:21 sort of talking about now, but we kind of hold those humans, those human loan officers to
0:21:24 a different standard than we would hold the algorithms.
0:21:25 That’s true.
0:21:27 I mean, who knows which way it will go in the future.
0:21:33 If we continue to have human loan officers and some computer loan officers, will we
0:21:39 up the constraints on the humans so that they pass the same qualifications?
0:21:45 Or will we drop the constraints on the computers so that they’re no more titrated than the
0:21:46 people?
0:21:47 Yeah.
0:21:48 That’s fascinating, right?
0:21:49 Who is master?
0:21:50 Who is the student, right?
0:21:55 The intuitive thing is let’s make the humans the models that we train our systems to approach,
0:21:56 right?
0:21:58 Like human competence being the goal.
0:22:00 The other way to think about it is no, right?
0:22:04 We can actually introduce constraints like, you know, equal number of men and women or
0:22:07 equal number of this ethnicity versus another ethnicity.
0:22:12 And our algorithms as a result of those constraints could be more fair than humans.
0:22:14 And so we invert it, right?
0:22:16 Let’s get humans up to that level of impartiality.
0:22:17 Right.
0:22:21 Like maybe the algorithm can end up teaching the human how to make those decisions in a
0:22:26 different way so that the fairness outcome you want is really achieved.
0:22:27 Yeah.
0:22:28 That’s fascinating.
0:22:32 And it’s great that sort of not just computer scientists are involved in this conversation
0:22:36 about the ethicists and the social scientists who are weighing in.
0:22:40 So that gives me hope that, you know, sort of smart people across disciplines are really
0:22:41 grappling with this.
0:22:43 So we get the outcomes that we want.
0:22:49 Well, sort of a related topic to this, right, sort of social impact of AI.
0:22:56 You recently co-wrote a paper with MIT’s Eric Bernholzen about the workplace implications.
0:23:00 And I think you also testified on Capitol Hill about what AI is going to do to jobs.
0:23:04 So why don’t you talk a little bit about what you guys found in the paper?
0:23:10 Well, this actually started with Eric and I co-chairing a National Academy study on
0:23:18 automation in the workforce, which was a two-year affair with a committee of about 15 experts
0:23:24 from around the country who were economists, social scientists, labor experts, technologists.
0:23:30 And in that study, I think we learned so much.
0:23:35 It turns out when you really dig into the question of what’s going to be the impact
0:23:39 of AI and automation on jobs.
0:23:46 You can’t escape noticing that there are many different forces that automation and technology
0:23:48 is exerting on the workforce.
0:23:50 One of them, of course, is automation.
0:23:55 Toll booth operators are going away, do not sign up to be a toll booth operator.
0:24:02 But in other kinds of jobs, instead of the job going away, there will be a shift, a redistribution
0:24:05 of the tasks.
0:24:07 So take, for example, a doctor.
0:24:12 A doctor has multiple tasks, for instance, they have to diagnose the patient, they have
0:24:17 to generate some possible therapies, they have to have a heart-to-heart discussion with
0:24:25 the patient about which of those therapies the patient elects to follow, and they have
0:24:26 to bill the patient.
0:24:31 Now computers are getting, computers are pretty good at billing, but they’re getting better
0:24:35 at diagnosis and they’re getting better at suggesting therapies.
0:24:41 For example, in just in the last couple of years, we’ve seen computers that are at the
0:24:49 same level, if not a little better than doctors at things like diagnosing skin cancer and
0:24:50 other kinds of diseases.
0:24:55 The radiologists, the tissue biopsies, all of these things, we’re using these computer
0:24:59 vision techniques to get very good performance.
0:25:02 So what does this mean about the future of doctors?
0:25:07 Well, I think what it means is automation happens at the level of the individual tasks,
0:25:09 not at the job level.
0:25:16 If a job is a bundle of tasks, like diagnosis, therapy, heart-to-heart chat, what’s going
0:25:24 to happen is computers will provide future doctors with more assistance, in some degrees,
0:25:30 hopefully automating billing, but some amount of automation or advice giving.
0:25:36 But for other tasks, like having that heart-to-heart chat, we’re very, very far from when computers
0:25:40 are going to be able to do anything close to that.
0:25:44 Good bedside manner is not going to be a future of your RoboDoc anytime soon.
0:25:53 And so what you find, if you look into this, and Eric and I recently had a paper in Science
0:25:59 with a more detailed study of this, but what you find is that the majority of jobs are
0:26:04 not like toll-booth operators, where there’s just one task, and if that gets automated,
0:26:05 that’s the end of the job.
0:26:15 The majority of jobs, like podcast interviewer, or computer, or professor, or doctor, really
0:26:16 are a bundle of tasks.
0:26:23 And so what’s going to happen is that, according to our study, the majority, more than half
0:26:29 of jobs, are going to be influenced, impacted by automation, but the impact won’t be a
0:26:30 elimination.
0:26:34 It’ll be a redistribution of the time that you spend on different tasks.
0:26:43 And we even conjecture that successful businesses in the future will, to some degree, be redefining
0:26:49 what the collection of jobs is that they’re hiring for.
0:26:53 Because they still have to cover the tasks through some combination of automation and
0:27:01 manual work, but the current bundles of tasks that form jobs today might shift dramatically.
0:27:06 So the key insight is to think of a job as a bundle of tasks, and that bundle might change
0:27:13 over time as AI enters and says, well, look, this specific task I’m very good at in algorithm
0:27:16 land, and so let’s get humans to focus on other things.
0:27:19 We just need to think of them as differently bundled.
0:27:24 Well, the last topic I wanted to talk with you about, Tom, is around whether this is
0:27:27 the best time ever for AI research.
0:27:29 So we started the grand campaign.
0:27:32 Some would argue summer of 1956 with the Dartmouth conference.
0:27:35 And we’ve had several winters and summers.
0:27:36 Where are we now?
0:27:39 And then what are you most excited about looking into the future?
0:27:45 I think we’re absolutely at the best time ever for the field of artificial intelligence.
0:27:48 And there have been, as you say, ups and downs over the year.
0:27:55 And for example, in the late ’80s, AI was very hot, and there was great expectation
0:27:57 of the things it would be able to do.
0:28:02 There was also great fear, by the way, of what Japan was going to do.
0:28:03 Yeah.
0:28:08 This is the fifth generation supercomputer and the entire national policy of Japan, right?
0:28:10 Focusing on this area.
0:28:16 And so in the U.S., there was great concern that this would have a big impact, Japan would
0:28:18 take over the economy.
0:28:21 So there are some parallels here.
0:28:24 Now again, AI, it’s very popular.
0:28:27 People have great expectations.
0:28:31 And there’s a great amount of fear, I have to say, about what China and other countries
0:28:34 might be doing in AI.
0:28:44 But one really, really important difference is that, unlike in the 1980s, right now, there’s
0:28:48 a huge record of accomplishment over the last 10 years.
0:28:57 We already have AI and machine learning being used across many, many different, really economically
0:29:00 valuable tasks.
0:29:06 And therefore, I think, really, there’s very little chance that we’ll have a crash, although
0:29:11 I completely agree with my friends who say, “But isn’t AI overhyped?”
0:29:13 Absolutely, it’s overhyped.
0:29:20 But there is enough reality there to keep the field progressing and to keep commercial
0:29:25 interest and to keep economic investment going for a long time to come.
0:29:27 So you would argue, this time it really is different.
0:29:28 It really is different.
0:29:31 Because we have real working stuff to point to.
0:29:36 And over the next 10 years, we’ll have a whole lot more real working stuff that influences
0:29:38 our lives daily.
0:29:44 So as a university researcher, I look at this and I say, “Where is this going and what should
0:29:47 we be doing in the university?”
0:29:51 If you want to think about that, you have to realize just how much progress there was
0:29:52 in the last 10 years.
0:29:59 When the iPhone came out, I guess that’s 11 years ago, computers were deaf and blind.
0:30:03 When the iPhone came out, you could not talk to your iPhone.
0:30:09 This is such a weird idea, but you could not talk to your iPhone because speech recognition
0:30:11 didn’t work.
0:30:19 And now computers can transcribe voice to text just as well as people.
0:30:26 Similarly, when you pointed your camera at a scene, it couldn’t recognize with any accuracy
0:30:29 the things that were on the table in the scene.
0:30:34 And now it can do that with about the same accuracy comparable to humans.
0:30:38 And in some visual tasks, like skin cancer detection, even better than you know.
0:30:40 Even better than trained doctors, yeah.
0:30:41 Better than trained.
0:30:44 So it’s hard to remember that it’s only been 10 years.
0:30:47 And that’s the thing about progress in AI.
0:30:53 You forget because it becomes so familiar, just how dramatic the improvement has been.
0:30:55 Now think about what that means.
0:31:00 That means we’re really in the first five years of having computers that are not deaf
0:31:01 and blind.
0:31:08 And now think about what are the kinds of intelligence that you could exhibit if you
0:31:13 were deaf and blind, well, you could do game playing and inventory control.
0:31:17 You could do things that don’t involve perception.
0:31:24 But once you can perceive the world and converse in the world, there’s an explosion of new
0:31:25 applications you can do.
0:31:30 So we’re going to have garage door openers that open for you because they recognize that
0:31:32 your car coming down the driveway.
0:31:37 We’re going to have many, many things that we haven’t even thought about that just leverage
0:31:43 off this very recent progress in perceptual AI.
0:31:49 So going forward, I think a lot about how I want to invest my own research time.
0:31:51 I’m interested still in machine learning.
0:31:53 I’m very proud of the field of machine learning.
0:31:55 It’s come a long way.
0:32:01 But I’m also somebody who thinks we’re only at the beginning.
0:32:05 I think if you want to know the future of machine learning, all you need to do is look
0:32:10 at how humans learn and computers don’t yet.
0:32:16 So we learn, for example, we do learn statistically like computers do.
0:32:24 My phone watches me over time and statistically, it eventually learns where it thinks my house
0:32:27 is and where it thinks my work is.
0:32:30 It statistically learns what my preferences are.
0:32:33 But I also have a human assistant.
0:32:39 And if she tried to figure out what I wanted her to do by statistically watching me do
0:32:43 things a thousand times, I would have fired her so long ago.
0:32:46 A lot of false positives and false negatives, right?
0:32:47 Right.
0:32:48 So she doesn’t learn that way.
0:32:51 She learns by having a conversation with me.
0:32:56 I go into the office and I say, “Hey, this semester I’m team teaching a course with
0:33:00 Katarina on deep reinforcement learning.
0:33:02 Here’s what I want you to do.
0:33:04 Whenever this happens, you do this.
0:33:10 Whenever we’re preparing to hand out a homework assignment, if it hasn’t been pretested by
0:33:16 the teaching assistants two days before handout, you send a note saying, “Get that thing pretested.”
0:33:22 So what I do is I teach her and we have a conversation, she clarifies.
0:33:32 So one of the new paradigms for machine learning that I predict we will see in the coming decade
0:33:35 is what I’ll call conversational learning.
0:33:42 Use the kind of conversational interfaces that we have, say, with our phones to allow
0:33:52 people to literally teach their devices what they want them to do instead of have the device
0:33:54 statistically learn it.
0:34:01 And if you go down that road, here’s a really interesting angle on it.
0:34:07 It becomes kind of like replacing computer programming with natural language instruction.
0:34:15 So I’ll give you an example of a prototype system that we’ve been working on together
0:34:19 with Brad Myers, one of our faculty in HCI.
0:34:25 It allows you to say to your phone something like, “Whenever it snows at night, I want
0:34:27 you to wake me up 30 minutes earlier.”
0:34:33 If you live in Pittsburgh, this is a useful app, and none of the California engineers
0:34:34 have created that app.
0:34:40 And today, I could create that app if I took the trouble of learning the computer language
0:34:43 of the phone, I could program it.
0:34:49 But only far less than 1% of phone users can actually have taken the time to learn the
0:34:51 language of the computer.
0:34:56 We’re giving the phone the chance to learn the language of the person.
0:35:03 So with our phone prototype, if you say, “Whenever it snows at night, wake me up 30 minutes
0:35:08 earlier,” it says, “I don’t understand, do you want to teach me?”
0:35:12 And you can say, “Yes, here’s how you find out if it’s snowing at night.”
0:35:17 You open up this weather app right here, and where it says current conditions, if that
0:35:20 says SNOW, it’s snowing.
0:35:25 Here’s how you wake me up 30 minutes earlier, you open up that alarm app, and this number
0:35:27 you subtract 30 from it.
0:35:36 So with a combination of showing, demonstrating, and telling voice, we’re trying to give users
0:35:44 the opportunity to create their own apps, their own programs with the same kind of instruction,
0:35:49 voice, and demonstration that you would use if you were trying to teach me how to do it.
0:35:50 I love that.
0:35:53 It’s sort of a natural language for an end to, we have an investment in a company called
0:35:57 ift, if this, then that, which is you can program those things, but you have to be a
0:35:59 little sophisticated.
0:36:03 You’d like to just be able to talk to your phone and have it figured out how do I fill
0:36:05 the slots into ift that ift wants.
0:36:07 Exactly, and if this, then that is a wonderful thing.
0:36:14 It has a huge library of these apps that you can download, but as you say, you still have
0:36:17 to learn the language of the computer to create those.
0:36:21 We’re trying to have the computer learn the language of the person.
0:36:27 If that line of research plays out, and I believe it will this decade, we’ll be in a very different
0:36:34 world because we’ll be in a world where instead of the elite few, less than 1% of phone users
0:36:39 being able to program, it’ll be 99% of phone users who can do this.
0:36:45 Now, think about what that does for the whole conception of how we think about human-computer
0:36:46 interaction.
0:36:47 Yeah.
0:36:49 That’s a profound shift in society, right?
0:36:55 Just like everybody became literate, not just the priests, and look what happened to society.
0:36:56 Exactly.
0:36:57 Yeah.
0:36:59 Think about what it means for the future of jobs.
0:37:04 Right now, if you have a computer introduced as your teammate, you, the human, and the
0:37:11 computer are a team, well, the computer is frozen, and the teammate who gets to do the
0:37:16 adapting is the human because the computer is a fixed functionality.
0:37:21 What if in that team, the human could just teach the computer how they want the computer
0:37:22 to help them do their job.
0:37:25 It would be a completely different dynamic.
0:37:27 It would change the future of work.
0:37:28 Yeah.
0:37:29 That’s fascinating.
0:37:33 And then I think another thread that you’re super interested in on the future of machine
0:37:37 learning is something around never-ending learning, so tell us about that.
0:37:38 Sure.
0:37:44 Again, I just go back to what do humans do that computers don’t yet, and computers are
0:37:49 very good at, say, learning to diagnose skin cancer.
0:37:54 You give it some very specific tasks and some data, but if you look at people, people learn
0:37:57 so many things.
0:37:59 We learn to all kinds of things.
0:38:00 You can tap the hands.
0:38:02 You can do double entry bookkeeping.
0:38:03 Right?
0:38:04 Right.
0:38:05 You can add numbers.
0:38:11 You can play music, all kinds of things, and a lot of those things we learn over time
0:38:17 in a kind of synergistic way, in a staged sequence.
0:38:21 First you learn to crawl, then you learn to walk, then you learn to run, then you learn
0:38:28 to ride a bike, and it wouldn’t make any sense to do them in the other sequence, because
0:38:31 you’re actually learning to learn.
0:38:37 When you acquire one skill, it puts you in a position that you now are capable of learning
0:38:38 the next skill.
0:38:44 So I’m very interested in what it would mean to give a computer that kind of capability
0:38:51 to do learning for days and weeks and years and decades.
0:38:59 We have a project we call our Never-Ending Language Learner, which started in 2010, running
0:39:03 24 hours a day trying to learn to read the web.
0:39:04 Fascinating.
0:39:07 And there’s something about sort of the longitudinal, right?
0:39:09 We started it in 2010 and it just keeps on going.
0:39:13 So it’s not just like transfer learning from one model to another.
0:39:15 It’s like long running.
0:39:20 It’s long running and it has many different learning tasks.
0:39:25 It’s building up a knowledge base of knowledge about the world.
0:39:33 But to keep it short, I’ll just say we’ve learned so much from that project about how
0:39:39 to organize the architecture of a system so that it can invent new learning tasks as it
0:39:46 goes so that it can get synergy once it learns one thing to become better at learning another
0:39:47 thing.
0:39:53 How, in fact, very importantly, it can use unlabeled data to train itself instead of
0:39:57 requiring an army of data labellers.
0:40:04 So I just think this is an area that’s relatively untouched in the machine learning field.
0:40:11 But looking forward, we’re already seeing an increasing number of embedded machine learning
0:40:14 systems in continuous use.
0:40:22 And as we see more and more of those in the internet of things and elsewhere, the opportunity
0:40:29 for learning continuously for days and weeks and months and years and decades is increasingly
0:40:30 there.
0:40:36 We ought to be developing the ideas, the concepts of how to organize those systems to take advantage
0:40:37 of that.
0:40:38 Yeah.
0:40:41 I love both of these design approaches in that they’re sort of inspired by humans and
0:40:50 sort of humans are mysteriously good at learning and adapting and they sort of shine a spotlight
0:40:53 on where machine learning algorithms are not yet.
0:40:56 So it’s such a fertile area to look for inspiration.
0:41:00 Well, Tom, it’s been a great delight having you on the podcast.
0:41:03 Thanks for sharing about the history and the future of machine learning.
0:41:06 We can tell you’re still fired up after all of these decades.
0:41:11 And so that’s a great delight just to see somebody who is committed, basically, their
0:41:17 life to understanding the mysteries of learning and wishing many good decades to come as you
0:41:19 continue working on it.
0:41:20 Thanks.
0:41:21 Thanks for doing this podcast.
0:41:25 I think it’s a great thing to get a conversation going and it’s a great contribution to do
0:41:25 that.
0:00:10 tax or investment advice, or be used to evaluate any investment or security and is not directed
0:00:14 at any investors or potential investors in any A16Z fund.
0:00:17 For more details, please see a16z.com/disclosures.
0:00:21 Hi, and welcome to the A16Z podcast.
0:00:22 I’m Frank Chen.
0:00:28 Today, I’m here with Carnegie Mellon’s Professor Tom Mitchell, who has been involved with machine
0:00:30 learning basically his entire career.
0:00:34 So I’m super excited to have this conversation with Tom, where he can tell us a little bit
0:00:37 about the history and where all of our techniques came from.
0:00:40 And we’ll spend time talking about the future, where the field is going.
0:00:45 So Carnegie Mellon’s been involved in sort of standing up the fundamental teaching institutions
0:00:50 and research institutions of the big areas, computer science, artificial intelligence
0:00:52 and machine learning.
0:00:56 So take us back to the early days, the you and Newell and Jeff Hinton are teaching this
0:00:57 class.
0:00:58 What was your curriculum like?
0:00:59 Like what were you teaching?
0:01:02 Pretty different, I imagine, than what we teach undergrads today.
0:01:03 That’s right.
0:01:05 Well, at the time, this was the 1980s.
0:01:13 So artificial intelligence at that point was dominated by what we would call symbolic methods.
0:01:19 Where things like formal logic would be used to do inference.
0:01:24 And much of machine learning was really about learning symbolic structures, symbolic representations
0:01:25 of knowledge.
0:01:31 But there was this kind of young whipper snapper, Jeff Hinton, who had a different idea.
0:01:41 And so he was working on a book with Rommelhart McClellan that became a very well-known parallel
0:01:47 data processing book that kind of launched the field of neural nets.
0:01:49 And they were, if I remember, psychologists, right?
0:01:58 Yeah. Jay McClellan is a psychologist here at CMU, Rommelhart, kind of a neuroscientist.
0:02:01 And more, he was a very broad person.
0:02:07 And Jeff, so the three of them were kind of the rebels who were taking things off in a
0:02:09 different paradigm.
0:02:14 The empire wanted us to do research on knowledge representation and inference and first-order
0:02:15 logic.
0:02:20 I remember as an undergrad, I took this computer aided class that John H.M.M.D. wrote called
0:02:24 Tarski’s World, where we learned all about first-world logic.
0:02:25 What could you prove?
0:02:26 What could you not prove?
0:02:30 And so that’s what the establishment, quote, unquote, was teaching.
0:02:37 And then Jeff was the rebel off in neural network land, and he gets his reprise later.
0:02:40 So take us back to the world of knowledge representation, because I’m actually seeing
0:02:44 a lot of startups these days who are trying to bring back some of these techniques to
0:02:49 complement deep learning, because there are well-known challenges with deep learning.
0:02:52 Like, we’re not encoding any priors, we’re learning everything for the first time, we
0:02:56 need tons of labeled data sets to make progress.
0:03:00 And so take us back to the days of knowledge representation.
0:03:04 What were we trying to solve with those set of techniques, and how might we use them today?
0:03:13 So back in the ’80s and the ’90s, and I have to say that some of the really senior people
0:03:20 in the field were totally devoted to this paradigm of logical inference, logical representations.
0:03:27 People like John McCarthy, for example, were very strong proponents of this, and really
0:03:34 essentially just saw that reasoning is theorem-proving, and therefore if we’re going to get computers
0:03:38 to do it, that’s what we have to do.
0:03:42 There were some problems with that, and there still are.
0:03:48 One thing that I remember from back then that was an example was the banana in the tailpipe
0:03:50 problem.
0:03:55 These logical systems were used to reason to do things like, how would you plan a sequence
0:03:57 of actions to achieve a goal?
0:03:59 Like, how would you get from here to the airport?
0:04:05 Well, you’d walk to your car, you’d put the key in, turn the car on, you’d drive out of
0:04:10 the parking lot, get on the interstate, go to the airport exit, et cetera.
0:04:12 But what if there’s a banana in the tailpipe?
0:04:17 Even back then, before it became a meme in Beverly Hills Cop, we were worried about
0:04:19 the banana in the tailpipe.
0:04:20 That’s right.
0:04:25 And the point of the banana in the tailpipe is there are an infinite number of other things
0:04:31 that you don’t say when you spin out a plan like that.
0:04:39 And any proof, if it’s a proof, really, is going to have to cover all those conditions,
0:04:43 and that’s kind of an infinitely intractable problem.
0:04:48 You couldn’t encode enough to do the inference you needed for your plans to be successful.
0:04:49 Right.
0:04:56 And so one of the big changes between the ’80s and 2019 is that we no longer really think
0:05:03 in the field of AI that inference is proving things, instead it’s building a plausible
0:05:05 chain of argument.
0:05:10 And it might be wrong, and if it goes wrong, if there is a banana in the tailpipe, you’ll
0:05:12 deal with it when it happens and when you figure it out.
0:05:13 Right.
0:05:18 So we move from certainty and proof to sort of probabilistic reasoning, right, Bayesian
0:05:20 techniques started becoming popular.
0:05:21 Right.
0:05:25 And so around 19– in the late ’90s, in fact.
0:05:31 So if you look at the history of machine learning, there’s an interesting trajectory where in
0:05:38 maybe up to the mid ’80s, things were pretty much focused on symbolic representations.
0:05:43 Actually, if you go back to the ’60s, there were the perceptron, but then it got swallowed
0:05:50 up by the end of the ’60s by symbolic representations and trying to reason that way and trying to
0:05:52 learn those kind of symbolic structures.
0:05:59 Then when the neural net wave came in around the late ’80s, early ’90s, that started competing
0:06:02 with the idea of symbolic representations.
0:06:09 But then in the late ’90s, the statisticians moved in and probabilistic methods became
0:06:11 very popular.
0:06:16 And at the time, there was this– if you look at this history, you can’t help but realize
0:06:22 what a social phenomenon technology advances in sciences and technology.
0:06:24 People influencing each other at conferences.
0:06:25 Right.
0:06:29 They were shaming them into adopting a new paradigm.
0:06:36 And so one of the slogans, or one of the phrases you kept hearing when people started working
0:06:41 on probabilistic, statistical probabilistic methods, they would never call them that.
0:06:48 They would have called them, instead, principled probabilistic methods, just to kind of shine
0:06:53 the light on the distinction between neural nets, which are just somehow tuning a gazillion
0:06:57 parameters in the principled methods that were being used.
0:07:03 And so that became really the dominant paradigm in the late ’90s and kind of remained in charge
0:07:12 of the field up through till about 2009, 2010, when now, as everybody kind of knows, deep
0:07:18 networks made a very serious revolution showing that they could do all kinds of amazing things
0:07:20 that hadn’t been done before.
0:07:21 Yeah.
0:07:26 We really are living in a golden age here in deep learning in neural network land.
0:07:29 But let’s go back to the original sort of rebel group.
0:07:35 This is Jeff Hinton hanging out in the shadow of sort of first order logic and saying, “No,
0:07:36 this is going to work.”
0:07:40 I think they were loosely inspired by the architecture of the brain.
0:07:41 Is that–
0:07:42 Definitely.
0:07:43 Definitely.
0:07:49 The kinds of arguments Jerry Feldman was one of the people who gave some of these arguments.
0:07:55 She said, “Look, you recognize your mother in about 100 milliseconds.
0:08:03 Your neurons can’t switch state in faster than a few milliseconds.
0:08:10 And so it looks like at most the chain of inference that you’re doing to go from your
0:08:16 retina to recognize your mother can only be about 10 deep, just from the timing.”
0:08:17 Oh, fascinating.
0:08:21 So it was an argument of sort of how long it took to recognize your mother.
0:08:22 Right.
0:08:23 And then how slow your neurons are, right?
0:08:26 Because they’re basically, these are biochemical processes, right?
0:08:27 Right.
0:08:28 Fascinating.
0:08:31 So really a computational efficiency argument.
0:08:37 And therefore, Jerry would say, “There must be a lot of stuff happening in parallel.
0:08:41 It must be a very wide chain of inference if it’s only 10 layers deep.”
0:08:45 And then he says, “Look at the brain, look at visual cortex.”
0:08:46 Yeah.
0:08:47 You got it.
0:08:51 And so neuroscientists at this time were making progress in understanding the structure of
0:08:55 neurons and how they connected to each other and how they formed connections, and those
0:08:58 connections could change strength over time, right?
0:09:02 All mediated by chemical interactions in the computer science community was inspired by
0:09:03 this.
0:09:04 Definitely.
0:09:12 And the level of abstraction at which the computational neural nets met up with the
0:09:19 real biological neural nets was not a very detailed level, but where they kind of became
0:09:28 the same was this idea of distributed representations, that in fact it might be a collection of hundreds
0:09:35 or thousands or millions of neurons that simultaneously were firing that represent your mother instead
0:09:37 of a symbol.
0:09:38 Right.
0:09:39 Right.
0:09:46 It’s a completely different notion of what it even means to represent knowledge.
0:09:51 And really, one of the most exciting things that has come out of the last decade of research
0:10:00 in neural and deep networks is a better understanding, although we still don’t fully understand of
0:10:07 how these artificial neural networks can learn very, very useful representations.
0:10:14 And for me, a simple example of that, that in a sentence summarizes it, is we have neural
0:10:22 networks now that can take as input an image of photograph and output a text caption for
0:10:23 that photograph.
0:10:28 What kind of representation must be in the middle of that neural network in order to
0:10:35 actually capture the meaning well enough that you can go from a visual stimulus to the equivalent
0:10:43 textual content, that it’s really, it must be capturing a very basic core representation
0:10:45 of the meaning of that photograph.
0:10:46 Yeah.
0:10:50 And one of my favorite things about the brain, which is otherwise this very sort of slow
0:10:54 computer, right, if you just look at neuron speeds, is that not only can they do this,
0:10:59 but they can actually use this, the representation they’re deriving to actually inform our actions
0:11:01 and our plans and our goals, right?
0:11:05 So not only is it like this picture has a chair in it, but like I can sit in that chair.
0:11:06 I can simulate sitting in that chair.
0:11:10 I think like that chair is going to support my weight, and all of these things happen
0:11:14 in like milliseconds, despite the fact that the basic components of the brain are very
0:11:15 slow.
0:11:16 Yeah.
0:11:17 It’s an amazing thing.
0:11:21 In fact, now that you mentioned it, I have to tell you, a half of my research life these
0:11:29 days is in studying how the human brain represents meaning of language.
0:11:32 We use brain imaging methods to do this.
0:11:39 And in one set of studies, we put people in a fMRI scanner, and we showed them just common
0:11:49 nouns like automobile, airplane, a knife, a chair, and so forth.
0:11:56 And we would get a picture literally with about three millimeter resolution of the three-dimensional
0:12:02 neural activity in their brain as they think about these different words.
0:12:06 And we’re interested in the question of all kinds of fundamental questions like what do
0:12:08 these representations look like?
0:12:12 Are they the same in your brain and my brain?
0:12:17 Given that they don’t appear instantaneously, by the way, it takes you about 400 milliseconds
0:12:22 to understand a word, if I put it on the screen in front of you.
0:12:24 What happens during that 400 milliseconds?
0:12:30 How do these representations evolve and come to be?
0:12:36 And one of the most interesting things we found, we studied this question by training
0:12:46 a machine learning system to take as input an arbitrary noun and to predict the brain
0:12:49 image that we will see if a person reads that noun.
0:12:56 Now, we only had data for 60 nouns at that time, so we didn’t train it on every noun
0:12:57 in the world.
0:13:00 We trained it on 60.
0:13:05 In fact, what we did was we trained it only on 58, so we could hold out two nouns that
0:13:06 we hadn’t seen.
0:13:12 And then we would test how well it could extrapolate to new nouns that you had never seen by showing
0:13:18 it the two held out nouns and having it predict the images.
0:13:21 Then we’d show it two images and we’d say, “Well, which of those is strawberry and which
0:13:23 of those is airplane?”
0:13:26 And it was right 80% of the time.
0:13:29 So, you could actually predict essentially brain state, right?
0:13:31 I’m going to show you a strawberry.
0:13:35 Let me predict the configuration of your neurons and who’s lighting up and who’s not.
0:13:42 So then we had a model that we trained with machine learning that captured something about
0:13:44 representations in the brain.
0:13:50 We used that to discover that the representations are almost identical in your brain and mind.
0:13:56 We could train on one set of people and decode what other people were thinking about.
0:14:02 And we also found that the representations themselves are grounded in parts of the brain
0:14:05 that are associated with perception.
0:14:10 So, if I give you a word like “peach,” the parts of your brain that code the meaning
0:14:16 of that are the ones associated with the sense of taste and manipulation because sometimes
0:14:19 you pick up a peach in visual color.
0:14:21 Yeah, that is fascinating.
0:14:25 Well, it’s so exciting to think that the brain structures are identical across people
0:14:30 because what everybody wants is sort of that, remember that scene in the matrix where you
0:14:34 sort of like you’re jacked straight into your brain and you’re like, “Oh, now I know Kung-Fu.”
0:14:35 Right?
0:14:36 Like this is what we want, right?
0:14:42 We want to learn new skills and sort of new facts and new inferences just like loading
0:14:43 an SD card, right?
0:14:48 And so, the fact that we’re sort of converging to the same structures in the brain at least
0:14:50 makes that theoretically possible.
0:14:51 We’re a ways away from that.
0:14:52 We’re a ways away from that.
0:14:53 But I’m with you.
0:14:54 Yeah, awesome.
0:15:01 So, another area that interests you is finding biases and why don’t we start by distinguishing
0:15:06 sort of two types of biases because when you hear the word “bias” today in machine learning,
0:15:10 you’re mostly thinking about things like, “Gee, let me make sure my data set is representative
0:15:13 so I don’t draw the wrong conclusion from that,” right?
0:15:18 So the classic example being here that I don’t do good recognition on people with darker skin
0:15:23 because I didn’t have enough of those samples in my data set and so the bias here is you’ve
0:15:29 selected a very small subset of the target data set that you want to cover and make predictions
0:15:31 on and therefore your predictions are poor, right?
0:15:33 So, that’s one sense of bias.
0:15:37 But there’s another sense of bias, that statistical bias, which is kind of what you want out of
0:15:38 algorithms.
0:15:39 So, maybe talk about this notion.
0:15:40 Yeah, sure.
0:15:45 And this is really a very important issue right now because now that machine learning
0:15:52 is being used in practice in many different ways, the issue of bias really is very important
0:15:54 to deal with.
0:16:00 You gave an example, another example would be, for instance, you have some historical
0:16:06 loan applications in which ones were approved, but maybe there’s some bias that say people
0:16:12 of one gender receive fewer loan approvals just because of their gender and if that’s
0:16:18 inherent in the data and you train a machine learning system that’s successful, well, it’s
0:16:22 probably going to learn the patterns that are in that data.
0:16:29 So, the notion of what I’ll call social bias, socially unacceptable bias, is really this
0:16:36 idea that you want the data set to reflect the kind of decision making that you want
0:16:39 the program to make if you’re going to train the program.
0:16:44 And that’s kind of the common sense notion of bias that most people talk about.
0:16:50 But there’s a lot of confusion in the field right now because bias is also used in statistical
0:16:55 machine learning to really with a very different meaning.
0:17:02 We’ll say that an algorithm is unbiased if the patterns that it learns, the decision
0:17:10 rules that it learns for approving loans, for example, reflect correctly the patterns that
0:17:12 are in the data.
0:17:18 Also that notion of statistically unbiased just means the algorithm’s doing its job of
0:17:21 recapitulating the decisions that are in the data.
0:17:28 The notion of the data itself being biased is really an orthogonal notion.
0:17:32 And there’s some interesting research going on now.
0:17:39 So for example, typically when we train a machine learning system, say to do loan approval,
0:17:45 a typical thing would be you can think of these machine learning algorithms as optimization
0:17:46 algorithms.
0:17:54 They’re going to tune maybe the parameters of your deep network so that they maximize
0:18:00 the number of decisions that they make that agree with the training examples.
0:18:06 But if your training examples have this kind of bias that maybe females receive fewer loan
0:18:09 approvals than males.
0:18:13 There’s some new work where people say, well, let’s change that objective that we’re trying
0:18:22 to optimize in addition to fitting the decisions that are in the training data as well as possible.
0:18:29 Let’s put another constraint that the probability of a female being approved for a loan has
0:18:33 to be equal to the probability of a male being approved.
0:18:38 And then subject to that constraint, we’ll try to match as many decisions as possible.
0:18:44 So there’s a lot of work right now in really technical work trying to understand if there
0:18:51 are ways of thinking more creatively, more imaginatively about how to even frame the
0:18:58 machine learning problem so that we can take what might be biased datasets but impose constraints
0:19:01 on the decision rules that we want to learn from those.
0:19:03 Yeah, that’s super interesting.
0:19:08 We’re sort of envisioning the world we want rather than the data of the world that we
0:19:13 came from because we might not be happy with the representation of the representedness
0:19:16 I guess, of the data that we came from.
0:19:20 And it’s causing people to look a lot more carefully at even the very notion of what
0:19:25 it means to be biased and what it means to be fair.
0:19:29 Are there good measures for fairness that the community is driving towards or do we
0:19:34 not really have a sort of an objective measure of fairness?
0:19:35 We don’t.
0:19:41 We don’t have an objective measure and there’s a lot of activity right now to discussing
0:19:48 that, including people like our philosophy professor, David Danks, who is very much part
0:19:54 of this discussion and social scientists, technology people, all getting together.
0:20:01 And in fact, there are now a couple of conferences centered around how to introduce fairness and
0:20:07 explainability and trust in AI systems.
0:20:11 This is a very important issue, but it’s not only technical.
0:20:16 It’s partly getting our philosophical, social, trying to get our heads around what it is
0:20:17 that we really want.
0:20:21 That’s a beautiful thing about AI and about computers in general.
0:20:27 It forces you to be way more precise when you are getting a computer to do it about
0:20:29 what you want.
0:20:36 And so even if you just think about self-driving cars, we have, when I was 16, I took a test
0:20:39 and I was approved to be a human driver.
0:20:43 They never asked me questions about whether I would swerve to hit the old lady or swerve
0:20:44 to hit the baby carriage.
0:20:45 Right.
0:20:47 Charlie’s problem was not on the DMV test.
0:20:48 Exactly.
0:20:50 But it’s on the test for the computers.
0:20:51 Right.
0:20:53 Yeah, it’s really interesting that we sort of hold computers to a different standard
0:20:55 because we’re programming them, right?
0:20:59 We can be explicit and we can have them sort of hit goals or not, right?
0:21:04 And those are design decisions rather than sort of, you know, bundled into a brain, right,
0:21:05 of a person.
0:21:06 Yeah.
0:21:12 And so I think of, you know, look, banks historically have hired loan officers.
0:21:16 Those loan officers may or may not be fair, right, according to the definitions that we’re
0:21:21 sort of talking about now, but we kind of hold those humans, those human loan officers to
0:21:24 a different standard than we would hold the algorithms.
0:21:25 That’s true.
0:21:27 I mean, who knows which way it will go in the future.
0:21:33 If we continue to have human loan officers and some computer loan officers, will we
0:21:39 up the constraints on the humans so that they pass the same qualifications?
0:21:45 Or will we drop the constraints on the computers so that they’re no more titrated than the
0:21:46 people?
0:21:47 Yeah.
0:21:48 That’s fascinating, right?
0:21:49 Who is master?
0:21:50 Who is the student, right?
0:21:55 The intuitive thing is let’s make the humans the models that we train our systems to approach,
0:21:56 right?
0:21:58 Like human competence being the goal.
0:22:00 The other way to think about it is no, right?
0:22:04 We can actually introduce constraints like, you know, equal number of men and women or
0:22:07 equal number of this ethnicity versus another ethnicity.
0:22:12 And our algorithms as a result of those constraints could be more fair than humans.
0:22:14 And so we invert it, right?
0:22:16 Let’s get humans up to that level of impartiality.
0:22:17 Right.
0:22:21 Like maybe the algorithm can end up teaching the human how to make those decisions in a
0:22:26 different way so that the fairness outcome you want is really achieved.
0:22:27 Yeah.
0:22:28 That’s fascinating.
0:22:32 And it’s great that sort of not just computer scientists are involved in this conversation
0:22:36 about the ethicists and the social scientists who are weighing in.
0:22:40 So that gives me hope that, you know, sort of smart people across disciplines are really
0:22:41 grappling with this.
0:22:43 So we get the outcomes that we want.
0:22:49 Well, sort of a related topic to this, right, sort of social impact of AI.
0:22:56 You recently co-wrote a paper with MIT’s Eric Bernholzen about the workplace implications.
0:23:00 And I think you also testified on Capitol Hill about what AI is going to do to jobs.
0:23:04 So why don’t you talk a little bit about what you guys found in the paper?
0:23:10 Well, this actually started with Eric and I co-chairing a National Academy study on
0:23:18 automation in the workforce, which was a two-year affair with a committee of about 15 experts
0:23:24 from around the country who were economists, social scientists, labor experts, technologists.
0:23:30 And in that study, I think we learned so much.
0:23:35 It turns out when you really dig into the question of what’s going to be the impact
0:23:39 of AI and automation on jobs.
0:23:46 You can’t escape noticing that there are many different forces that automation and technology
0:23:48 is exerting on the workforce.
0:23:50 One of them, of course, is automation.
0:23:55 Toll booth operators are going away, do not sign up to be a toll booth operator.
0:24:02 But in other kinds of jobs, instead of the job going away, there will be a shift, a redistribution
0:24:05 of the tasks.
0:24:07 So take, for example, a doctor.
0:24:12 A doctor has multiple tasks, for instance, they have to diagnose the patient, they have
0:24:17 to generate some possible therapies, they have to have a heart-to-heart discussion with
0:24:25 the patient about which of those therapies the patient elects to follow, and they have
0:24:26 to bill the patient.
0:24:31 Now computers are getting, computers are pretty good at billing, but they’re getting better
0:24:35 at diagnosis and they’re getting better at suggesting therapies.
0:24:41 For example, in just in the last couple of years, we’ve seen computers that are at the
0:24:49 same level, if not a little better than doctors at things like diagnosing skin cancer and
0:24:50 other kinds of diseases.
0:24:55 The radiologists, the tissue biopsies, all of these things, we’re using these computer
0:24:59 vision techniques to get very good performance.
0:25:02 So what does this mean about the future of doctors?
0:25:07 Well, I think what it means is automation happens at the level of the individual tasks,
0:25:09 not at the job level.
0:25:16 If a job is a bundle of tasks, like diagnosis, therapy, heart-to-heart chat, what’s going
0:25:24 to happen is computers will provide future doctors with more assistance, in some degrees,
0:25:30 hopefully automating billing, but some amount of automation or advice giving.
0:25:36 But for other tasks, like having that heart-to-heart chat, we’re very, very far from when computers
0:25:40 are going to be able to do anything close to that.
0:25:44 Good bedside manner is not going to be a future of your RoboDoc anytime soon.
0:25:53 And so what you find, if you look into this, and Eric and I recently had a paper in Science
0:25:59 with a more detailed study of this, but what you find is that the majority of jobs are
0:26:04 not like toll-booth operators, where there’s just one task, and if that gets automated,
0:26:05 that’s the end of the job.
0:26:15 The majority of jobs, like podcast interviewer, or computer, or professor, or doctor, really
0:26:16 are a bundle of tasks.
0:26:23 And so what’s going to happen is that, according to our study, the majority, more than half
0:26:29 of jobs, are going to be influenced, impacted by automation, but the impact won’t be a
0:26:30 elimination.
0:26:34 It’ll be a redistribution of the time that you spend on different tasks.
0:26:43 And we even conjecture that successful businesses in the future will, to some degree, be redefining
0:26:49 what the collection of jobs is that they’re hiring for.
0:26:53 Because they still have to cover the tasks through some combination of automation and
0:27:01 manual work, but the current bundles of tasks that form jobs today might shift dramatically.
0:27:06 So the key insight is to think of a job as a bundle of tasks, and that bundle might change
0:27:13 over time as AI enters and says, well, look, this specific task I’m very good at in algorithm
0:27:16 land, and so let’s get humans to focus on other things.
0:27:19 We just need to think of them as differently bundled.
0:27:24 Well, the last topic I wanted to talk with you about, Tom, is around whether this is
0:27:27 the best time ever for AI research.
0:27:29 So we started the grand campaign.
0:27:32 Some would argue summer of 1956 with the Dartmouth conference.
0:27:35 And we’ve had several winters and summers.
0:27:36 Where are we now?
0:27:39 And then what are you most excited about looking into the future?
0:27:45 I think we’re absolutely at the best time ever for the field of artificial intelligence.
0:27:48 And there have been, as you say, ups and downs over the year.
0:27:55 And for example, in the late ’80s, AI was very hot, and there was great expectation
0:27:57 of the things it would be able to do.
0:28:02 There was also great fear, by the way, of what Japan was going to do.
0:28:03 Yeah.
0:28:08 This is the fifth generation supercomputer and the entire national policy of Japan, right?
0:28:10 Focusing on this area.
0:28:16 And so in the U.S., there was great concern that this would have a big impact, Japan would
0:28:18 take over the economy.
0:28:21 So there are some parallels here.
0:28:24 Now again, AI, it’s very popular.
0:28:27 People have great expectations.
0:28:31 And there’s a great amount of fear, I have to say, about what China and other countries
0:28:34 might be doing in AI.
0:28:44 But one really, really important difference is that, unlike in the 1980s, right now, there’s
0:28:48 a huge record of accomplishment over the last 10 years.
0:28:57 We already have AI and machine learning being used across many, many different, really economically
0:29:00 valuable tasks.
0:29:06 And therefore, I think, really, there’s very little chance that we’ll have a crash, although
0:29:11 I completely agree with my friends who say, “But isn’t AI overhyped?”
0:29:13 Absolutely, it’s overhyped.
0:29:20 But there is enough reality there to keep the field progressing and to keep commercial
0:29:25 interest and to keep economic investment going for a long time to come.
0:29:27 So you would argue, this time it really is different.
0:29:28 It really is different.
0:29:31 Because we have real working stuff to point to.
0:29:36 And over the next 10 years, we’ll have a whole lot more real working stuff that influences
0:29:38 our lives daily.
0:29:44 So as a university researcher, I look at this and I say, “Where is this going and what should
0:29:47 we be doing in the university?”
0:29:51 If you want to think about that, you have to realize just how much progress there was
0:29:52 in the last 10 years.
0:29:59 When the iPhone came out, I guess that’s 11 years ago, computers were deaf and blind.
0:30:03 When the iPhone came out, you could not talk to your iPhone.
0:30:09 This is such a weird idea, but you could not talk to your iPhone because speech recognition
0:30:11 didn’t work.
0:30:19 And now computers can transcribe voice to text just as well as people.
0:30:26 Similarly, when you pointed your camera at a scene, it couldn’t recognize with any accuracy
0:30:29 the things that were on the table in the scene.
0:30:34 And now it can do that with about the same accuracy comparable to humans.
0:30:38 And in some visual tasks, like skin cancer detection, even better than you know.
0:30:40 Even better than trained doctors, yeah.
0:30:41 Better than trained.
0:30:44 So it’s hard to remember that it’s only been 10 years.
0:30:47 And that’s the thing about progress in AI.
0:30:53 You forget because it becomes so familiar, just how dramatic the improvement has been.
0:30:55 Now think about what that means.
0:31:00 That means we’re really in the first five years of having computers that are not deaf
0:31:01 and blind.
0:31:08 And now think about what are the kinds of intelligence that you could exhibit if you
0:31:13 were deaf and blind, well, you could do game playing and inventory control.
0:31:17 You could do things that don’t involve perception.
0:31:24 But once you can perceive the world and converse in the world, there’s an explosion of new
0:31:25 applications you can do.
0:31:30 So we’re going to have garage door openers that open for you because they recognize that
0:31:32 your car coming down the driveway.
0:31:37 We’re going to have many, many things that we haven’t even thought about that just leverage
0:31:43 off this very recent progress in perceptual AI.
0:31:49 So going forward, I think a lot about how I want to invest my own research time.
0:31:51 I’m interested still in machine learning.
0:31:53 I’m very proud of the field of machine learning.
0:31:55 It’s come a long way.
0:32:01 But I’m also somebody who thinks we’re only at the beginning.
0:32:05 I think if you want to know the future of machine learning, all you need to do is look
0:32:10 at how humans learn and computers don’t yet.
0:32:16 So we learn, for example, we do learn statistically like computers do.
0:32:24 My phone watches me over time and statistically, it eventually learns where it thinks my house
0:32:27 is and where it thinks my work is.
0:32:30 It statistically learns what my preferences are.
0:32:33 But I also have a human assistant.
0:32:39 And if she tried to figure out what I wanted her to do by statistically watching me do
0:32:43 things a thousand times, I would have fired her so long ago.
0:32:46 A lot of false positives and false negatives, right?
0:32:47 Right.
0:32:48 So she doesn’t learn that way.
0:32:51 She learns by having a conversation with me.
0:32:56 I go into the office and I say, “Hey, this semester I’m team teaching a course with
0:33:00 Katarina on deep reinforcement learning.
0:33:02 Here’s what I want you to do.
0:33:04 Whenever this happens, you do this.
0:33:10 Whenever we’re preparing to hand out a homework assignment, if it hasn’t been pretested by
0:33:16 the teaching assistants two days before handout, you send a note saying, “Get that thing pretested.”
0:33:22 So what I do is I teach her and we have a conversation, she clarifies.
0:33:32 So one of the new paradigms for machine learning that I predict we will see in the coming decade
0:33:35 is what I’ll call conversational learning.
0:33:42 Use the kind of conversational interfaces that we have, say, with our phones to allow
0:33:52 people to literally teach their devices what they want them to do instead of have the device
0:33:54 statistically learn it.
0:34:01 And if you go down that road, here’s a really interesting angle on it.
0:34:07 It becomes kind of like replacing computer programming with natural language instruction.
0:34:15 So I’ll give you an example of a prototype system that we’ve been working on together
0:34:19 with Brad Myers, one of our faculty in HCI.
0:34:25 It allows you to say to your phone something like, “Whenever it snows at night, I want
0:34:27 you to wake me up 30 minutes earlier.”
0:34:33 If you live in Pittsburgh, this is a useful app, and none of the California engineers
0:34:34 have created that app.
0:34:40 And today, I could create that app if I took the trouble of learning the computer language
0:34:43 of the phone, I could program it.
0:34:49 But only far less than 1% of phone users can actually have taken the time to learn the
0:34:51 language of the computer.
0:34:56 We’re giving the phone the chance to learn the language of the person.
0:35:03 So with our phone prototype, if you say, “Whenever it snows at night, wake me up 30 minutes
0:35:08 earlier,” it says, “I don’t understand, do you want to teach me?”
0:35:12 And you can say, “Yes, here’s how you find out if it’s snowing at night.”
0:35:17 You open up this weather app right here, and where it says current conditions, if that
0:35:20 says SNOW, it’s snowing.
0:35:25 Here’s how you wake me up 30 minutes earlier, you open up that alarm app, and this number
0:35:27 you subtract 30 from it.
0:35:36 So with a combination of showing, demonstrating, and telling voice, we’re trying to give users
0:35:44 the opportunity to create their own apps, their own programs with the same kind of instruction,
0:35:49 voice, and demonstration that you would use if you were trying to teach me how to do it.
0:35:50 I love that.
0:35:53 It’s sort of a natural language for an end to, we have an investment in a company called
0:35:57 ift, if this, then that, which is you can program those things, but you have to be a
0:35:59 little sophisticated.
0:36:03 You’d like to just be able to talk to your phone and have it figured out how do I fill
0:36:05 the slots into ift that ift wants.
0:36:07 Exactly, and if this, then that is a wonderful thing.
0:36:14 It has a huge library of these apps that you can download, but as you say, you still have
0:36:17 to learn the language of the computer to create those.
0:36:21 We’re trying to have the computer learn the language of the person.
0:36:27 If that line of research plays out, and I believe it will this decade, we’ll be in a very different
0:36:34 world because we’ll be in a world where instead of the elite few, less than 1% of phone users
0:36:39 being able to program, it’ll be 99% of phone users who can do this.
0:36:45 Now, think about what that does for the whole conception of how we think about human-computer
0:36:46 interaction.
0:36:47 Yeah.
0:36:49 That’s a profound shift in society, right?
0:36:55 Just like everybody became literate, not just the priests, and look what happened to society.
0:36:56 Exactly.
0:36:57 Yeah.
0:36:59 Think about what it means for the future of jobs.
0:37:04 Right now, if you have a computer introduced as your teammate, you, the human, and the
0:37:11 computer are a team, well, the computer is frozen, and the teammate who gets to do the
0:37:16 adapting is the human because the computer is a fixed functionality.
0:37:21 What if in that team, the human could just teach the computer how they want the computer
0:37:22 to help them do their job.
0:37:25 It would be a completely different dynamic.
0:37:27 It would change the future of work.
0:37:28 Yeah.
0:37:29 That’s fascinating.
0:37:33 And then I think another thread that you’re super interested in on the future of machine
0:37:37 learning is something around never-ending learning, so tell us about that.
0:37:38 Sure.
0:37:44 Again, I just go back to what do humans do that computers don’t yet, and computers are
0:37:49 very good at, say, learning to diagnose skin cancer.
0:37:54 You give it some very specific tasks and some data, but if you look at people, people learn
0:37:57 so many things.
0:37:59 We learn to all kinds of things.
0:38:00 You can tap the hands.
0:38:02 You can do double entry bookkeeping.
0:38:03 Right?
0:38:04 Right.
0:38:05 You can add numbers.
0:38:11 You can play music, all kinds of things, and a lot of those things we learn over time
0:38:17 in a kind of synergistic way, in a staged sequence.
0:38:21 First you learn to crawl, then you learn to walk, then you learn to run, then you learn
0:38:28 to ride a bike, and it wouldn’t make any sense to do them in the other sequence, because
0:38:31 you’re actually learning to learn.
0:38:37 When you acquire one skill, it puts you in a position that you now are capable of learning
0:38:38 the next skill.
0:38:44 So I’m very interested in what it would mean to give a computer that kind of capability
0:38:51 to do learning for days and weeks and years and decades.
0:38:59 We have a project we call our Never-Ending Language Learner, which started in 2010, running
0:39:03 24 hours a day trying to learn to read the web.
0:39:04 Fascinating.
0:39:07 And there’s something about sort of the longitudinal, right?
0:39:09 We started it in 2010 and it just keeps on going.
0:39:13 So it’s not just like transfer learning from one model to another.
0:39:15 It’s like long running.
0:39:20 It’s long running and it has many different learning tasks.
0:39:25 It’s building up a knowledge base of knowledge about the world.
0:39:33 But to keep it short, I’ll just say we’ve learned so much from that project about how
0:39:39 to organize the architecture of a system so that it can invent new learning tasks as it
0:39:46 goes so that it can get synergy once it learns one thing to become better at learning another
0:39:47 thing.
0:39:53 How, in fact, very importantly, it can use unlabeled data to train itself instead of
0:39:57 requiring an army of data labellers.
0:40:04 So I just think this is an area that’s relatively untouched in the machine learning field.
0:40:11 But looking forward, we’re already seeing an increasing number of embedded machine learning
0:40:14 systems in continuous use.
0:40:22 And as we see more and more of those in the internet of things and elsewhere, the opportunity
0:40:29 for learning continuously for days and weeks and months and years and decades is increasingly
0:40:30 there.
0:40:36 We ought to be developing the ideas, the concepts of how to organize those systems to take advantage
0:40:37 of that.
0:40:38 Yeah.
0:40:41 I love both of these design approaches in that they’re sort of inspired by humans and
0:40:50 sort of humans are mysteriously good at learning and adapting and they sort of shine a spotlight
0:40:53 on where machine learning algorithms are not yet.
0:40:56 So it’s such a fertile area to look for inspiration.
0:41:00 Well, Tom, it’s been a great delight having you on the podcast.
0:41:03 Thanks for sharing about the history and the future of machine learning.
0:41:06 We can tell you’re still fired up after all of these decades.
0:41:11 And so that’s a great delight just to see somebody who is committed, basically, their
0:41:17 life to understanding the mysteries of learning and wishing many good decades to come as you
0:41:19 continue working on it.
0:41:20 Thanks.
0:41:21 Thanks for doing this podcast.
0:41:25 I think it’s a great thing to get a conversation going and it’s a great contribution to do
0:41:25 that.
La ventaja más profunda de crear una nueva categoría de mercado no es solo la innovación en sí, sino el poder unilateral para fijar tu propio precio. En lugar de competir en una arena establecida donde el valor ya está mercantilizado, los creadores de categorías definen tanto el problema como el valor económico de su solución, controlando fundamentalmente el destino de su negocio desde el principio.
Esta conversación profundiza en las duras realidades de llevar un concepto empresarial novedoso a la vida, enfatizando que la creación de categorías tiene menos que ver con un destello de genialidad técnica y más con el trabajo meticuloso de marketing y cambio organizacional. Los fundadores subrayan que, tras identificar un problema latente, el primer trabajo del emprendedor es plantar un nuevo concepto en la mente del cliente, convirtiéndolo en una prioridad principal. La fase posterior, y posiblemente más crítica, es adjuntar una narrativa de valor a ese concepto, que dicta directamente el valor del contrato anual (ACV) y, por extensión, la valoración de la empresa y sus márgenes de crecimiento. Este proceso requiere navegar un complejo escenario de “el huevo o la gallina” donde debes vender una solución de alto valor antes de tener pruebas generalizadas.
Un tema central y recurrente es el papel indispensable del marketing de producto para escalar este esfuerzo. Se describe como el alma de una empresa empresarial en etapa temprana, que abarca tres funciones principales: posicionamiento (enmarcar el problema y la solución), habilitación de ventas (crear herramientas y formación) y, eventualmente, desarrollo de manuales (codificar movimientos de ventas repetibles). Esta función sintetiza las señales del mercado de los primeros clientes visionarios y las transforma en un algoritmo escalable y repetible para el equipo de ventas, sacando a la empresa de la dependencia del carisma del fundador.
La discusión también explora el panorama cambiante de las TI empresariales, donde la línea entre las unidades de negocio y las TI tradicionales se está difuminando. Los creadores de categorías exitosos a menudo venden en esta zona de transición, lo que requiere presentaciones duales tanto a partes interesadas técnicas como no técnicas. En última instancia, el mayor logro de la creación de categorías se considera cambiar la forma en que trabajan las empresas, literalmente creando nuevos roles y estructuras organizativas, lo que cede el control del campo de juego al innovador y obliga a otros a seguir sus reglas.
Ideas Sorprendentes
- La competencia principal en la creación de categorías suele ser el statu quo, no otro proveedor. Estás vendiendo contra un proceso interno existente, una solución temporal o planes de “construirlo nosotros mismos”, lo que requiere una narrativa matizada que no aliena a las personas que poseen el sistema defectuoso actual.
- Los primeros “clientes” empresariales a menudo compran educación, no un ajuste producto-mercado. Un fundador carismático puede generar ingresos y compromiso de clientes que pagan para aprender sobre un nuevo espacio, lo que no es una señal confiable de que el producto se ajuste verdaderamente a una necesidad de mercado escalable.
- Las discusiones de “construir vs. comprar” son una señal positiva fuerte. Si un prospecto está considerando seriamente construir tu solución internamente, valida que el problema es agudo y valioso. El mercado para un producto sigue siendo gigantesco porque a la mayoría de las empresas les faltan los recursos o el compromiso para construirlo y mantenerlo bien.
- La señal real de ajuste producto-mercado en el ámbito empresarial es tardía. No se revela por las ventas tempranas, sino por patrones como despliegues en producción (no solo pilotos), casos de uso repetibles en múltiples clientes y, críticamente, acuerdos cerrados por vendedores sin la participación directa del fundador.
Conclusiones Prácticas
- Retrasa las discusiones de precios hasta después de un cierre técnico. En una nueva categoría, el cliente no puede valorar adecuadamente la solución (o el riesgo de *no* resolverla) hasta que haya visto que funciona. Involúcrate profundamente primero con pilotos o pruebas de concepto, luego negocia el precio.
- Invierte en una función de marketing de producto desde el principio. No contrates solo a vendedores y esperes que ellos descifren la narrativa. El marketing de producto es responsable de crear el posicionamiento, habilitar al equipo de ventas con herramientas y luego desarrollar estrategias de ventas repetibles.
- Analiza rigurosamente los primeros acuerdos para encontrar patrones reales. Separa los “acuerdos por relación” de la señal genuina. Busca puntos en común en el título del comprador, las iniciativas organizativas y los resultados de despliegue para comprender qué es realmente repetible y escalable.
- Cultiva la paciencia y la flexibilidad intelectual. Debes ser firme en tu creencia de un gran mercado pero enormemente abierto a equivocarte en los detalles: la persona inicial, el conjunto de características o el posicionamiento casi seguro evolucionarán en base a los comentarios reales de los clientes a lo largo de los años.
- Construye un consejo asesor de clientes con tus primeros 20 clientes. Usa este grupo para crear pruebas creíbles para futuros prospectos; no hay una herramienta de ventas más poderosa que los clientes existentes que detallan sus casos de uso y valor a sus pares.
Según Bernard J. Tyson, el problema fundamental de la atención médica estadounidense es que hemos construido toda una industria en torno a la transacción de la enfermedad, en lugar de en torno a la relación vitalicia con la salud. Como CEO de Kaiser Permanente, Tyson dirige un sistema deliberadamente estructurado en contra de esa corriente, donde los incentivos financieros se alinean no con llenar camas de hospital, sino con mantener a la gente fuera de ellas. Esta conversación con Ben Horowitz profundiza en la mecánica del modelo integrado y “per cápita” de Kaiser, donde la organización recibe una cantidad fija de dólares por miembro y asume la plena responsabilidad —y el riesgo— de su atención de por vida. Esto crea un poderoso impulso para invertir fuertemente en cuidados preventivos y diagnósticos tempranos, en marcado contraste con el modelo de pago por servicio que domina el resto de la industria y que a menudo obtiene ganancias del volumen y los procedimientos.
La estructura integrada de Kaiser, que posee hospitales, grupos de médicos y seguros bajo un mismo techo, le permite ser pionero en enfoques como la telemedicina generalizada y la atención domiciliaria. Tyson explica que, dado que los médicos no son penalizados ni recompensados financieramente por elegir una hospitalización sobre una recuperación en casa, pueden tomar decisiones basadas únicamente en la necesidad clínica y la eficiencia. Esto ha llevado a tasas de utilización hospitalaria dramáticamente más bajas en comparación con los promedios nacionales. La discusión también explora el papel masivo y a menudo pasado por alto de los “determinantes sociales de la salud” —factores como la vivienda segura, el acceso a alimentos frescos y el empleo estable— que Kaiser aborda activamente en las comunidades a las que sirve, reconociendo que la salud se crea fuera de las paredes de la clínica.
De cara al futuro, Tyson es franco sobre los obstáculos tecnológicos y culturales que enfrenta la innovación en atención médica. Aunque está ansioso por adoptar la IA y el análisis predictivo, enfatiza la importancia de desplegar la tecnología con objetivos claros para los resultados de los pacientes y una eficiencia escalable en todo el sistema, no solo persiguiendo “objetos brillantes”. Reserva sus comentarios más apasionados para la integración crítica de la salud mental y física, describiendo los esfuerzos de Kaiser para desestigmatizar la atención de salud mental integrándola directamente en la atención primaria. En última instancia, Tyson argumenta que la Ley de Cuidado de Salud Asequible, a pesar de su lanzamiento accidentado, fue un paso necesario para cubrir a los “pobres trabajadores”, pero el próximo gran desafío para la nación es abordar la mera asequibilidad de la atención para todos, una tarea que requerirá superar por completo un sistema de “atención para enfermos”.
Ideas Sorprendentes
- En el modelo de Kaiser Permanente, sus 39 hospitales se consideran centros de gasto, no de ingresos. Esto invierte el incentivo estándar de la industria, ya que el sistema pierde dinero cuando un miembro es hospitalizado, lo que impulsa un enfoque en la atención ambulatoria y preventiva.
- El miembro promedio de Kaiser Permanente permanece en la organización durante 18 años. Esta relación a largo plazo proporciona datos de salud longitudinales invaluables y justifica una inversión inicial significativa en la salud de un miembro.
- Un revelador estudio interno encontró que aproximadamente el 30% de las visitas de atención primaria estaban fundamentalmente impulsadas por un problema de salud mental subyacente, como ansiedad o depresión, a menudo enmascarado por quejas físicas como dolor de espalda o insomnio.
- Tyson replantea la Ley de Cuidado de Salud Asequible no como una invención radicalmente nueva, sino como “la versión del siglo XXI del enfoque de Medicaid”, diseñada para cubrir a los “pobres trabajadores”, una clase en gran medida no reconocida por la legislación original de 1965.
- Señala que más del 50% de las personas que mueren por suicidio no mostraban señales de advertencia claras, lo que destaca la profunda complejidad de las crisis de salud mental y la urgente necesidad de mejores herramientas predictivas y apoyo sistémico.
Consejos Prácticos
- Invertir aguas arriba en la prevención: Ya sea como individuo o como organización, asignar recursos a pruebas tempranas, diagnósticos y apoyo para un estilo de vida saludable ahorra costos y sufrimientos significativamente mayores en el futuro, al gestionar condiciones crónicas antes de que se agraven.
- Integrar la salud mental en la atención rutinaria: Trabajar activamente para desestigmatizar la salud mental tratándola como una parte rutinaria del bienestar general. En un entorno clínico, esto puede verse como médicos de atención primaria que tienen especialistas en salud conductual fácilmente disponibles para derivaciones inmediatas y cálidas.
- Evaluar la tecnología a través de la lente de resultados escalables: Al considerar una nueva tecnología, especialmente en sistemas grandes, enfocarse en si mejora la calidad, optimiza la experiencia y puede escalarse efectivamente, en lugar de perseguir innovaciones fragmentadas y de nicho que no impacten al conjunto.
- Abordar el entorno comunitario: Reconocer que los resultados de salud están profundamente ligados a “determinantes sociales” como la vivienda, la nutrición y la seguridad. Las estrategias de salud efectivas deben involucrarse con estos factores comunitarios más amplios, no solo con intervenciones clínicas.
- Abogar por barreras regulatorias que permitan una competencia saludable: En políticas o modelos de negocio, establecer reglas claras —como mecanismos de ajuste de riesgo— que eviten prácticas desestabilizadoras como la selección de pacientes sanos y permitan a los competidores centrarse en ofrecer mejor valor y atención.
A vantagem mais profunda de criar uma nova categoria de mercado não é apenas a inovação em si—é o poder unilateral de definir seu próprio preço. Em vez de competir em um cenário estabelecido onde o valor já foi commoditizado, os criadores de categoria definem tanto o problema quanto o valor econômico de sua solução, controlando fundamentalmente o destino de seu negócio desde o início.
Esta conversa mergulha nas realidades árduas de dar vida a um novo conceito empresarial, enfatizando que a criação de categoria tem menos a ver com um lampejo de genialidade técnica e mais com o trabalho meticuloso de marketing e mudança organizacional. Os fundadores destacam que, após identificar um problema latente, a primeira tarefa do empreendedor é plantar um novo conceito na mente do cliente—tornando-o uma prioridade máxima. A fase subsequente, e indiscutivelmente mais crítica, é vincular uma narrativa de valor a esse conceito, que dita diretamente o valor anual do contrato (ACV) e, por extensão, a avaliação e as margens de crescimento da empresa. Este processo exige navegar por um complexo cenário de “o ovo ou a galinha”, onde é necessário vender uma solução de alto valor antes de ter provas generalizadas.
Um tema central e recorrente é o papel indispensável do marketing de produto na escala desse esforço. Ele é descrito como o elemento vital de uma empresa de enterprise em estágio inicial, abrangendo três funções principais: posicionamento (enquadramento do problema e da solução), capacitação de vendas (criação de ferramentas e treinamento) e, eventualmente, desenvolvimento de playbooks (codificação de movimentos de vendas repetíveis). Esta função sintetiza sinais de mercado de clientes visionários iniciais e os transforma em um algoritmo escalável e repetível para a equipe de vendas, levando a empresa além da dependência do carisma do fundador.
A discussão também explora o cenário em evolução do TI empresarial, onde a linha entre unidades de negócio e TI tradicional está se desvanecendo. Criadores de categoria bem-sucedidos frequentemente vendem para esta zona de transição, exigindo abordagens duplas tanto para partes interessadas técnicas quanto não técnicas. Em última análise, a maior conquista da criação de categoria é vista como mudar a forma como as empresas funcionam—literalmente criando novas funções e estruturas organizacionais—o que cede o controle do campo de jogo ao inovador e força outros a seguir suas regras.
Insights Surpreendentes
- A principal concorrência na criação de categoria costuma ser o status quo, não outro fornecedor. Você está vendendo contra um processo interno existente, uma solução paliativa ou planos de “construir nós mesmos”, o que exige uma narrativa sutil que não afaste as pessoas responsáveis pelo sistema defeituoso atual.
- Os “clientes” iniciais de enterprise muitas vezes estão comprando educação, não fit com o mercado de produto. Um fundador carismático pode gerar receita e engajamento de clientes que estão pagando para aprender sobre um novo espaço, o que não é um sinal confiável de que o produto realmente se encaixa em uma necessidade de mercado escalável.
- Discussões de “construir versus comprar” são um forte sinal positivo. Se um prospect está seriamente considerando construir sua solução internamente, isso valida que o problema é agudo e valioso. O mercado para um produto ainda é gigantesco porque a maioria das empresas não tem recursos ou comprometimento para construí-lo e mantê-lo adequadamente.
- O sinal real de fit com o mercado de produto no enterprise é tardio. Não é revelado pelas vendas iniciais, mas por padrões como implantações em produção (não apenas pilotos), casos de uso repetíveis em vários clientes e—criticamente—negócios fechados pelos vendedores sem o envolvimento direto do fundador.
Conclusões Práticas
- Atraso as discussões de preços até após um fechamento técnico. Em uma nova categoria, o cliente não pode valorizar adequadamente a solução (ou o risco de *não* resolvê-la) até vê-la funcionar. Envolva-se profundamente com pilotos ou provas de conceito primeiro, depois negocie o preço.
- Invista em uma função de marketing de produto desde o início. Não apenas contrate vendedores e espere que eles descubram a narrativa. O marketing de produto é responsável por elaborar o posicionamento, capacitar a equipe de vendas com ferramentas e, posteriormente, desenvolver plays de vendas repetíveis.
- Analise rigorosamente os primeiros negócios para encontrar padrões reais. Separe os “negócios por relacionamento” do sinal genuíno. Procure pontos comuns no cargo do comprador, nas iniciativas organizacionais e nos resultados da implantação para entender o que é realmente repetível e escalável.
- Cultive paciência e flexibilidade intelectual. Você deve ser teimoso em sua crença em um grande mercado, mas totalmente aberto a estar errado nos detalhes—a persona inicial, conjunto de recursos ou posicionamento quase certamente evoluirá com base no feedback real dos clientes ao longo dos anos.
- Construa um conselho consultivo de clientes com seus primeiros 20 clientes. Use este grupo para criar provas credíveis para prospects futuros; não há ferramenta de vendas mais poderosa do que clientes existentes detalhando seus casos de uso e valor para seus pares.
De acordo com Bernard J. Tyson, o problema fundamental do sistema de saúde americano é termos construído toda uma indústria em torno da transação da doença, em vez do relacionamento vitalício com a saúde. Como CEO da Kaiser Permanente, Tyson lidera um sistema deliberadamente estruturado contra essa corrente, onde os incentivos financeiros estão alinhados não para encher leitos hospitalares, mas para manter as pessoas longe deles. Esta conversa com Ben Horowitz aprofunda-se na mecânica do modelo integrado e “per capita” da Kaiser, no qual a organização recebe um valor fixo por membro e assume total responsabilidade – e risco – pelos seus cuidados ao longo da vida. Isso cria um poderoso impulso para investir fortemente em cuidados preventivos e diagnóstico precoce, contrastando fortemente com o modelo de fee-for-service que domina o resto do setor e frequentemente lucra com volume e procedimentos.
A estrutura integrada da Kaiser, que possui hospitais, grupos médicos e seguros sob o mesmo teto, permite-lhe pioneirar abordagens como a telemedicina generalizada e os cuidados domiciliares. Tyson explica que, como os médicos não são penalizados ou recompensados financeiramente por optar por uma internação hospitalar em vez de recuperação domiciliar, eles podem tomar decisões baseadas puramente na necessidade clínica e na eficiência. Isso levou a taxas de utilização hospitalar dramaticamente mais baixas em comparação com a média nacional. A discussão também explora o enorme e frequentemente negligenciado papel dos “determinantes sociais da saúde” – fatores como moradia segura, acesso a alimentos frescos e emprego estável – que a Kaiser aborda ativamente nas comunidades que serve, reconhecendo que a saúde é criada fora dos muros das clínicas.
Olhando para o futuro, Tyson é franco sobre os obstáculos tecnológicos e culturais que a inovação em saúde enfrenta. Embora ansioso para adotar IA e análises preditivas, ele enfatiza a importância de implantar a tecnologia com objetivos claros para os resultados dos pacientes e eficiência escalável em todo o sistema, não apenas perseguindo “objetos brilhantes”. Ele reserva seu comentário mais apaixonado para a integração crítica da saúde mental e física, descrevendo os esforços da Kaiser para desestigmatizar os cuidados de saúde mental, incorporando-os diretamente aos cuidados primários. No final, Tyson argumenta que o Affordable Care Act, apesar de seu lançamento conturbado, foi um passo necessário para cobrir os “pobres trabalhadores”, mas o próximo grande desafio para a nação é enfrentar a mera acessibilidade dos cuidados para todos, uma tarefa que exigirá ir além de um sistema de “cuidados para doentes”.
Conhecimentos Surpreendentes
- No modelo da Kaiser Permanente, seus 39 hospitais são considerados centros de custo, não centros de receita. Isso inverte o incentivo padrão do setor, pois o sistema perde dinheiro quando um membro é hospitalizado, impulsionando o foco em cuidados ambulatoriais e preventivos.
- O membro médio da Kaiser Permanente permanece na organização por 18 anos. Esse relacionamento de longo prazo fornece dados longitudinais de saúde inestimáveis e justifica um investimento inicial significativo na saúde de um membro.
- Um revelador estudo interno descobriu que aproximadamente 30% das consultas de cuidados primários eram fundamentalmente impulsionadas por um problema de saúde mental subjacente, como ansiedade ou depressão, muitas vezes mascarado por queixas físicas como dor nas costas ou insônia.
- Tyson reformula o Affordable Care Act não como uma invenção radicalmente nova, mas como “a versão do século XXI da abordagem do Medicaid”, projetada para cobrir os “pobres trabalhadores”, uma classe largamente não reconhecida pela legislação original de 1965.
- Ele aponta que mais de 50% das pessoas que morrem por suicídio não mostraram sinais de alerta claros, destacando a profunda complexidade das crises de saúde mental e a necessidade urgente de melhores ferramentas preditivas e apoio sistêmico.
Conclusões Práticas
- Invista a montante na prevenção: Quer como indivíduo ou organização, alocar recursos para testes precoces, diagnóstico e apoio a estilos de vida saudáveis economiza custos e sofrimentos significativamente maiores no futuro, ao gerenciar condições crônicas antes que elas se agravem.
- Integre a saúde mental nos cuidados de rotina: Trabalhe ativamente para desestigmatizar a saúde mental tratando-a como uma parte rotineira do bem-estar geral. Em um ambiente clínico, isso pode parecer com médicos de cuidados primários tendo especialistas em saúde comportamental prontamente disponíveis para transferências imediatas e acolhedoras.
- Avalie a tecnologia através da lente de resultados escaláveis: Ao considerar novas tecnologias, especialmente em grandes sistemas, concentre-se em saber se ela melhora a qualidade, aprimora a experiência e pode ser dimensionada com eficácia, em vez de buscar inovações fragmentadas e de nicho que não impactarão o todo.
- Aborde o ambiente comunitário: Reconheça que os resultados de saúde estão profundamente ligados a “determinantes sociais” como habitação, nutrição e segurança. Estratégias de saúde eficazes devem envolver esses fatores comunitários mais amplos, não apenas intervenções clínicas.
- Defenda regulamentações que permitam uma competição saudável: Em modelos de política ou negócios, estabeleça regras claras do jogo – como mecanismos de ajuste de risco – que impeçam práticas desestabilizadoras, como a seleção de pacientes saudáveis, e permitam que os concorrentes se concentrem em oferecer melhor valor e cuidados.
How have we gotten to where were are with machine learning? Where are we going?
a16z Operating Partner Frank Chen and Carnegie Mellon professor Tom Mitchell first stroll down memory lane, visiting the major landmarks: the symbolic approach of the 1970s, the ”principled probabalistic methods” of the 1980s, and today’s deep learning phase. Then they go on to explore the frontiers of research. Along the way, they cover:
- How planning systems from the 1970s and early 1980s were stymied by the ”banana in the tailpipe” problem
- How the relatively slow neurons in our visual cortex work together to deliver very speedy and accurate recognition
- How fMRI scans of the brain reveal common neural patterns across people when they are exposed to common nouns like chair, car, knife, and so on
- How the computer science community is working with social scientists (psychologists, economists, and philosophers) on building measures for fairness and transparency for machine learning models
- How we want our self-driving cars to have reasonable answers to the Trolley Problem, but no one sitting for their DMV exam is ever asked how they would respond
- How there were inflated expectations (and great social fears) for AI in the 1980s, and how the US concerns about Japan compare to our concerns about China today
- Whether this is the best time ever for AI and ML research and what continues to fascinate and motivate Tom after decades in the field
The views expressed here are those of the individual AH Capital Management, L.L.C. (“a16z”) personnel quoted and are not the views of a16z or its affiliates. Certain information contained in here has been obtained from third-party sources, including from portfolio companies of funds managed by a16z. While taken from sources believed to be reliable, a16z has not independently verified such information and makes no representations about the enduring accuracy of the information or its appropriateness for a given situation.
This content is provided for informational purposes only, and should not be relied upon as legal, business, investment, or tax advice. You should consult your own advisers as to those matters. References to any securities or digital assets are for illustrative purposes only, and do not constitute an investment recommendation or offer to provide investment advisory services. Furthermore, this content is not directed at nor intended for use by any investors or prospective investors, and may not under any circumstances be relied upon when making a decision to invest in any fund managed by a16z. (An offering to invest in an a16z fund will be made only by the private placement memorandum, subscription agreement, and other relevant documentation of any such fund and should be read in their entirety.) Any investments or portfolio companies mentioned, referred to, or described are not representative of all investments in vehicles managed by a16z, and there can be no assurance that the investments will be profitable or that other investments made in the future will have similar characteristics or results. A list of investments made by funds managed by Andreessen Horowitz (excluding investments and certain publicly traded cryptocurrencies/ digital assets for which the issuer has not provided permission for a16z to disclose publicly) is available at https://a16z.com/investments/.
Charts and graphs provided within are for informational purposes solely and should not be relied upon when making any investment decision. Past performance is not indicative of future results. The content speaks only as of the date indicated. Any projections, estimates, forecasts, targets, prospects, and/or opinions expressed in these materials are subject to change without notice and may differ or be contrary to opinions expressed by others. Please see https://a16z.com/disclosures for additional important information.
Leave a Reply
You must be logged in to post a comment.