0:00:05 The content here is for informational purposes only, should not be taken as legal business
0:00:10 tax or investment advice or be used to evaluate any investment or security and is not directed
0:00:14 at any investors or potential investors in any A16Z fund.
0:00:19 For more details, please see a16z.com/disclosures.
0:00:21 Hi and welcome to the A16Z podcast.
0:00:22 I’m Hannah.
0:00:26 The advent of new gene and cell therapies are beginning to approach that holy grail of
0:00:31 medicine, that of a possible cure, but they are also more expensive than any medicines
0:00:32 ever sold before.
0:00:37 In this episode, MIT economist Andrew Lowe and A16Z general partner on the biofund Jorge
0:00:43 Conde discuss how exactly we place an economic value on a cure, the questions we still need
0:00:47 to figure out, like who should pay for what and how, and how we need to start thinking
0:00:51 about handling the coming influx of highly priced medicines like this into our healthcare
0:00:52 system.
0:00:57 Jorge and Andrew also talk about the economics and risk of drug discovery and development
0:01:02 overall and how our markets might just function more like biological systems than anything
0:01:03 else.
0:01:06 So what does an economist know about healthcare?
0:01:08 Yeah, that’s a good question.
0:01:12 I probably should start with a disclaimer, kind of like the disclaimer that you find
0:01:14 on drugs.
0:01:20 This podcast could cause confusion, irritability, and drowsiness, but it should all pass within
0:01:21 a few minutes.
0:01:22 That’s the black box warning.
0:01:23 Right.
0:01:24 Exactly.
0:01:25 My background’s really in financial economics.
0:01:29 I got interested in this because friends and family were dealing with cancers of various
0:01:33 different types, and so the more I learned about what they were going through, the more
0:01:37 I got curious about the underlying economics of cancer drug development.
0:01:42 A lot of the work that you have done as an economist has been around thinking about value.
0:01:46 We have one approved gene therapy that sparks therapeutics, gene therapy for a rare inherited
0:01:48 form of blindness.
0:01:51 There are many, many more in the clinic.
0:01:55 There are many engineered cells or cell therapies that are also in the clinic.
0:02:01 These therapies have one great promise that has been an all too short supply in our industry,
0:02:03 which is the promise of a potential cure.
0:02:08 How do we think about the value of a cure in a system, and specifically how do we think
0:02:14 about pain for cures in a system that really isn’t designed for that?
0:02:16 Yeah, that’s a really interesting challenge.
0:02:20 That challenge, by the way, is only going to grow because although we only have one gene
0:02:26 therapy right now on the market, there are over 300 gene therapy clinical trials registered
0:02:28 at clinicaltrials.gov.
0:02:32 My guess is that within the next three years, we’re probably going to see somewhere between
0:02:35 five and 10 gene therapy approvals.
0:02:40 We’re talking about a pretty significant impact on payers and patients.
0:02:45 Just to give people a sense of scale, so look, Sterna, the spark therapeutics, gene therapy
0:02:48 for blindness, what’s the sticker price on that?
0:02:55 The list price right now is $850,000 for both eyes, $425,000 per eye.
0:02:59 We’re talking about some really expensive therapies, and that’s not even the most expensive.
0:03:05 People are talking about therapies for, say, hemophilia A, that if it gets approved, a
0:03:09 price tag could be well north of a million and a half dollars.
0:03:13 When you think about these kind of numbers, it’s really shocking, but the way I think
0:03:20 about it as an economist is, is it justified given the kind of value that it creates for
0:03:21 consumers?
0:03:25 Typically, the way I think about value is very much the way that someone would think
0:03:28 about value from any kind of a commodity purchase.
0:03:35 If you buy a home, the price of the home should be related to some degree to the housing services
0:03:41 that the home will provide you over the course of your lifetime and beyond.
0:03:46 We typically think about a house is not simply what is going to provide you in terms of shelter
0:03:48 for the next month.
0:03:51 That would be an apartment that you would rent one month at a time.
0:03:56 When we buy a house, as opposed to renting an apartment, we’re buying a stream of future
0:03:58 housing services.
0:04:01 I think about that the same way that I think about gene therapies.
0:04:07 When you are cured of a disease, you’re basically getting a lifetime of health, as opposed to
0:04:13 renting health one pill at a time with a chronic manageable condition.
0:04:18 When thinking about that framework, it stands to reason that if you’re going to get value
0:04:23 from a particular cure over a period of your life, you ought to be able to amortize your
0:04:26 payments over that kind of a period.
0:04:32 If you think about the idea behind a mortgage for a home, we can also apply that to thinking
0:04:35 about a mortgage for drugs, drug mortgages.
0:04:37 To mortgage for a cure.
0:04:38 Yeah.
0:04:39 Who pays that mortgage?
0:04:41 Well, that’s a good question.
0:04:47 In my view, insurance companies should pay for it because that’s why we have health insurance.
0:04:51 There are challenges, though, with the current system in getting insurance companies to pay
0:04:56 for it simply because they’re going to have a hard time in terms of balancing the cost
0:05:00 of the insurance with the payouts that they have to make.
0:05:05 Typically the way insurance works, if you join an insurance plan, you pay premiums
0:05:09 every year and in the event that you get ill, they cover your cost.
0:05:13 And the idea is that the amount that you’re paying every year, on average, should roughly
0:05:18 equal the cost that they’re paying out, maybe a little bit less so that they can make a
0:05:21 profit from their business.
0:05:26 That economic calculus becomes very difficult when you’ve got patients that can leave a
0:05:28 plan after two or three years.
0:05:36 So imagine a health insurance company, A, paying for a cure for you and with the expectation
0:05:39 that now that you’re healthy, you’re going to be able to pay premiums for the rest of
0:05:42 your life and that will make up for the cost of the cure.
0:05:48 But suppose that after three years, you leave plan A and move to plan B. Now, plan B has
0:05:52 the benefit of your health and the premiums that you’re going to be paying for the rest
0:05:57 of your life, whereas plan A is stuck having paid for your therapy.
0:06:04 So the solution to this is to allow plan A to amortize, to space out the payment of your
0:06:07 therapy over the course of many years.
0:06:12 And if you leave after three years, the remaining payments, the remaining drug mortgages that
0:06:18 the health plan A was supposed to pay, that moves to plan B. And in a way, that creates
0:06:23 a level playing field for all insurance companies so nobody is disincentivized to provide those
0:06:24 cures to their patients.
0:06:30 So that’s a fascinating concept because one of the things when people think about insurance,
0:06:35 when they think about people moving around from plan to plan, and we have now legislation
0:06:41 around this, is, well, what’s the obligation of the payer to pay for a pre-existing condition?
0:06:45 We’ve always assumed a pre-existing condition as a disease, but what you’re describing
0:06:50 is in some cases in the future, the pre-existing condition might actually be a cure.
0:06:54 And so there’ll be a liability not associated with the treatment of a disease, but a liability
0:06:56 associated with the treatment for a cure.
0:06:57 Exactly.
0:07:02 And so we can actually fix the system with one stroke of a pen by changing in the Affordable
0:07:09 Care Act the sentence that prevents insurers from refusing to serve a planned participant
0:07:14 because of pre-existing conditions to change that to pre-existing financial conditions as
0:07:16 well.
0:07:20 And if we do that, then this whole issue about mobility is not a problem.
0:07:23 But good luck trying to get legislators to do that right now.
0:07:25 I think it’s going to be a challenge.
0:07:30 Do we know that debate has even taken place because it does feel like a logical solution?
0:07:35 As far as I know, the debate has not yet taken place, largely because there’s no need.
0:07:40 But if we institute these payment plans for one-time therapies and we’re able to start
0:07:46 developing cures for some of the larger prevalence diseases, for example, we’re on the verge of
0:07:50 developing a cure for sickle cell anemia, well, there are 100,000 patients in the United
0:07:55 States with sickle cell, and that’s going to hit health plans pretty hard.
0:08:00 A few weeks ago in the UK, a group announced the launch of a clinical trial for applying
0:08:04 gene therapy to deal with age-related macular degeneration.
0:08:09 There are 600,000 AMD patients in the UK, 2 million patients in the US.
0:08:13 If we have a gene therapy for AMD that succeeds, that’s going to be a real problem for our
0:08:15 health care system.
0:08:21 The concept of a drug mortgage or a cure, a mortgage for cures, it’s a fascinating one.
0:08:27 But when we use the analogy of a home, does it start to break down in the sense that I
0:08:34 know how much I’m going to pay for my mortgage because the value of the house is agreed upon
0:08:37 and determined when I buy that house.
0:08:42 And in some ways, the equity I get in my home is the difference between what the value was
0:08:48 when I bought it versus the value when I sell it and I end up paying off the mortgage at
0:08:49 that point in time.
0:08:57 In the case of the cure, how do we get agreement on what the value of the cure is at the moment
0:08:59 of treatment?
0:09:05 Because I would imagine that for a patient, they’re going to put one value on it, society
0:09:08 may put a different value on it, the payer may put a different value on it.
0:09:14 So how do we get to agreement on value so we can from there determine the mortgage payments?
0:09:17 Well, different countries answer that question differently.
0:09:22 For example, in the United Kingdom, their process for determining what the price of
0:09:28 a drug should be or what an acceptable price is is a group of individuals that do pharmacoeconomic
0:09:31 analysis to try to calculate cost effectiveness.
0:09:34 This is an organization called NICE, N-I-C-E.
0:09:39 And this organization does economic studies to try to determine whether or not at a given
0:09:43 price it’s worthwhile for their society.
0:09:48 And not everybody agrees with what NICE comes up with, but the bottom line is that the National
0:09:52 Health Service in the United Kingdom will take the recommendations of NICE and follow
0:09:53 through.
0:09:59 Now, in our country, we would call that a death panel and it’s a politically loaded term,
0:10:02 but effectively that’s unfortunately what it is.
0:10:05 You have to make trade-offs between money and lives saved.
0:10:08 Now, in the United States, we have a very different system.
0:10:10 It’s a multi-payer system.
0:10:15 And at first I thought, oh, that just means it’s a free market, but in fact, the complexity
0:10:21 of the regulation surrounding the pharma industry is anything but a free market.
0:10:26 There are all sorts of incentives in certain cases that cause drug prices to be higher than
0:10:30 usual and in other cases to force them to be lower than usual.
0:10:35 I think that the best way to think about it is to start back and ask the question, what
0:10:37 do patients get out of it?
0:10:39 What is the value to a patient?
0:10:44 We can layer on top of that all sorts of other considerations, but to me, the patient should
0:10:46 be the first consideration.
0:10:48 And there are economists who study that all the time.
0:10:51 They come up with the notion of quality-adjusted life years.
0:10:54 And they ask the question, if you’re going to save a patient’s life, if you’re going
0:11:00 to literally cure the patient of an unfortunate early death, you can actually calculate the
0:11:04 number of quality-adjusted life years you’ve given back to that patient.
0:11:07 And that’s the beginning of where value comes from.
0:11:13 The other argument would be, if you cure a disease, you avoid the cost of treating that
0:11:15 disease over the lifetime.
0:11:21 So one other argument for pricing or giving value to a cure is to say, well, let’s take
0:11:25 the net present value of all of the avoided costs.
0:11:28 And so I’m going to ask you an overly simplified question.
0:11:36 Would a way to price these cures be, let’s take the value of the qualities, the quality-adjusted
0:11:43 life years, and add to that the net present value of the costs avoided?
0:11:48 Well, that’s certainly a consideration that people have used in making an argument one
0:11:50 way or the other for pricing.
0:11:53 But I think that you have to be careful because there are many other factors then that you
0:11:55 might want to also bring in.
0:12:01 For example, if a patient is now going to live a natural life, that patient will pay taxes
0:12:03 for the rest of his or her life.
0:12:06 Should we factor that into the calculus?
0:12:10 That patient, on the other hand, will be consuming other aspects of society.
0:12:15 And for example, if that particular patient ends up becoming unemployed, then they’ll
0:12:17 be drawing unemployment insurance.
0:12:21 So you can go down this rabbit hole of all sorts of costs and benefits that you have
0:12:23 to do the math for.
0:12:28 And unless we’re really set up to do that, it’s very easy to cherry-pick the particular
0:12:33 statistic that makes your case look stronger than the opposing side.
0:12:37 So I think that the approach that we seem to be gravitating towards is to try to come
0:12:42 up with an objective notion of value, and there’s an organization called ICER.
0:12:47 It’s a nonprofit organization that’s job is to focus on developing the cost-effectiveness
0:12:53 studies, maybe using that as a starting point, and then going from there to try to see whether
0:12:58 or not we can come up with a rational pricing model that basically all stakeholders can
0:12:59 live with.
0:13:03 As ICER looked into the spark therapeutic struggle, I believe they have.
0:13:08 I know that they did a study on gene therapies in general, and they were very positively
0:13:13 inclined, particularly given the cost of many of the diseases without the gene therapy.
0:13:16 Hemophilia is a good example, even at a price of, say, one and a half million.
0:13:19 We don’t actually know what the price will be.
0:13:23 One and a half million is actually a bargain relative to what it costs to deal with a current
0:13:30 hemophilia patient, which can be anywhere from 300,000 to 500,000 a year for the rest
0:13:32 of that patient’s natural life.
0:13:37 So at one and a half million, that seems to be a bargain from that kind of calculus.
0:13:40 So it’s a fascinating conundrum.
0:13:46 If we look at the history of the biotechnology industry, we’ve seen a small-scale version
0:13:50 of this challenge with rare genetic diseases.
0:13:55 So to pick one sort of well-known example, when Genzyme comes onto the scene, the founder
0:14:02 and CEO of that company, Henry Tirmier, pioneered the idea of, you know, we can develop and
0:14:08 commercialize therapies for these patients that have these rare diseases.
0:14:12 And there’s a lot of value associated with treating them and alleviating the conditions
0:14:14 to the extent that we can.
0:14:18 And as a result, we can charge, we can price it based on that value.
0:14:23 So those therapies were on the order of $100,000 and above.
0:14:29 Well, oh my gosh, how can we possibly pay $100,000 for a therapy?
0:14:35 And part of the rationale for doing it was, well, these are rare diseases, so collectively
0:14:42 the probability that you would have many patients in any given plan that are going to have this
0:14:49 therapy in general, on scale, these were going to be very much one-off events for most plans,
0:14:54 and therefore they could shoulder that kind of payment.
0:14:58 What’s fascinating about what we’re describing here, and you’ve given a lot of great examples,
0:15:06 like SMA, hemophilia, AMD, these are all diseases that even if they were individually rare, which
0:15:11 many of these are not, collectively this is going to be a common condition.
0:15:16 And so we’re going to see another reckoning, I think, across the industry in having to
0:15:23 think through this problem that Gensheim tackled from the micro scale, at the more macro scale.
0:15:26 And my take on this is, I’m an optimist.
0:15:31 My assumption is that the innovation is going to lead the regulation, the policy, the coverage
0:15:36 here, and that we will get to an answer on how to actually make sure that patients get
0:15:37 these cures.
0:15:42 Well, I’m absolutely optimistic that we’ll come up with an answer, but I don’t believe
0:15:46 we’re going to get to that answer until the system is forced to deal with this issue in
0:15:48 a direct way.
0:15:53 And maybe that ends up being when we develop a gene therapy for a really big indication
0:15:55 like Alzheimer’s.
0:15:58 There are currently 5 million patients that have Alzheimer’s.
0:16:03 If we develop a gene therapy to stop the progression of the disease or to be able to delay the
0:16:08 onset, that’s going to create tremendous pressure on the healthcare system.
0:16:11 And at that point, we’re really going to have to confront this as a nation.
0:16:14 And it’s complicated because you’re dealing with life and death issues.
0:16:19 When we think about pricing things like a car, it’s not a big deal because, well, if you
0:16:22 can’t afford a car, maybe you’ll get a cheaper car.
0:16:25 But if you need a drug, you can’t afford to get a cheaper drug if there’s only one drug
0:16:27 that cures your disease.
0:16:31 So I think that’s one of the reasons why economics, as much as I love the subject and feel that
0:16:34 it’s critical, these are not just economic considerations.
0:16:39 We have to bring in all of the relevant stakeholders to make these decisions, and that’s really
0:16:41 what our policymakers are trying to do.
0:16:46 It’s clear we’re on the cusp of a new age in medicine, and it’s going to be fascinating
0:16:48 to see how all of this plays out.
0:16:51 One of the things that I found most surprising about your work is what the work you’ve done
0:16:57 around trying to develop risk metrics for drug discovery.
0:17:00 Can we talk a little bit about some of the conclusions that you drew from that and how
0:17:01 we can apply them?
0:17:05 That’s a really interesting aspect of the healthcare industry that puzzled me for the
0:17:07 longest time.
0:17:11 When I first started looking at the way drugs were developed, I couldn’t understand why
0:17:17 it was the case that so many amazing breakthroughs have been made over the years while at the
0:17:21 same time, I kept hearing about the fact that there’s this valley of death and there’s not
0:17:25 enough funding at the early stages of drug discovery.
0:17:30 The more I looked into it, the more I realized that these incredible breakthroughs are actually
0:17:34 increasing the economic risks of drug discovery.
0:17:39 When I first looked at Eroom’s law, I never heard a professor at Eroom, it took me a while
0:17:44 to realize, “Oh, Eroom is more spelled backwards, the opposite of Moore’s law,” and the idea
0:17:49 that as we get smarter, as pioneers in the drug development field develop all of these
0:17:56 amazing therapies, that it actually gets harder from an investment’s perspective, that was
0:17:57 really counter-intuitive.
0:18:01 Usually, in my field, as we get smarter, typically things get easier.
0:18:06 The more you know about a company, for example, the less risky is an investment in that company.
0:18:07 That’s right.
0:18:08 But that’s not true with drug development.
0:18:12 Thinking about drug development requires thinking carefully about risk.
0:18:17 The more you know about the underlying biology of disease, the riskier it can get from a
0:18:19 financial perspective.
0:18:24 The reason for that is that as we learn more about these various different mechanisms of
0:18:30 disease and how to deal with them, it can actually increase the risk of a drug becoming
0:18:36 obsolete because some young upstart decides to try this new pathway that ends up working
0:18:37 really well.
0:18:40 A good example is combination therapies.
0:18:46 We now know that one or two drugs that don’t work particularly well can work magnificently
0:18:48 well when put together.
0:18:53 The best example is the HIV cocktail, the five antiretroviral therapies that by themselves
0:18:57 don’t do very much, but when you put them together, they can turn a deadly disease into
0:18:59 a chronic manageable condition.
0:19:02 That’s an interesting example, because if you look at HIV cocktail therapy, there was
0:19:08 a theory for why the combination of the compounds that go into the cocktail would work, to some
0:19:13 block the attachment of the virus to the cell, some don’t allow the cell to replicate.
0:19:18 That had a logical basis for the combination therapy, but that’s not always true.
0:19:24 Sometimes the synergy of therapies happened experimentally or they’re observed experimentally.
0:19:26 They’re not sort of a priori known.
0:19:32 But the fact that we now know that combination therapies can work, that means that from a
0:19:37 scientific as well as an ethical perspective, we’re obligated to search for combinations
0:19:38 to come up with new therapies.
0:19:41 In fact, some people say that we don’t need any more new drugs.
0:19:44 We’ve got all the drugs we need of the 2800 drugs that are approved.
0:19:48 All we need to do is to find the right combination to treat all disease.
0:19:51 So now that we are smarter and we know the combination therapies work, what does that
0:19:52 mean?
0:19:57 It means that the drug development landscape has become that much more complex.
0:20:01 And it also means that for existing pharma companies that have drugs that are producing
0:20:07 good revenues, their revenues can be wiped out by a new combination that just comes out
0:20:12 of the blue because some researchers hit upon that just by accident or by theory.
0:20:15 But the flip side of that is there is latency in the system, right?
0:20:21 In other words, it takes, assuming that one could hypothesize or test a combination, the
0:20:27 period of time from which that original hypothesis is tested until it’s actually impacting patient
0:20:29 care can be on the order of years.
0:20:34 It could be, but that just means that we now understand the risks can be drawn out over
0:20:36 a period of years.
0:20:41 And so that really surprised me in that idea that we actually, as we get smarter, things
0:20:43 could get riskier.
0:20:44 That’s fascinating.
0:20:50 So to set the table a bit in terms of the drug discovery and development paradigm, I think
0:20:56 in the U.S. alone, the biopharmaceutical industry spends about $75 billion per annum investing
0:20:57 in R&D.
0:21:00 And a big chunk of that is the D side of the equation, right?
0:21:05 The statistic that gets thrown around a lot is that it takes on average about 10 to 15
0:21:12 years for a new drug to reach approval, to get FDA approval and reach patients.
0:21:17 And when you account for all of the failures along the way, the total amount of investment
0:21:23 to get to that one drug 10 to 15 years later is something on the order of $2.5 billion.
0:21:28 If you just look at sort of the survival statistics, it’s something on the order of 1 in 10,000
0:21:32 compounds actually makes it all the way through this gauntlet.
0:21:35 Are those statistics real?
0:21:37 Have they been born out in the work that you’ve done?
0:21:45 Well, the short answer is no, mainly because what we’re trying to do is to understand how
0:21:48 these statistics change over time.
0:21:53 So the numbers that you cited are accurate when you look at the entire expanse of history.
0:21:57 But the problem with that is that medicine has changed a lot.
0:22:02 In particular, if we think about the fact that it was only since 2003 that the human
0:22:06 genome was sequenced, we’re not even 20 years out.
0:22:11 And nowadays, we understand that sequencing the human genome is fundamental for lots of
0:22:14 diseases and therapies like cancer, immunotherapies.
0:22:19 So we’re in a very different period today than we were even 10 years ago.
0:22:24 So when you take a look at statistics like it takes $2.6 billion to develop a drug or
0:22:28 the probability of success for developing a drug in oncology is 5%, those numbers are
0:22:34 aggregated over a period of time and over a particular sample of firms.
0:22:37 What we’ve done over the course of the last couple of years is to try to come up with
0:22:43 better numbers, numbers that use a larger data set over a longer period of time, but
0:22:47 looking at it through time so that we can actually see what the trends are.
0:22:49 So here’s a case in point.
0:22:55 If you take data from 2000 to 2015, so that’s 15 years of data, the probability of success
0:23:00 in oncology in cancer drugs is actually less than 5%.
0:23:06 But if you look at the last five years, the probability of success is triple the historical
0:23:07 rate.
0:23:12 So just within the last five years, the probability of success in oncology is about 10% to 15%.
0:23:16 Especially if you look at lead indications and you add biomarkers, so when you stratify
0:23:19 the data in that way, you get a very different picture.
0:23:24 Even if the probability of success is 10% to 15%, that obviously means that 85% to 90%
0:23:25 can still fail.
0:23:30 The main reason that drugs tend to fail is either because the target that you’re trying
0:23:36 to hit with your drug, with your molecule, doesn’t necessarily modulate the disease.
0:23:39 That doesn’t have the intended effect.
0:23:44 Or it does modulate the disease, but it’s also important elsewhere, so you get sort
0:23:48 of on target toxicity for that molecule.
0:23:54 Or you made the wrong molecule and the molecule either hits the target, maybe not hard enough
0:23:58 or hits that target and a lot of other targets, and so then you get off target toxicity.
0:24:03 We’re learning a lot more about biology and what confounded you and as you entered in
0:24:05 the space that actually has increased the risk.
0:24:10 How do you sort of connect the dots between that insight with the last data point you just
0:24:13 gave us that the success rates are actually going up and not down?
0:24:16 How do I connect those two thoughts in my mind?
0:24:20 Because I would imagine if the risk was going up, that means success is going down, but it
0:24:22 sounds like you’re saying both are happening.
0:24:25 That’s right, and that’s really a fascinating conundrum.
0:24:31 First of all, when my coauthors and I calculate success rates, what we’re looking at is path
0:24:37 by path success rates, meaning if you start off with a compound at phase one, what are
0:24:41 the chances that that will eventually become a drug?
0:24:44 That is, you can file successfully for a new drug application.
0:24:51 We’re actually following a drug coupled with an indication through the entire process.
0:24:56 It is true that when you look at it path by path, the success rates are going up.
0:25:01 At the same time, what I was talking about was the financial risks of investing in these
0:25:05 things, and that’s also going up.
0:25:06 Here’s the key.
0:25:12 The key is that how many investors do you know that put money in a phase one trial and leave
0:25:16 it there and follow through until NDA?
0:25:17 Very, very few.
0:25:23 Yeah, so they basically may trade out of that investment at a value inflection point, whether
0:25:27 that’s an IPO or an acquisition, and that’s uncoupled with the approval of the drug.
0:25:28 That’s right.
0:25:34 In fact, those are the good events, but more often than not, they get wiped out because
0:25:40 a compound has these off-target effects, toxicity, or it’s not doing what it’s supposed to do.
0:25:45 As a result, they basically get the plug pulled on the project, and they lose all of their
0:25:46 initial investment.
0:25:52 A good example is a drug Velcade, which is a very successful drug, but it probably wiped
0:25:58 out two or three sets of investors before it actually got to approval.
0:25:59 That’s the problem.
0:26:02 Drug development is actually not a marathon.
0:26:07 Actually, I think it’s more of a relay race of marathons, and the problem is that very
0:26:11 often we drop the baton, and that’s the challenge.
0:26:16 That’s the increase in risk, while at the same time, we are also increasing approval
0:26:17 rates.
0:26:20 It’s ironic, but they’re both true.
0:26:25 The work that you’ve done on understanding risk and benefit and drug discovery, does
0:26:27 it hold true across different disease areas?
0:26:32 Because we’ve spent a lot of time talking about cancer and combination therapies.
0:26:37 You mentioned HIV as well, but how universal a truth is this across disease areas?
0:26:41 Well, there are a couple of things that are common across disease areas, and then there’s
0:26:44 certain aspects that are very different.
0:26:50 The commonality is that drug development takes a very long time.
0:26:55 Because of the length, if the funding gets interrupted midway, it really destroys a tremendous
0:26:57 amount of value.
0:27:01 Where there are differences, though, in the risks across the therapeutic areas is that
0:27:03 certain areas are just harder than others.
0:27:08 For example, we now know that Alzheimer’s is probably the most difficult challenge facing
0:27:11 our nation and frankly the world today.
0:27:17 It’s since 2003, the probability of success for Alzheimer’s drug development is zero.
0:27:22 We have had literally no drugs since 2003 to treat Alzheimer’s.
0:27:26 So we need to think about a different approach for that particular area.
0:27:28 Now let me turn to vaccines.
0:27:33 The probability of success for developing a vaccine is north of 30%.
0:27:36 So it seems like vaccines is a slam dunk.
0:27:40 And yet, why aren’t we developing vaccines for lots of the diseases that we need to
0:27:41 deal with?
0:27:44 Because the economics of vaccines- Financial incentives are not in place.
0:27:45 Exactly, they don’t work.
0:27:50 So that’s another area where despite the differences in the probability of success, the economics
0:27:54 are really preventing us from dealing with that terrible set of afflictions.
0:28:00 Going back to the first point you made earlier in terms of the risk going up as we learn
0:28:01 more about disease.
0:28:08 I think that’s very true in the world where each drug discovery program was its own sort
0:28:10 of bespoke thing.
0:28:16 So you made a target, you made a molecule for a specific target and if that was successful,
0:28:17 fantastic.
0:28:27 But when you had a second program as an innovator, there’s really not a whole lot of knowledge
0:28:33 or value you could take from that first program into the next one.
0:28:36 Value doesn’t really accrue from one program to the next.
0:28:39 And of course, by extension, if there’s a failure, there’s not a whole lot of experience
0:28:44 that you can take from that first program into the second.
0:28:48 But that starts to change when you start to talk about things like gene therapy and cell
0:28:54 therapy in that if we figure out how to deliver genes to specific cell types, in other words
0:29:00 if the vehicles get very sophisticated, that could be used across a number of diseases.
0:29:02 And that was one of the big risk factors for gene therapy.
0:29:07 So if we went back over the last couple of decades as this approach was being developed,
0:29:11 the same as I’m sure going to be very true for things like CAR-T, what follows in terms
0:29:14 of engineered cells, there are going to be built off of some of the innovations, all
0:29:17 of the components that are used in these kinds of therapies.
0:29:19 So value will start to accrue over time.
0:29:25 We’ll have therapeutic platforms that could be very broadly applied and won’t be as bespoke.
0:29:29 Have you given thought as to how that impacts sort of the risk model?
0:29:34 Because in this case, it might be that as we learn more risk actually goes down because
0:29:38 you know if it worked in delivering a gene to one cell, it is more likely to work in
0:29:41 delivering a different gene to a different cell.
0:29:42 So you’re absolutely right.
0:29:46 You give a great example where as we learn more about delivering genes, we’re going to
0:29:49 actually reduce the risk across all gene therapies.
0:29:51 We only have one approved gene therapy, of course, but there are a number of clinical
0:29:56 trials and many of them are using this AAV vector.
0:29:59 What happens if we discover that there are some off-target effects?
0:30:03 That’s going to affect all of these gene therapies at the same time.
0:30:09 So in a way, using these commonly used techniques, that’s a good thing because we’ve learned
0:30:14 that they seem to work, but at the same time they also build certain kinds of common risk
0:30:17 factors across all of these therapies.
0:30:22 So I think we need to be aware of that and we need to assess that kind of risk when we’re
0:30:24 dealing with these types of therapies.
0:30:25 It cuts both ways.
0:30:26 Obviously, we learn more.
0:30:27 I see.
0:30:28 Yes, it’s systemic risk.
0:30:29 But it’s systemic.
0:30:30 Exactly right.
0:30:31 I see.
0:30:32 Yeah.
0:30:34 So you’ve mentioned that one of your early surprises or at least one of your early insights
0:30:39 when you came into the healthcare space is that in many ways it’s not really a rational
0:30:40 market.
0:30:46 There are structurally a lot of impediments to have a sort of truly well-functioning market
0:30:48 in healthcare.
0:30:55 There’s regulation, there’s cross-incentives, there’s general issues around economic feasibility
0:31:00 of being able to run healthcare systems in the first place.
0:31:05 And I read a quote from you where you said, “Markets behave more like biological systems
0:31:07 than physical devices.”
0:31:09 Can you explain what you mean by that?
0:31:14 Well, for one thing, we ought to think about using evolutionary biology as a paradigm for
0:31:18 understanding how markets change over time.
0:31:24 Markets are not immutable objects that are set in a particular path that never changes.
0:31:28 Because we’ve got a variety of different species that are interacting with each other, we actually
0:31:33 understand that now markets can be very different from even one day to the next depending on
0:31:35 who shows up.
0:31:39 And once you understand the kind of paradigm that biologists use to think about various
0:31:43 interactions in their domain, you can see that it applies almost directly.
0:31:44 It’s not even metaphorical.
0:31:50 It is actually literal that markets are biological adaptations to dealing with certain challenges
0:31:52 that we face every day.
0:31:57 And through that lens, all sorts of behaviors that we’re puzzling from a pure traditional
0:32:00 economic perspective become a lot more understandable.
0:32:06 Take for example, how stock markets have these incredible booms and busts.
0:32:10 We think that that’s irrational because, well, either you’re overvaluing stocks during the
0:32:15 boom or you’re dramatically undervaluing them during the bust.
0:32:19 But that doesn’t take into account the fact that at the early stages of innovation, nobody
0:32:22 knows whether or not that’s going to have any value.
0:32:23 Gene therapy is a good example.
0:32:27 The first gene therapy was a big leap of faith.
0:32:31 And in fact, even before that first gene therapy, there was an example in Philadelphia of a
0:32:35 gene therapy trial that went awry and a patient died.
0:32:36 The Jesse Gelsinger.
0:32:37 Exactly.
0:32:43 We need to develop an understanding about these kinds of therapeutics and technologies.
0:32:48 And in order for us to do that, we have to take these kinds of leaps of faith.
0:32:53 And as these leaps of faith turn out to work out, we become eventually over enthusiastic
0:32:56 and the market overshoots.
0:33:01 And once we realize that they’ve overshot, we then start to unwind and correct and the
0:33:03 market sometimes crashes.
0:33:08 So that kind of feast and famine, the boom and bust cycle, that may not be rational
0:33:14 from a strictly physical perspective, but from a biological perspective, it’s actually
0:33:15 expected.
0:33:17 Truly evolutionary landscape.
0:33:18 Red and tooth and claw.
0:33:19 Biology, economics.
0:33:20 Absolutely.
0:33:21 Thank you, professor.
0:33:21 Thank you.
0:33:31 [BLANK_AUDIO]

0:00:05 Hi and welcome to the A16Z podcast. Today’s episode is a special conversation with Lorraine
0:00:10 Powell Jobs, founder and president of the Emerson Collective, a firm she founded in 2004 to drive
0:00:15 social impact through investments across a broad range of areas such as education, immigration,
0:00:21 and the environment. Lorraine Powell Jobs is interviewed by A16Z co-founder Ben Horowitz,
0:00:25 and their discussion covers everything from Lorraine’s childhood in mountainous rural New
0:00:31 Jersey and how it shaped her to what the Emerson Collective is driven by and does and why it’s a
0:00:38 collective for that matter. For more about Lorraine Powell Jobs’ work, see www.emersoncollective.com.
0:00:43 This conversation originally took place at our most recent summit event in November 2018.
0:00:48 Welcome, everybody. I’m Ben. This is Lorraine. Coming in, I was just talking to Lorraine.
0:00:54 I was saying it’s really hard to introduce her because she’s not like our other co-investors
0:01:01 like at all in a good way, but she’s got a much more complex thing that she does and background.
0:01:05 So I could read the read. I could say, well, she’s a media mogul. She’s a tech investor.
0:01:11 She’s an education reformer and so forth. But you won’t even know really who she is if I just did
0:01:16 it like that. And none of them would be really honest. I don’t think anybody would call me a
0:01:22 media mogul except you. Or what you are a media mogul, technically. I’m so excited to be here,
0:01:28 to be having this conversation because there is nobody else like this. And so you’re in for a treat.
0:01:32 Why don’t we start at the beginning back in West Milford, New Jersey, where you grew up.
0:01:37 You didn’t have the whole Silver Spoon. No, we were solidly middle-class family.
0:01:46 Yeah. And what was that town like? West Milford, New Jersey is about 20 miles west of the George
0:01:52 Washington Bridge and then another 10 miles north. If people don’t know New Jersey, there is a
0:01:59 mountainous part of New Jersey. And in fact, one can go skiing and one can ice skate. And so we
0:02:04 did all of those things. We were just in the beginning of the mountainous parts. And so now
0:02:12 as an adult looking back, it’s lovely. It’s wooded. It’s wild. I grew up with three brothers who were
0:02:19 very wild. So we had sort of this connection to nature in the natural world that we would all
0:02:25 hope for our children. However, growing up there was also gritty. And it had a lot of
0:02:32 Jersey in it. Yeah, Jersey. Like, what is Jersey? It really did. It was kind of hard to scrap all.
0:02:41 People had big hearts, but also big edges. And so you learn really quickly, as a child,
0:02:48 where the boundaries are and which boundaries one shouldn’t cross. We were all of us
0:02:56 put to work really early, which was great. So we developed an extraordinary work ethic.
0:03:02 As kids, you know, they were in the end, my mom remarried, and then we had a mix of blended
0:03:08 family. My dad died when I was three in a playing crash. And so we ended up with six kids,
0:03:13 like the Brady Bunch, three girls and three boys. But in order to have controlled chaos,
0:03:21 we all had chores. And we had very set times for eating and sleeping. And I shared a room my whole
0:03:27 life. And I shared a bathroom. There were six of us that shared one bathroom. And if you can imagine
0:03:34 what it was like getting to school on time, it was a mess. That was sort of my childhood. I was
0:03:40 always trying to eke out a little bit of privacy. And so now I can empathize with those in social
0:03:48 media who would like to regain their privacy. Get out of my bathroom, Facebook. It’s a joke.
0:03:55 It’s a joke, people. Anyway, I think a lot of what shaped me from that experience in West
0:03:59 Melbourne, New Jersey, and then, you know, our big trip each year was going down the shore. We’d
0:04:07 rent a little house and drive down there and get terribly sunburned. We were probably five blocks
0:04:12 back from the ocean, but it was our favorite time. And then we’d drive back up in our station wagon
0:04:19 back up to Northern New Jersey. That sort of circumscribed our life. Chores. And then one
0:04:28 vacation and get sunburned. But we were steeped in core values of real dedication and a sense that
0:04:35 there was always a way out. And that was through education. And that was communicated early on to
0:04:45 me. And luckily, I loved books and I love school. And I think I sought out teachers and tried to
0:04:51 ingratiate myself and just, you know, find a little place where I could excel and where I could feel
0:04:57 that there was a reward for the work that also gave me joy, which was, you know, doing school.
0:05:02 Right. It wasn’t just a chore. It was work with the benefit. That’s right. That’s a great inspiration.
0:05:06 And you went to University of Penn which is like a super prestigious school.
0:05:12 I did. I did. Yeah. I was the first person in my high school to go to an Ivy League school.
0:05:18 West Milford Township is massive and stretched all the way up to New York State. Probably 20%
0:05:24 of our graduating class went on to any further schooling. Certainly not all four years. A lot
0:05:28 of them went on to trade school. Yeah. I was reading something that you said that I thought was
0:05:34 very interesting. So when you were a kid, I guess you donated like $20 to the Southern Poverty
0:05:43 Leadership Law Center. Yeah. And they would send you letters. And the thing that you were very focused
0:05:49 on was who got the opportunity and who didn’t. Yes. Because I was focused on it because they
0:05:57 were focused on it. Yeah. SPLC had a huge impact in my young life because I read about them in,
0:06:02 I think, the reader’s digest. I think that’s where I read about them. There was a profile on
0:06:09 Morris D’s. So I saved up some newspaper money and I sent them for me, which was a lot of money
0:06:15 at the time, $20. I think I was either in late middle school or high school. Yeah. I don’t know
0:06:20 if you remember. I don’t know if you remember when you’re a kid getting mail with such an exciting
0:06:26 thing. That’s the only place that ever sent me any mail, that in one of my grandmas. So they would
0:06:33 send me mail reliably, probably every quarter. Of course, it was a beautiful form letter, but it
0:06:40 always told a story of justice meeting injustice and opened my eyes to things that were going on
0:06:48 in the world in the early 70s that I otherwise wasn’t privy to. So I really hung on there everywhere.
0:06:54 I remember saving all the letters from Morris D’s in actually one of my most wonderful moments,
0:06:59 and it only just happened a couple of years ago at Emerson Collective. We do all of our
0:07:08 philanthropic giving anonymously, but we had been funding SPLC and we decided that we wanted to do
0:07:13 some teaching tolerance curriculum in partnership with them. And so one of our team members was
0:07:19 actually talking to Morris D’s who still works there. And so I said to her, “Can I get on the phone
0:07:25 with Morris D’s?” I got to tell him that he was the one. He planted that seed in me that individual
0:07:34 people could pay attention and engage and maybe do something about injustices that seem intractable
0:07:41 or far away or impenetrable. Oh, that’s a great story. And coming from your Big Bruce Springsteen
0:07:46 fan, and it’s really interesting that in his work, even though he’s Bruce Springsteen,
0:07:51 it’s still all about New Jersey. And do you feel that way about your work, where that
0:07:58 a lot of the inspiration is still from that time? A lot of inspiration. Yes, although I don’t find
0:08:06 that the work is necessary all about New Jersey, but a lot of how I see the world and how I think
0:08:14 about basic fairness comes from New Jersey. I think people actually want to know what’s going on.
0:08:20 And they have clearly held opinions. This is what I recall and work hard, but also are open
0:08:28 to hearing from other people because New Jersey was certainly a melting pot in the first wave
0:08:33 of immigrants from Europe. And so they had to accommodate. It’s not a glamorous place to be
0:08:40 from. And so people were distracted by glamour or coolness in any way. Lyrics of Bruce Springsteen
0:08:47 still echo through my head. It’s one of the soundtracks of my life, I think. So he pops up
0:08:53 when I do not expect him to pop up when I’m observing a situation. But to your point,
0:09:01 a lot of the work that I do references something that I learned a little over 20 years ago from
0:09:07 College Track. And I think it’s true to say that all of the work that we do at Emerson Collective
0:09:14 now is informed and certainly influenced and shaped by what was learned at College Track.
0:09:19 And so maybe College Track is my New Jersey. Yeah, interesting. And so tell us about
0:09:24 College Track and how what you learned, what you tried to do, and then what you learned,
0:09:30 and then how you moved it to be the entire Emerson Collective. Sure. It’s sort of a beautifully
0:09:35 long and winding story because it all took place probably starting about 22 years ago.
0:09:42 After graduate school, I was running a natural foods company. And per chance, I met the California
0:09:48 Teacher of the Year. She was teaching at a local high school. And I had not ever visited a high
0:09:53 school in California because I had moved just for graduate school. She asked me if I could come and
0:09:59 talk to her students, that all of her students in this one class were from East Palo Alto. And this
0:10:03 was at Carlemont High School, which is where all of the East Palo Alto students stood out. And that’s
0:10:07 when East Palo Alto was super dangerous, right? That’s right around the time of the murder capital.
0:10:13 Yeah, right around that time. I jumped at the opportunity because when I was at Stanford,
0:10:20 I found it really odd to have the juxtaposition of Palo Alto and East Palo Alto because I had moved
0:10:26 to Palo Alto from New York City, where all the demographics are right on top of each other. And
0:10:31 so you can have one block of rich and one block of poor, and people really mix. And it felt really
0:10:37 awful and awkward to have one-on-one dividing this socioeconomic group from this socioeconomic
0:10:41 group. So I said to her, “Yes, I’d love to. I’d love to come and talk to them.” She told me they
0:10:46 were seniors. It was September of their senior year, and they were all going to college. So I
0:10:50 said, “Well, what do you want me to say?” And she said, “Just tell them about what college is like,
0:10:54 and what it’s like to go to graduate school, and what it’s like to start your company.” So I said,
0:11:01 “Easy, okay.” So I went and I started talking to them about college and just wondering if they were
0:11:07 excited and where they had visited and who they knew. And I could tell I was starting to get some
0:11:12 blank dates, and I thought, “I’m losing my audience.” So I said, “Okay, guys, help me out here. How many
0:11:17 of you have visited a college? Maybe a few. How many of you know somebody that’s in college? Maybe
0:11:21 one or two. There were 35 students in this class.” And I said, “Okay, how many of you have already
0:11:26 taken your SATs?” And it was no one. And I said, “Okay, how many of you are going to take your SATs?
0:11:32 It’s already September, so you must be studying.” No one. And then I said, “Do you guys know that you
0:11:36 have to take your SATs and you have to do these standardized tests in order to go to college?”
0:11:41 And they said, “Well, we heard about it.” And the teacher said, “You know, we were going to talk
0:11:48 about this.” I thought, “Well, this is a bizarre lack of information to give to college-bound
0:11:54 seniors.” So I said, “Guys, how many of you have visited with the college counselor to talk this
0:12:03 through?” No one. As a result of Prop 13, California started cutting all nonessential portions of their
0:12:10 education system, which included arts and drama and PE and college counselors in addition to science
0:12:16 labs and higher mathematics. And it is amazing on that Proposition 13 was like in the late 70s.
0:12:25 Yeah. Exactly. And we were living with that legacy today. So I told the students I would come back
0:12:32 every Friday afternoon, and I would be their college counselor. And so I did that for the next
0:12:39 eight weeks. And what I discovered was of all the students, the 35 students, only two students
0:12:45 had actually taken the courses that they needed to take in order to apply to a four-year college.
0:12:54 So 33 students had missed a year of English or math passed Algebra I, or no one had taken a
0:13:02 lab site. No one ever told them what classes they needed to take. They were all graduating,
0:13:07 so they all had sufficient credits, but they didn’t have the credits that lined up
0:13:12 to apply to a four-year college. So for them, we got them to apply to a two-year college,
0:13:17 but it was such colossal waste, and it was just such an awareness that I had immediately that
0:13:24 this is actually a solvable problem. This is lack of access to the kind of information and guidance
0:13:30 that can be provided for all of these students. So I met with subsequently juniors and sophomores
0:13:35 and freshmen, and then I talked to parents at back to school night, and I just did this
0:13:40 as a volunteer because I thought this is something I can do. I can address this problem. I know this
0:13:46 stuff. But then there were layers of issues that I started to become aware of with a friend of mine,
0:13:53 Carlos Watson. We started visiting in the community. Yeah, exactly. He now runs Aussie. Just in the
0:14:00 communities, the families and understanding what happens actually when you’re first in your family.
0:14:04 What happens when you’re first in your family to graduate high school? What does that mean for
0:14:08 the information that you get from your family? What happens when you’re first to want to go to
0:14:13 college to apply and to thrive and to complete college? What happens when you’re first in your
0:14:17 family if you’re a college grad? Sometimes they resent you when you’re first. Many things. But
0:14:23 also to have that aspiration. You’re a leader in your family and you’re a problem solver in your
0:14:28 family. That’s a wonderful thing, but it also means you get sucked back into all problems because
0:14:34 you’re a good problem solver. And sometimes if you’re a child of immigrants and you’re the translator
0:14:40 for the family, the family relies on you for that. So there are all sorts of really interesting
0:14:43 issues that were just in this one community. And that’s when I decided to start
0:14:51 College Track to do a more holistic support for students and families so that we could
0:14:58 seriously prepare students for college and make sure that they persisted and completed college
0:15:04 and work with their families to solve problems within the family so that the individual didn’t
0:15:09 have to be the problem solver. And so that’s how we started College Track and we started it with
0:15:18 25 high school freshmen at Kalman High School and I would go to the lunch times and I’d find the
0:15:22 ring leaders because I had a sense that everyone has to come with a friend because that would be
0:15:27 a reinforcing mechanism. And some people came with whole groups of friends and that was even better
0:15:33 because they would hold each other accountable. So we built out a cohort and so they were responsible
0:15:38 for and to each other as well as to the adult. So we started in East Palo Alto this year.
0:15:45 We have grown. We have 3,000 high school students. We have 1,000 college students and we have 550
0:15:52 college grads. Wow, that’s amazing. I’m listening to you and it’s so different than normally in
0:15:57 Silicon Valley. People go to a fancy dinner. Somebody gives a presentation. They put some
0:16:03 money on it. Here you are. You go and you go. I want to go teach this class and then from there
0:16:08 build it all the way out into College Track. It seems like your philosophy. I think that’s
0:16:14 probably how entrepreneurs work, right? You see a problem and you actually think this is something
0:16:19 that I can bring some energy and some problem solving and some smarts and maybe a little bit
0:16:26 of innovation to this problem, to this issue. Really, really amazing. So then take us from that to
0:16:33 Emerson Collective, which is like the world like now you’ve really expanded your horizons.
0:16:39 And we’ll maybe start with the name because the name is so unusual. Emerson, which I know from
0:16:43 reading about it is Ralph Waldo Emerson and then tell me about that and then also about
0:16:48 Collective because like there’s no other like VCs we work with called Collective.
0:17:02 In Brooklyn, there’s the cheese board collective. That’s funny. So I’ll tell you what we’re not.
0:17:06 We’re not the cheese board collective. We are not a group governance organization,
0:17:15 which actually makes it much better. We’re more of a collective of leaders and innovators
0:17:21 across different sectors. And we see the connectedness of every issue. So just in the simple
0:17:30 college track story, we talked about educational inequities and access and the need for enhanced
0:17:39 and robust curriculum, but also good immigration laws and fair immigration practices so that that
0:17:46 actually could be a foundation instead of because that’s essential to the education agenda.
0:17:52 Of course, of course. And so many of our first generation college bound students
0:17:58 are recent immigrants to America. And certainly in California, many of our recent immigrants
0:18:04 are undocumented. And so understanding what it means to be undocumented as a child growing
0:18:09 up in the United States, you cannot access any state or federal funding for your education.
0:18:14 And so you might be the valedictorian of your high school and you can’t get state or federal
0:18:19 funding. And in fact, there are only a handful of states that would allow you to pay in state
0:18:25 tuition. But for undocumented families, even in state tuition at $17,000 a year, when you can
0:18:31 get generally no scholarships, no loans to fund it, that’s out of reach too. So that’s another
0:18:37 issue that we learned about firsthand environmental toxicity in low income neighborhoods. So the
0:18:42 access to clean air, water and soil and access to food and access to financial services and
0:18:48 banking services, all of these things you learn about in one community. And they’re all connected
0:18:54 because these are all systems and experiences that touch the individual’s life. And so you pull one
0:18:59 thread, the whole fabric follows. We were pulling one thread and we made it a really tough thread
0:19:07 around this educational support in our own way. But we understood we can’t just solve
0:19:14 the education issue without looking at the holistic issues of what actually are the,
0:19:20 what are all the touch points to an individual’s life? And how do we make sure if we can help
0:19:27 remove obstacles in all of these different systems or even better redesign these systems
0:19:34 so that an individual can just have access to the opportunity that they’re qualified for,
0:19:40 that will be the whole work. So that’s why it’s a collective because we actually collectively wrap
0:19:47 around problems and understand that complex problems require complex problem solving and
0:19:52 complex solutions. Yeah, really complex. So it’s interesting, Arnie Duncan had a quote where he
0:19:58 said, “Well, you know, Lorraine said to me, you’re trying to solve these really intractable
0:20:03 problems, so why don’t you let me help?” And he said, “You know, I think she was attracted to the
0:20:07 degree of difficulty.” It was like the harder it was, like that’s what she wanted to do. Is that
0:20:18 true? Yeah, I mean, for me, it’s so exciting and invigorating to think about devoting my life
0:20:24 to solving the problems of our day. You know, these are intractable problems. They’re not
0:20:28 problems that no one has worked on before, quite the opposite. These are problems that people have
0:20:34 worked on. And now they pass the baton to us. We have the great privilege of trying to make a dent
0:20:39 in them and trying to maybe redesign the system a little more elegantly and make things a little
0:20:44 more fair and more just. And we’ll work on them for 20, 30 years. And then someone else is going
0:20:49 to come up and we get to pass the baton to them. So that’s how we think about it. And it’s joyful
0:20:58 work. If you can, at one point, be part of that inflection point in another person’s life. If
0:21:04 you can, for one person, know that for you, that person has opportunities they otherwise wouldn’t
0:21:10 have had, it’s the most intoxicating feeling. And you want to do it again and again and again.
0:21:14 So you do it, you try to change the world, but you do it for the individual.
0:21:19 So the intractable-ness of the world doesn’t bother you at all. Yes. That’s how I can think about
0:21:25 it. And that’s how I can actually, and I think for all of us, we understand where we’re trying to go
0:21:30 and then we backwards map it. So most everything we’re working on, we have a 10-year time horizon.
0:21:36 Some things we have a 10-year and then we’ll renew it for another 10 years. For Arnie, he’s hoping
0:21:43 to put a dent in gun violence and specifically in the number of homicides that are gun-related
0:21:49 deaths in Chicago in 14 specific neighborhoods. And he wants to see impact within the next few years.
0:21:58 Yeah. When you go after education, immigration, gun violence, environmental issues, you end up with a
0:22:05 pretty unusual kind of diverse team. So you’ve got Mark Echo, Arnie Duncan, Steve McDermott,
0:22:11 my old friend Steve McDermott. Did you put all those? I can’t even imagine all those three in a
0:22:18 meeting. That just seems so wild. How does that work? How do you? Every Monday, including today,
0:22:24 we have all staff meetings. And because we have five different offices around the country, so we
0:22:30 have on the screen, like Hollywood Squares, generally a three-by-three matrix with different
0:22:37 teams populating in the nine squares. And we have different teams report out. Sometimes we have
0:22:42 guests who come and speak who are either in the philanthropic portfolio or the for-profit portfolio
0:22:50 or who are policy advocates or policy writers or just brilliant people that happen to be passing
0:22:57 through one of our cities that we bring to the table to listen to and learn from. And then we have
0:23:03 follow-up. So it’s sort of like every single Monday, we set the stage and we know where we are,
0:23:07 we give a report back and look forward and then we keep going. And then you want everybody out
0:23:12 in the field? Then you want everybody out in the field. Yeah. If people sat in an office more than
0:23:17 five, six days in a row, that would be bad. And that’s because all the knowledge is out in the field?
0:23:24 Yeah, exactly. And so when we decided to bring a big idea forward in education, for example,
0:23:31 with the ExQ Institute, Ruslan Alley, who was running our education practice at the time,
0:23:36 came out of the Obama administration. She was the assistant secretary of education. She ran
0:23:42 the office of civil rights. I love that we have these extraordinary people who have a body of work
0:23:46 and a network and there’s something big that they want to accomplish that maybe they didn’t get to
0:23:53 do in government or in business or in the social sector. And so we say, okay, come, do you think
0:23:58 we can do this in 10 years? And what does it look like? What kind of team do you need? And so we
0:24:07 build out like that. So we’ve built our organization from these individuals who come in this extraordinary
0:24:13 way. And then the newer hires, the more junior people hear about the fact that they might get
0:24:17 to work with some of these amazing people. And so it’s become kind of this really nice
0:24:27 giant magnet for talent in that way. I was telling you about Ruslan and ExQ. So we both
0:24:32 had worked in the Ed sector for over a decade and we both had a sense of the brokenness of the
0:24:39 design of the system and the fact that students who come to school needing the most receive the least
0:24:47 in all educational resources. And you just have to wrap your head around that. Students who come in
0:24:55 who are already behind in hearing just the number of words, let alone the quality of words
0:25:03 at home actually are behind in kindergarten and they rarely catch up. And the students who are
0:25:08 slow to learn to read by the end of the third grade, every school in this country shifts from
0:25:13 learning to read to reading to learn. So then when you enter the fourth grade, if you’re not a really
0:25:19 good reader in the third grade, all of your learning comes from reading text. And so you’re
0:25:26 further and further behind. And whoever came up with the idea that humans learn best by sitting
0:25:31 still for six to eight hours. Especially little kids. Yeah, especially little kids. That’s not how
0:25:38 synapses are formed. We came with all of this frustration about how we’re just ruining the
0:25:44 humans and the massive potential that’s in every single person’s skull. You know, the brains that
0:25:53 we carry around can solve any problem, any person. The talent and the IQ that is randomly distributed
0:26:00 does not meet up with opportunity. Opportunity is siloed. So we decided, well, we want to flip the
0:26:07 system. We want to flip the system from measuring learning in the high school. So we’re both also
0:26:11 fixated on the high school as a fulcrum for what happens after high school and what happens before
0:26:19 high school. So K8 shifts when high school shifts and obviously access to higher ed and career
0:26:25 also is influenced by what happens in high school. So if you change high school, you can actually
0:26:33 change the whole system. But you get a high school degree by sitting through 120 hours of a list of
0:26:39 subjects, mathematics and English and some history and some science. And the reason for that is because
0:26:48 in 1906. 1906. I’m sorry. Right. Good thing there’s been no technological change since 1906.
0:26:56 1906. And if you walked into high school in 1906, you’d feel, if you time traveled to 2018, you’d
0:27:03 feel very much at home. It’s still the same. Still the same life. Awesome. In 1906, they decided that
0:27:09 what they wanted to do was systematize. Great for Rip Van Winkle. Yeah, that’s right. That would be a
0:27:16 good movie. What they wanted to do was bring the learnings of industrial revolution and productivity
0:27:23 into the school setting. It was actually a really good idea that we would standardize schools because
0:27:31 schools at that point had only been generally for young men and they wanted to have universal access
0:27:37 to high school. And they also wanted to standardize the lessons across the country. So this was smart
0:27:44 and this was innovative for its time in 1906. Unfortunately, that system that got set up
0:27:49 is still the system today. That’s still how you get your high school diploma. Not to mention
0:27:55 the fact that you actually don’t need a proxy for learning. You don’t need for time to be the
0:28:00 standard. You actually need content mastery to be the standard and time should be the variable.
0:28:07 And we can do that. We actually know how to measure what you know about anything at any time
0:28:14 in any classroom. We also know a lot more about neuroscience now and we know how the brain develops
0:28:22 when it’s actually engaged in a task and we also know that you don’t learn things in silos, isolated
0:28:27 silos. You don’t learn math and then leave math and then start to learning this and leave it. It’s
0:28:34 actually much more robust and sticky when things are integrated and connected. And so we decided
0:28:42 that we wanted to change high schools in America and we started it a few years ago and we started
0:28:47 it with a competition across America and we wanted to put students in the center. And so we
0:28:53 crisscross the United States. I think we’re on our fifth trip across the United States and we visit
0:29:02 communities and we hold student roundtables and we hold civic leader and business leader and parents
0:29:08 and teacher roundtables and we sit and we talk to people and we listen to them. So then by the time
0:29:13 we issued this challenge for communities across America to redesign high schools in their own
0:29:18 community that mapped on to the workforce demands in their communities, that repurposed assets in
0:29:23 their communities, we had already talked to thousands and thousands of people. And they were
0:29:28 able to pick this up and there were no who you always hear about all they do and there are so many
0:29:32 inhibitors in terms of changing how things are done and you know between the you know structure and
0:29:37 the unions and this and then that but people were able to pick it up and redesign these high schools.
0:29:44 We weren’t sure. We weren’t sure. We definitely were stepping into unknown territory. We weren’t
0:29:50 sure how many people would actually take this up. We had designed the we used a design thinking
0:29:58 set of modules. We had 13 modules that every team had to go through. We made these kits with posters
0:30:04 and workbooks and cards that you could use and you had to have on your team students, parents,
0:30:11 business leaders, designers of any sort and even in small communities. In the end,
0:30:18 it was a seven month process to go through this whole thing. So communities really had to dig in.
0:30:28 We had 700 full applications for brand new high schools in all 50 states over 10,000 people participated
0:30:34 for what we’re going to be 10 schools that we funded and built. Oh wow. Yeah. It was amazing.
0:30:43 And after the competition, so we awarded our 10 and then after our competition, we followed
0:30:49 about 140 that we’re continuing on and building their own even though they didn’t win. Oh wow.
0:30:57 And since then, we’ve awarded another nine the XQ Super School moniker because their models
0:31:03 are so inspiring and breakthrough. And so they’re part of this cohort of super schools and we bring
0:31:10 them together. And so now 17 of the 19 are open. The other two will open next August. And of course,
0:31:16 along the way, we’re learning all sorts of wonderful things and they’re teaching each other and they
0:31:20 have professional learning communities. And of course, they have to break from the Carnegie
0:31:26 unit, which is time as a proxy for learning. So they all have to have competency based learnings.
0:31:34 We have a standardized learner outcome. And so all of the students in all of the schools can tell
0:31:39 you the XQ learner outcomes are about being a synthesizer and a collaborator and a creative
0:31:46 problem solver. They happen to know that for them 65% of the jobs that they will hold haven’t been
0:31:52 created yet. And so they understand they have to be agile nimble thinkers. They have to be creative
0:31:57 problem solvers. They have to understand critical thinking skills. And these are schools that are
0:32:02 not skimming. These are schools that exist in communities and they’re open enrollment public
0:32:07 schools. Yeah, so with the kids who actually need to help. But they actually bring the answers.
0:32:12 They bring the answers and the teachers are heroically scrambling to catch up and design
0:32:18 new curriculum. And it’s really exciting journey to be on. Because you have Arne Duncan on the team.
0:32:25 How do you think about taking that those models and changing policy and, you know, kind of
0:32:33 We do think that. So in many of the sectors where we work, policy is the tip of the spear. So certainly
0:32:40 for immigration reform policies at the tip of the spear and there’s work that we can do, each of us
0:32:46 can do to be welcoming to immigrants. And there are individual policies that municipalities can
0:32:54 instill around drivers licenses and not using local law enforcement to enforce immigration laws.
0:33:00 On the other hand, with education, it’s very, very layered. And so we do have a policy team
0:33:08 that is mapping out specific state and federal policies that we advocate for. So part of the
0:33:16 beauty of being an LLC or a series of LLCs is that we can be policy writers, policy advocates,
0:33:22 right? We can do whatever you want. We can be investors and we do do all those things. We use
0:33:27 every possible tool that we missed out on some tax breaks. That’s right. That’s right. So if you
0:33:32 don’t care about tax preference, there’s a whole heck of a lot that you can do. A lot more flexibility.
0:33:38 You get ultimate flexibility. That’s really great. Yeah. When we were really digging in on the work
0:33:45 about six years ago, I was contemplating what structure we should have. And most people start a
0:33:52 foundation so that they migrate, pretext either any dollars or stock into that generally, whatever
0:34:00 asset they’re going to be using, and then use that foundation construct. They use kind of the
0:34:06 5% payout to do the work that they’re going to do. Well, first of all, I felt that if I really
0:34:12 care about impact, if I actually really care about solving problems and I don’t care about
0:34:20 increasing wealth, then I would be foolish to close off any avenues by which change happens.
0:34:25 And a lot of change happens in brilliant for-profit companies. So I wouldn’t want to close that off.
0:34:30 And a lot of change happens at the policy level. And so we wouldn’t want to close that off.
0:34:35 So then so I thought, well, why doesn’t everybody do this? I don’t understand money.
0:34:39 But if people have taken the giving pledge and they plan to give away all their money,
0:34:45 why does they even care about a tax break? I still actually don’t understand it because
0:34:51 I’m hoping we go through as much of the wealth as we possibly can. I mean, that’s the purpose of it.
0:34:59 And we’re living in times of urgency and extreme crisis. I don’t understand what the 95% of the
0:35:03 corpus is waiting around for because it’s not going to get worse than we’re in right now.
0:35:06 So I hope people put more work, more money to work.
0:35:10 Let’s talk a little bit about profit because you invest in tech companies.
0:35:15 And there are a lot of tech companies here who I’m sure, and a lot of them ask me,
0:35:20 how do I get money from Amherst and Collective? What are you looking for on the for-profit side?
0:35:24 Almost all of our investments are mission aligned. We have an environmental practice.
0:35:31 So within the environmental practice, there’s a robust portfolio. We have an ed tech portfolio
0:35:37 that’s obviously aligned with our education practice. We have cancer companies that we’ve
0:35:42 invested in because we invest in oncology research and some policy there as well.
0:35:48 We started in immigration incubator. There isn’t a lot of technology that’s migrated into the
0:35:54 immigration sector. So we’d love to encourage that, but we would love to invest in people who
0:36:00 are bringing differentiated thinking or new thinking to old problems in this way.
0:36:08 And so there’s a lot of wonderful opportunity for entrepreneurs to marry their passion and
0:36:13 their purpose with their company. Those are the entrepreneurs that we get super excited by.
0:36:20 So one last question, and you’ve kind of answered this, but there’s another answer that I’m looking
0:36:25 for. So how do you know when you’re succeeding? On the micro level, I definitely get it and you’re
0:36:31 changing lives. But on these big agendas that you have, how do you know when you’re kind of on the
0:36:37 track and getting there? Yeah, we collect data on everything that we do so we can understand
0:36:44 if we’re trending in the right direction. In education around XQ, I’ll just use that as an
0:36:51 example because I’ve been talking about that. In addition to the schools, we now have the
0:37:00 district of Tulsa and the state of Rhode Island, which want to have kind of a complete redesign
0:37:04 of all of their high schools. So in Rhode Island, they have 45 high schools. So you can do experiments
0:37:10 in Rhode Island or at least use it as a laboratory for other states. So all 45 of their high schools,
0:37:14 they want to become XQ super schools. So we’re working with them on a statewide competition
0:37:22 in that way. So that’s moving in the right direction. In Chicago, there’s very good data on
0:37:31 both fatal and non-fatal gun violence. And so we have really good metrics there. But we also see
0:37:37 success in smaller ways, in anecdotal ways, which I think are very powerful as well,
0:37:44 where people come back to us and they tell us no one has ever talked to us before. No one’s
0:37:49 ever taken a shot on me. No one’s actually ever listened to me and then given me the chance
0:37:56 to try a big hairy idea. And so that to me is also success. And that’s moving things forward.
0:38:02 I think there are also other more subtle ways. We didn’t even talk about, you know, how do you make
0:38:07 sure that you’re part of the cultural narrative and how do you make sure that some of our most
0:38:13 imperiled and important institutions, like the media, like high quality journalism, are supported
0:38:19 and sustained. But seeing how many people are going into those disciplines is actually another
0:38:25 measurement of success. Seeing where IQ is migrating is really important measure of success.
0:38:28 We see that in the ed sector. We’re seeing that in other sectors as well.
0:38:33 That’s great. Well, thank you so well. We all appreciate you being you and fixing the world.
0:38:40 So everybody, please join me in thanking Maureen. Thank you so much.

0:00:05 The content here is for informational purposes only, should not be taken as legal business
0:00:10 tax or investment advice or be used to evaluate any investment or security and is not directed
0:00:14 at any investors or potential investors in any A16Z fund.
0:00:19 For more details, please see a16z.com/disclosures.
0:00:22 Hi everyone, welcome to the A6NZ podcast.
0:00:26 I’m Sonal and I’m here today with David Yulovich, one of our new general partners who covers
0:00:30 all things enterprise, but honestly, that can mean so many different things to so many
0:00:31 different people.
0:00:36 So we briefly discussed what enterprise products really mean today for entrepreneurs, companies
0:00:42 and users, especially given the latest shifts driving SaaS beyond the cliche of consumerization
0:00:44 of the enterprise.
0:00:48 We also cover specific advice on the topics of pricing and packaging, how to balance being
0:00:53 a product visionary with being a product manager, when and how to scale out and hire
0:00:57 your leadership team, and how do you know that’s working or not, plus how to best manage
0:01:02 your own time and your own psychology as a leader while doing all this.
0:01:06 For context, David founded OpenDNS, where he actually went through a rough period in
0:01:12 going from CEO to CTO in 2009 and then back to CEO again, in a company that itself pivoted
0:01:14 from consumer to enterprise.
0:01:18 We discuss how did he make a comeback, I mean it’s not like he changed instantly overnight,
0:01:20 so what and how were the lessons learned?
0:01:25 In 2015, Cisco later acquired OpenDNS and David ran their security business where he
0:01:27 also led the acquisition of three companies before coming here.
0:01:31 So he’s seen startups from all sides, from being acquired to being the acquirer, from
0:01:36 small to big to small to big again, from on the inside and from on the outside.
0:01:40 But the real theme of this episode is the journey many founders make, from technical
0:01:42 to product to sales CEO.
0:01:47 And while we end with the story of OpenDNS and the most important lesson learned there,
0:01:51 we begin with what is the one piece of advice David has for founders?
0:01:55 So I think that there’s not usually like one, I’m not a big fan of the platitudes where
0:01:59 you just like say one thing that applies to everyone, because there’s never one thing
0:02:02 that makes the difference between success and failure.
0:02:05 As a founder, generally you’re at different stages in the company building journey.
0:02:08 Sometimes you’re a technical CEO trying to build a product to make sure it actually
0:02:09 is feasible.
0:02:13 Then you’re constantly in the market with customers doing customer discovery, making
0:02:17 sure that you are solving the problems you’re trying to solve, you end up becoming sort
0:02:19 of a product CEO, making sure that you have product market fit.
0:02:22 Then you end up becoming a sales CEO on the enterprise side, where you’re trying to generate
0:02:26 revenue and figure out how you can go acquire more customers.
0:02:29 And then if that works out, you become this really general sort of manager, go to market
0:02:32 CEO, and you’re thinking about how do you scale and accelerate a business.
0:02:37 And so I like to think about those different journeys as where there’s like the right decisions
0:02:41 for the right time, and really trying to help founders understand like what time is it and
0:02:43 where are they in that journey and where do they want to go.
0:02:47 Is it possible to be all four at once, or is it really tied to the stage of the company?
0:02:50 So generally you want to be more focused than less focused, and the reason I think you’re
0:02:55 probably also not all those things at the same time is like a company is a set to me
0:02:59 of like these like interconnected little knobs, and you can never just like optimize one and
0:03:01 then forget it and not come back to it.
0:03:04 You end up going to another knob, like if you fix pricing and packaging, then you’re
0:03:05 going to control panel.
0:03:06 Yeah, it’s like a control panel.
0:03:08 Then you’re going to move to demand gen to try to like increase the top of the funnel.
0:03:11 And if you fix that, then you want to make sure that your SDRs and salespeople are like
0:03:15 making sure they’re converting all the marketing leads into qualified leads, then you’re closing,
0:03:16 then you’re doing customer success.
0:03:20 And like once you tighten one of those knobs, it just creates slack in one of the other
0:03:21 knobs.
0:03:24 So you might switch those hats from time to time, but I think you’re rarely going to
0:03:26 be wearing more than one of those hats at the same time.
0:03:30 So I have to ask this then, since we’re going with this theme of let’s not do the platitudes.
0:03:33 A lot of people say it’s about the customer and the customer journey and understanding
0:03:34 the customer.
0:03:37 Honestly, when I hear that, I’m like, they can pull you in a million different directions.
0:03:39 You don’t even know what to do, especially if you’re a technical founder.
0:03:41 You don’t know to sell the Fortune 500.
0:03:43 Like there’s a bit of a chicken egg.
0:03:47 So how do you sort of figure out how to sell to the right customer?
0:03:50 It’s okay for a company in the early stages not to know exactly who they want to go after,
0:03:54 but they do have to understand the consequences of the customers that they’re targeting.
0:03:59 I think we’re living today in one of the best times to be an enterprise software startup.
0:04:04 And to me, one of the reasons is that because so many companies today are SaaS and subscription
0:04:08 software companies where there’s a recurring revenue component, it’s better for the customer
0:04:11 because they know that the customer experience is going to be good or they’ll stop paying
0:04:12 for the subscription.
0:04:13 It’s a repeat business.
0:04:14 It’s not a runtime sale.
0:04:15 That’s a repeat business.
0:04:16 Right.
0:04:17 I never think about the first sale when I look at a business.
0:04:19 I always think about like, what are you going to do in year two, year three to make sure
0:04:21 you renew the account, to grow the account?
0:04:25 It’s actually, it’s a way of peeling back the onion to figure out like how confident
0:04:27 are you and where your product is today?
0:04:30 Because if you say, oh, we’re only doing three-year contracts, well, is that because
0:04:34 it’s really hard to implement and tough to get customers onboarded until you need runway
0:04:35 to get them happy?
0:04:38 Or is it because you think you’re overselling your capability set and you just don’t want
0:04:41 the customer to figure it out within a year and not renew?
0:04:46 But if the startup I’m talking to says, oh, well, we did a couple three-year contracts,
0:04:49 but we realized that we were priced really low.
0:04:52 And so now if a customer wants a three-year contract, we’re actually going to charge them
0:04:53 more in the out years.
0:04:56 Well, that tells me your product is really good and getting better.
0:04:57 That’s fascinating.
0:05:00 Another way to think about that is that what used to just be like a product experience is
0:05:02 now much more of a customer lifecycle experience.
0:05:07 It starts before you sell, building a Vangelis, then there’s an onboarding part, then there’s
0:05:09 making sure the customer is really happy.
0:05:11 How do you market to your existing customers to make sure they’re getting full utilization
0:05:12 of your product?
0:05:17 And so that customer lifecycle makes it much easier as a startup is getting started to
0:05:21 start to really identify who is the target customer and then thinking about does that
0:05:23 actually map to the business I want to build?
0:05:26 Is it the big Fortune 500, the Global 5000?
0:05:27 Is it SMBs?
0:05:29 Because it has all these downstream effects.
0:05:32 So a lot of startups will come in here and in my first six months here, I’ve now met
0:05:37 with over 200 companies and a lot of them, they really have this ambition to go after
0:05:38 SMBs.
0:05:42 And one of the cool things about SaaS is that SaaS can take something that in the olden
0:05:46 days of enterprise computing, you’d have to buy the biggest server, the biggest box
0:05:47 to get the best solution.
0:05:51 But with cloud and with SaaS applications, you can now have the power to get this massively
0:05:54 great CRM system or this massively great HR system.
0:05:57 I actually like to think about it as SaaS is very democratizing.
0:05:58 SaaS is totally democratizing.
0:06:02 Because it enables smaller and medium-sized companies to have the accesses to big company
0:06:03 resources.
0:06:07 They don’t have the in-house engineers, but they can essentially as a service it into
0:06:08 their company.
0:06:09 Absolutely.
0:06:10 Small companies have unlimited compute.
0:06:11 They have unlimited storage.
0:06:13 They have unlimited bandwidth now.
0:06:17 And so when I meet with startups, they often want to like, ambitiously and altruistically
0:06:21 want to go satisfy this pain for SMBs, but it turns out that the reality is if you want
0:06:26 to charge a high price point, if you want to pay an expensive sales force, then you’re
0:06:28 going to realize that your average sales sizes have to be higher.
0:06:32 If I really want to go after a target market where the price point is going to be lower,
0:06:36 then I have to think about bottoms-up sales, about self-serve offerings, because I might
0:06:40 not be able to afford to have a sales force or a huge customer success engine.
0:06:43 And so I love when I see startups that think not just about who they want to go after,
0:06:46 but then they build that into their whole sort of customer experience model and like
0:06:50 marketing programs, pricing and packaging, renewals, sales, and the whole business model.
0:06:55 I mean, nowadays, think about how many emails you get where it reminds you of a new feature
0:06:57 that you may not have even known existed.
0:07:01 For instance, like I’m using an email product called Superhuman, and every week I basically
0:07:05 had an email from the team saying, “Did you know that you could use this functionality?
0:07:09 If you press Apple I, it’ll automatically route someone to BCC in their reply, or you
0:07:11 just press Apple C, it’ll copy the whole email.
0:07:12 You don’t have to select it first.”
0:07:16 Actually, I saw this awesome tweet from Patrick Cullis and the CEO of Stripe where he said,
0:07:20 “I feel like Rahul Vora, the CEO of Superhuman, is essentially inventing new user experience
0:07:23 interaction paradigms that will eventually cascade into other products, much like Steve
0:07:27 Jobs did with letting us learn new behaviors, like how to touch a phone.”
0:07:32 Patrick’s tweet was right on, and Rahul’s weekly marketing email to existing users,
0:07:34 it helps teach me these new things that they’ve unlocked.
0:07:35 They become very intuitive.
0:07:37 You still have to learn about them.
0:07:40 One of the best things about building a company today is it’s easier than ever to get close
0:07:45 to customers, to get constantly get iterative real-time feedback, both from an analytical
0:07:49 standpoint and just from customer surveys, MPS scores, all these kinds of things.
0:07:53 They have all this telemetry through SaaS products where you actually see how people
0:07:54 use the product.
0:07:58 But the second thing that’s happened is people talk about this bottoms-up SaaS motion, but
0:07:59 it’s not always just that.
0:08:02 It’s really about making sure they understand that there are evangelists in the company
0:08:06 that you have to win over before you’re going to get signed off from the CFO, the CIO, the
0:08:07 CISO.
0:08:10 Whoever he or she is that makes that decision, you’re going to have to get some champions
0:08:12 underneath that person to be your evangelist internally.
0:08:16 So this is a little counterintuitive too though, because the other piece of advice I’ve often
0:08:20 heard from folks is that the number one mistake a lot of consumerization of the enterprise
0:08:24 type of founders make is that they go too heavy on bottom-up to the point of ignoring
0:08:27 the importance of top-down sales.
0:08:28 What’s your view on that?
0:08:29 I like to frame it a little bit differently.
0:08:33 When people talk about the consumerization of enterprise, enterprise customers today
0:08:36 are being bombarded by so many different vendors.
0:08:39 Their attention span is so limited that the products today, like I think when people say
0:08:41 consumer, they mean easy.
0:08:43 They don’t really mean consumer.
0:08:48 The value proposition that I’m hearing when I talk to customers is that the time to value
0:08:49 needs to be short.
0:08:50 There’s actually two parts.
0:08:54 The first is I want value almost immediately, sometimes even before I pay for it.
0:08:57 I want a trial where I want to get up and running on my own, and then I’ll talk to a
0:08:58 salesperson.
0:09:03 So the time to value can either be like T minus zero days, like negative time, or it
0:09:04 has to be very short from hours to days.
0:09:09 And so I always encourage founders, think about the first hour, the first day experience,
0:09:11 the first week experience, the first month experience.
0:09:16 The second part is that enterprise software can get quite complex.
0:09:19 And so Zoom is in the news because they went public.
0:09:23 And Zoom’s a great example of something where they earn the right to be more complicated.
0:09:24 Wait, let’s pause on that for a minute.
0:09:27 Earn the right to be more complicated.
0:09:29 So this goes hand in hand with a short time to value.
0:09:33 So the short time to value gets you in the door, but we know that you and I could download
0:09:37 Zoom on our phones and be in a video conference call, and now we’re like, well, wait a minute.
0:09:39 Maybe all of our conference rooms should have Zoom.
0:09:42 Maybe we should integrate Zoom with our Google Calendar, G Suite.
0:09:45 You’ve earned the right to do that complexity because you’ve already proven so much value.
0:09:49 And not only that, the value you get by doing the integration with G Suite or by adding some
0:09:53 cameras to your conference room so that you can have room-based Zoom rooms, that complexity
0:09:56 is commensurate with the value you’re getting.
0:09:59 And so when people say, oh, consumerization of enterprise just means it has to be easy
0:10:02 or simple, that’s not quite what it is.
0:10:03 To me, it’s two things.
0:10:07 It’s a short time to value, and then the complexity curve is commensurate with the value proposition.
0:10:11 So then I want to ask you more about what needs to go into that time to value.
0:10:13 So let’s be a little bit more specific.
0:10:16 I mean, I get the point that you’re talking about, it’s incredibly competitive.
0:10:19 So you’ve got to differentiate fast and show the value.
0:10:20 But what are the things that drive that?
0:10:26 Is it a great, like a cute little Jiffy that jumps out at you and makes cute, like a clippy
0:10:27 type of thing?
0:10:28 I mean, what is it?
0:10:29 Wait, did you say a Jiffy?
0:10:30 Like a Giff?
0:10:31 Are we going to rumble?
0:10:32 Wait, I don’t know if we can do this.
0:10:33 Are we going to just stop?
0:10:35 I can’t tell if you’re trolling me or not.
0:10:36 We can’t be.
0:10:37 I am not trolling you.
0:10:38 I don’t believe.
0:10:40 I’m one of the people who calls Jiff’s not Giff’s.
0:10:41 I hate that.
0:10:43 You know there’s a world of people that think they should be J.
0:10:47 I don’t know if I would have ever agreed to this podcast if I knew you called him Jiff.
0:10:48 It’s like nail scratching on a chalkboard.
0:10:49 You’re literally right now in my ears.
0:10:51 It’s like someone’s poking pins in it right now.
0:10:54 How have you been so successful in your career this whole time calling him Jiff?
0:10:57 I actually feel like I kind of hate you right now, to be honest.
0:10:58 This is amazing.
0:10:59 Oh my goodness.
0:11:01 You don’t have any friends that pronounce Jiff Jiff.
0:11:02 I don’t think so.
0:11:03 You’re it.
0:11:04 Chris Dixon, is he a friend of yours?
0:11:05 He is.
0:11:07 I’m just shouting him on the podcast because I’m not going to go down on this thing.
0:11:08 I’m not going down on this ship alone.
0:11:11 We might have to edit that part out because I don’t know if that can be out there.
0:11:16 There’s like this dissonance in my brain because you and him are so smart but you also call
0:11:17 it a Jiff.
0:11:20 Like I don’t know what to do right now.
0:11:21 All right.
0:11:22 Okay.
0:11:24 So Slack for instance, they did a lot of really creative things.
0:11:28 I remember I was at Wired and the product that we use was Hip Chat.
0:11:31 And the thing that kind of eventually got me into Slack was the fact that you could do
0:11:34 all these like Jiffs, whatever.
0:11:38 You could do kind of more fun things and even I know that sounds really…
0:11:40 And they had integrations.
0:11:44 Other things could drive information into your Slack channel.
0:11:46 And that was not something that Hip Chat had for a long time.
0:11:47 That’s right.
0:11:48 Like Google documents and…
0:11:49 That’s right.
0:11:50 Dropbox files.
0:11:51 Exactly.
0:11:52 Or even automated updates.
0:11:55 Like if you’re a developer, when somebody would do a push to production, it could notify
0:11:57 people inside the Slack channel.
0:11:58 Right.
0:12:00 But now that’s not a case where IT has to decide the integration.
0:12:01 That’s right.
0:12:02 And they made it easy for individual users.
0:12:06 As long as you could use like Google Auth to authenticate, anybody could basically set
0:12:08 up a Slack channel inside their organization.
0:12:10 After a while, IT says, “Hey, wait a minute.
0:12:13 We have all these teams that are chatting on this thing.
0:12:14 They’re doing integrations.
0:12:15 Files are being shared.
0:12:18 We need to have a little bit more visibility, a little bit more access control.
0:12:22 And even for like security and compliance reasons, it became an enterprise sale that
0:12:23 went wall to wall.
0:12:25 It’s now already entrenched in the organization.
0:12:28 There’s already integrations happening with some of the developer tools and workflows.
0:12:30 And at that point, they were in the right to be more complicated.
0:12:32 I’ve noticed this resurgence.
0:12:36 And I don’t know if it’s just like a zeitgeist thing or just anecdotal evidence of design
0:12:40 focused startups precisely because of the thing you’re saying, because that’s one of
0:12:41 the ways to instantly differentiate.
0:12:46 I think it’s also design thinking is sort of another way of saying, thinking about…
0:12:47 Oh, I hate that phrase.
0:12:48 I know.
0:12:49 Talk about the platitude of all platitudes.
0:12:51 That phrase drives me fucking up a wall.
0:12:54 So here’s a better way to frame it, because I also don’t like that phrase, is it’s really
0:12:58 about that extension of the product experience and really taking that more holistic approach.
0:13:00 It’s not just about the UI.
0:13:03 It’s not even just about the user experience of a particular workflow.
0:13:05 It’s about that whole customer experience.
0:13:07 We’re actually entering a period of time where more and more people in the workforce
0:13:11 are sort of digital natives, and they want to be power users.
0:13:14 Why isn’t there an equivalent to Microsoft Excel on the web?
0:13:15 Like Google Sheets is not Excel.
0:13:21 The current state of collaborative tools in SaaS apps is just so weak and they don’t
0:13:22 let you be a power user.
0:13:25 It’s also, I think, ignoring the realities of organizations today, which used to be so
0:13:29 siloed, and now you have people collaborating cross-functionally in different ways.
0:13:33 You could argue that Google Docs did create a multiplayer mode where you could have collaborative
0:13:37 editing, but it was just such a garbage experience from a functionality standpoint.
0:13:38 It was afterthought.
0:13:40 It wasn’t baked in from natively.
0:13:41 That’s basically my rule of thumb for all of this.
0:13:43 If it’s an add-on, it’s not important.
0:13:47 I think that’s what, well, I mean, all of, I would say, Google G Suite is an add-on.
0:13:50 Google should just shut down G Suite altogether, even though the whole Silicon Valley would
0:13:51 get crazy.
0:13:55 I mean, there are rounding error in their business, and it’s a rounding error to productivity
0:13:56 versus what Microsoft has.
0:14:01 I don’t think they’ll do that, but strategically, it’s just so unimportant for them.
0:14:06 What I would say, though, is that I like software that is easy to use, that has that
0:14:09 short time to value, but allows me to be a power user if I want to be.
0:14:14 In fact, as an investor, when I talk to companies, I always try to figure out what is their pricing
0:14:16 and packaging strategy.
0:14:17 What is packaging?
0:14:18 Actually?
0:14:23 Packaging is usually a convenient bunch of things, but to me, packaging is what set of
0:14:24 features.
0:14:27 Are you going to put into an offering to a customer?
0:14:30 I always try to think that you want to make it easy for your customer to give you money.
0:14:31 That doesn’t matter.
0:14:35 It’s like a foundational principle for me, and so packaging is our way to do that.
0:14:38 We’ve all been to the restaurant where the All Our Cart menu is all over the place, but
0:14:40 sometimes restaurants just say, “Well, here’s the three options.”
0:14:44 It comes with one of these appetizers, you get this main course, and you get this dessert.
0:14:47 If you want to make things easy for people to give you money, generally, people come
0:14:52 up with packages, and the friction is removed to becoming a buyer.
0:14:57 In the SaaS world, sometimes there might be a tier that says, “You’re going to get the
0:15:01 full functionality of the product, but you’re not going to get archiving and logging and
0:15:03 all this detailed reporting and analytics.”
0:15:07 It allows the company that maybe doesn’t want to spend as much or isn’t as big to get the
0:15:12 full functionality of your product, but then there’s a hurdle, and usually when I think
0:15:17 about packaging, usually there’s a key product milestone that happens that forces somebody
0:15:18 to jump to the next tier.
0:15:19 Interesting.
0:15:20 What do you mean?
0:15:21 Give me an example of that.
0:15:22 Sign on is a good one.
0:15:25 You have to generate accounts and use a product, but if you want to tie it to your octa directory
0:15:30 or some other directory service, you’re going to have to jump to a much more expensive tier,
0:15:33 but generally, the customers that have to jump that tier are more enterprise companies.
0:15:34 They have a directory service.
0:15:36 They have a single sign-on service.
0:15:39 They might want two-factor authentication with tokens.
0:15:41 The security person in me doesn’t love that one being a tier.
0:15:45 I always think you want all your customers to be secure, but there are other tiers, so
0:15:46 like compliance.
0:15:50 If you’re in a regulated industry, you might not just be satisfied with 30 days of logging.
0:15:52 You might need 365 days of logging.
0:15:55 You might need to be able to export your logs to another data store.
0:15:58 So far, if I heard that as an entrepreneur, though, I would assume that all packages are
0:15:59 tiered.
0:16:02 Are there untiered packages where it’s just like a different combo that’s all like kind
0:16:03 of horizontal?
0:16:05 I don’t think I’ve seen that.
0:16:09 Generally it’s much more of like a ladder where the next package includes everything
0:16:14 in the previous package, and I think that while there’s usually a number of features
0:16:17 that get unlocked when you go to the next package, to me, there’s always one that has
0:16:19 that forcing function.
0:16:23 In fact, when I think of packaging, it’s oftentimes a way to segment your customer base because
0:16:27 you’re going to say like, “We know SMB and mid-market under 1,000 employee companies,
0:16:28 they’re going to be in this package.”
0:16:32 Everything we do, the product manager in that package is thinking about those features,
0:16:35 thinking about that persona, and then the next package, the person saying, “Wait a minute,
0:16:39 I want to go after the 1,000 to 10,000 employee company,” and this is what they need.
0:16:41 This is how I communicate with them.
0:16:43 This might be how I do webinars to them.
0:16:46 This is how I’m going to do pricing that’s more fits their model.
0:16:48 Maybe you can’t do a three-year contract if you’re on the low-end product.
0:16:53 So all these things are puts and takes that reflect where’s the product, who is the customer
0:16:57 you’re targeting, and then how do you want to market and create demand with that audience.
0:16:59 Is there a balance or a rule of thumb?
0:17:03 I know I’m sure it must vary by business in what the ideal number of packages are or
0:17:07 how many customer segments you should be trying to reach as a startup.
0:17:09 I think generally fewer is better because focus is key.
0:17:10 Less is more.
0:17:11 Yeah, less is more.
0:17:15 Time is always the most valuable currency in an individual’s life and a company life.
0:17:19 And then aligning all that time behind the most important, like putting more wood behind
0:17:22 fewer arrows to me is always much more important.
0:17:26 I think generally like two packages, three packages, when you make it too complicated
0:17:30 for the customer to figure it out, that creates friction to the sales cycle.
0:17:34 Now with that said, one thing that startups often do is they share their pricing publicly
0:17:35 on the site.
0:17:38 And the engineer and all of us, like the pragmatic person and all of us, we’re like, “Well, of
0:17:43 course we want to share pricing because as customers we hate not knowing the price, but
0:17:47 as products get much more nuanced and organizations that are buyers and you actually don’t know
0:17:50 what your pricing discovery looks like, you’re better off not sharing your pricing.”
0:17:51 Okay.
0:17:55 And one way you know you have a great product is when your salespeople are the ones demanding
0:17:58 you remove the pricing because that means that they’re telling you…
0:17:59 You can get more money.
0:18:00 You can get more money.
0:18:05 Maybe you’re a technical CEO who’s becoming a product CEO, who’s becoming a sales CEO.
0:18:08 If you’re listening, you’re going to be like, “Wait a minute, they’re telling me we’re
0:18:09 leaving money on the table.”
0:18:11 It’s generally a very strong signal.
0:18:14 I have a stage question on this though because if you think about the definition of a startup,
0:18:19 a startup by definition is a business under a high condition of uncertainty compared to
0:18:20 more established business.
0:18:22 I wouldn’t even peg it to a particular size.
0:18:25 And given that, a startup is an experiment.
0:18:26 You’re running an experiment.
0:18:29 And the product, you can run multiple experiments at the same time.
0:18:32 We’ve heard of the famous pivot, the dreaded P word, there’s all these different flavors
0:18:33 of this.
0:18:37 How do you run multiple experiments and also strike a balance with focus and the pricing
0:18:38 and packaging strategy?
0:18:41 Well, that is the art of running a business.
0:18:42 Not a science.
0:18:43 Yeah.
0:18:46 And everything is multivariate, but you can generally, if you have smart people paying
0:18:50 attention to the numbers, paying attention to the data, collecting the analytics and giving
0:18:54 yourself enough time to collect that data, the worst thing for a company to do is make
0:18:58 a decision and then not allow there to be enough time to collect the outcome of that
0:19:02 decision and understand the consequences of that decision and then they make another decision.
0:19:06 So the question of how do you make decisions and run multiple experiments, I don’t think
0:19:07 it’s that complicated.
0:19:11 As long as you’re paying attention to what are the outputs from those decisions that
0:19:15 you should be looking for and you should be looking at what’s changing across the business.
0:19:20 We’re living in an era today of running companies where it’s much easier to collect and analyze
0:19:21 data than it ever has been.
0:19:25 You have data lakes where you can bring in product data, your CRM can tie into that product
0:19:26 data.
0:19:27 We’ve never had that.
0:19:28 We have BI tools now that we have open source.
0:19:29 Business intelligence tools, right?
0:19:30 Yeah.
0:19:31 That’s right.
0:19:32 We have open source business intelligence tools.
0:19:34 We actually run complex analytics and say, “Wait a minute.
0:19:38 My West Coast territory is just doing so much better than my East Coast territory.
0:19:39 What is the difference that’s pushing there?
0:19:43 Is it because we actually are running more demand-gen campaigns on the West Coast and
0:19:46 the marketing team on the West Coast is separated or is it just that the West Coast sales reps
0:19:47 are better?”
0:19:48 You need to be able to tease apart those.
0:19:51 You need to be able to tease those things apart, but it’s easier to get access to the
0:19:55 data and analyze it quickly and avoid that sort of analysis paralysis than I think it
0:19:57 ever has been in the past.
0:20:01 So a big part of this, so the big theme I’m hearing from you is a lot of these things
0:20:05 have intentionality, even if you don’t know the outcome, and that you can actually control
0:20:09 that intentionality by kind of being introspective, understanding your decision making, understanding
0:20:10 what works.
0:20:11 That sounds great.
0:20:17 Now, as a leader of the company, how do you, the CEO, figure out what to work on and depending
0:20:18 on what stage you’re at?
0:20:22 This whole journey from technical to product to sales to go to market, that’s not necessarily
0:20:23 perfectly linear.
0:20:24 So how do you figure this out?
0:20:25 It’s not linear at all.
0:20:28 I mean, sometimes in retrospect, we like to look and think that it was linear.
0:20:29 Of course, yeah.
0:20:32 But I think that there’s different ways to figure out sort of how do you prioritize
0:20:33 your time?
0:20:35 Where do you spend your mental calories?
0:20:36 Mental calories.
0:20:37 I love that phrase.
0:20:38 Yeah.
0:20:39 And that’s how I think about my days.
0:20:40 What do I want to, like you only have so many mental calories.
0:20:41 So I think about my day too.
0:20:45 I think of the nutrition density in terms of return on, so I have a phrase that I use
0:20:48 for all my editing, which is ROE, which is return on energy.
0:20:49 Oh, that’s good.
0:20:55 So I refuse to spend time on something that the output is going to be vastly low proportion
0:20:59 outsize win to what the amount of work I put in in terms of energy, creative.
0:21:01 So I have a whole framework for thinking about this.
0:21:02 I love that.
0:21:03 I’m ridiculously productive on this.
0:21:06 Well, hopefully this podcast gets published because I don’t know that it had a high ROE.
0:21:09 So I think figuring out how you spend your mental calories is a really important question
0:21:10 to ask.
0:21:14 And sometimes the act of asking that question itself is just like part of the process of
0:21:18 figuring out how to spend your time and spend it wisely.
0:21:20 And there’s different things that happen along different stages.
0:21:23 If you’re like, I always just look at like, what is the problem in the company?
0:21:26 Is it that we can’t get customers and then figuring out who that right customer is.
0:21:30 But as a company starts to mature, a lot of these companies get to this like $2 million,
0:21:32 $3 million in annual recurring revenue.
0:21:33 That’s a huge milestone.
0:21:35 Very few companies ever get there.
0:21:39 But yet it’s tiny when you should be doing $20 million, $30 million if you aspire to
0:21:40 be there.
0:21:43 Like you can celebrate the milestone, but it’s clearly you have a long way to go to build
0:21:46 an enduring iconic company.
0:21:48 And so at that point though, you start to have a leadership team.
0:21:51 One of the biggest things that we see when we give technical founders advice is they need
0:21:53 to bring on like a VP of engineering.
0:21:55 They need to bring on like a head of sales.
0:22:00 And they keep resisting this thing because they’re kind of attached to their early startup
0:22:01 team.
0:22:05 So how do they figure out when to really, there’s a lot of religious advice and debates
0:22:06 around this actually.
0:22:07 Yeah.
0:22:10 So I think that there’s a bunch of, you know, I always look at questions like, what time
0:22:11 is it?
0:22:12 Like, what is the priority?
0:22:15 Like, are you trying to figure out product market fit or are you focusing on go to market?
0:22:16 Like what time is it?
0:22:18 Are you hiring sales people and ramping up?
0:22:21 Are you like figuring out the customers are churning and you better go fix your product?
0:22:26 This is so important that if you were to ask all your leaders and all the people in your
0:22:29 organization, like what is the most important thing for our company right now, they should
0:22:30 have an answer.
0:22:34 But one of the, just one of the more tactical conversations that I have with leaders, when
0:22:37 they’re, especially when they’re a startup and they have this core founding team.
0:22:38 And then they’re thinking about scaling.
0:22:42 And they say, well, you know, I have this engineering manager, he or she was with me
0:22:43 from the beginning.
0:22:44 And I think, I think they’re doing a great job managing.
0:22:48 One of the things I highlight is like bringing in a world-class VP of engineering, like that
0:22:52 could rock the boat, it could cause issues, but it’s not an indictment of your current
0:22:53 engineering manager.
0:22:54 Like that’s not what’s happening.
0:22:58 Part of bringing on these high performing leaders and these really well respected leaders
0:23:03 that have a cult-like following with the people that have worked with them and for them before
0:23:06 is that they are going to help you accelerate your ability to recruit world-class talent.
0:23:09 And when you deliver that message to that person on your team who’s been there from
0:23:12 the beginning and is doing a great job, like that should resonate.
0:23:13 It’s like, oh, wait a minute.
0:23:14 Yeah.
0:23:15 We can get way better people way faster.
0:23:16 Like, yeah, let’s bring that person on.
0:23:19 Again, you have to be very careful about knowing, like, what are the problems you’re
0:23:21 trying to solve in the organization.
0:23:26 But oftentimes, you know, and I think VCs have a bad rep for this, is it like they shove
0:23:30 in somebody who’s way too senior and comes from way too big of a company.
0:23:33 And so you have to think about, like, what is the right team I need for the right time?
0:23:37 I think Ben wrote this in his book, actually, which is the mistake that people hire for
0:23:39 the future instead of the hire for the thing they need now.
0:23:43 This often comes up with like VP’s of sales hires where somebody maybe has run, you know,
0:23:46 a 10 or 20 person team, but you’re like, well, can they run a 500 person sales team?
0:23:48 Well, you don’t have a 500 person sales team problem.
0:23:51 People often think about the executives that are hiring like, well, is this person going
0:23:54 to be with me for four years or five years, for six years?
0:23:56 I think that’s not always the right question to ask.
0:24:00 In fact, I had a board member once, Dave Strom, who’s a mentor to me.
0:24:02 I think of him as like the Yoda in my life.
0:24:06 And he once said an expression that I’d never heard before, horses for courses.
0:24:07 So have you ever heard of horses for courses?
0:24:08 No, I don’t even know what that is.
0:24:09 Yeah.
0:24:10 So I think it’s like it’s sort of an archaic expression.
0:24:11 Yeah.
0:24:14 And in race, and in horse track racing, you know, there’s like dirt courses, there’s
0:24:15 grass courses.
0:24:16 Oh, I get it.
0:24:17 You know, you run the right horse for the right course.
0:24:18 Oh my God.
0:24:19 That’s a great phrase.
0:24:21 There’s a bad part of this phrase too, though.
0:24:25 There’s a bad connotation, which is that sometimes when horses, they run their few races and
0:24:26 then they’re finished.
0:24:28 They run their course.
0:24:29 That’s where it comes from, that expression.
0:24:30 They’ve run their course.
0:24:32 And do you know what happens to horses that run their course?
0:24:33 No, I don’t want to know.
0:24:34 Are they turned into gelatin?
0:24:35 Something like that.
0:24:36 I always used to joke.
0:24:39 It wasn’t very nice probably, but I would joke with the VP of sales I had to open yesterday.
0:24:43 Like, you know, maybe every quarter was his last quarter because he just constantly out
0:24:44 performed.
0:24:47 And we always wondered when we hired him, like, is this guy going to scale?
0:24:48 Now, he scaled wonderfully.
0:24:49 He’s an incredible sales leader.
0:24:54 He went from, you know, a 20-person sales team to ultimately a 200 or 400-person sales
0:24:55 team.
0:24:57 Then once we got to Cisco, he did wonderfully, but we didn’t know when we hired him and how
0:24:59 far he’d get past 20 people.
0:25:01 You got to hire horses for courses.
0:25:02 The right team for the right play.
0:25:07 This is a good way of really figuring out, like, is my leadership team adding capacity
0:25:08 for me?
0:25:10 Are they helping me understand what’s happening in the business?
0:25:15 Because as a certain point as a CEO, you’re going to start to spend less time on engineering,
0:25:16 less time on product.
0:25:20 Ideally, you’re going to spend more time in the field with customers, with partners,
0:25:21 with customer success.
0:25:25 And as you start to spend less time with any individual function, you’re going to need
0:25:29 to have leaders in place that really are spending all their time really understanding
0:25:31 closer to the medal what is happening.
0:25:35 I love that you said close to the medal because that’s the exact phrase I use when I think
0:25:36 of this.
0:25:37 It’s like bare-metal leadership.
0:25:38 Totally.
0:25:41 And that’s actually the biggest challenge as a product-oriented person or a visionary
0:25:45 for whatever the product is in any field is how do you kind of keep that close to the
0:25:49 medal insight yet you can’t actually be close to the medal if you’re scaling.
0:25:52 So this actually comes up a lot in startups.
0:25:56 This idea that if you are the product visionary, you’re the founder of the company, that means
0:25:59 you are the product manager for the company, but at the same time, you need to scale an
0:26:00 organization.
0:26:05 And I think it’s important to differentiate the product manager from the product visionary.
0:26:06 Oh, great.
0:26:10 As the founder and CEO, you can always be the product visionary, but there is going
0:26:14 to be a time where you’re not going to be able to spend hours of time with the engineers
0:26:18 hearing how they’re working on our product or how it’s technically going to work.
0:26:21 You’re not going to spend hours and hours of time looking at all the NPS survey data
0:26:24 or the customer support tickets that are coming in.
0:26:27 And so like oftentimes I’ll meet these startups are like, oh, I can’t hire a product manager.
0:26:28 I am the product manager.
0:26:29 That’s a common thing for technical.
0:26:30 Totally common.
0:26:32 And it feels like it’s your baby.
0:26:33 You don’t want to let it go.
0:26:36 You’re only going to have five seconds a day to think about different decisions you make.
0:26:40 And so if your engineering team and the rest of the organizations constantly come into you,
0:26:41 you’re going to end up getting paralyzed.
0:26:42 Yeah.
0:26:45 The worst thing for a product visionary is to make some decisions, and then they know
0:26:48 where the wrong decisions because they lack data or they lack the time to be thoughtful
0:26:49 about it.
0:26:52 And then they start to undermine their own thinking about whether or not they even are
0:26:56 a product visionary when the reality is just hire a product manager.
0:26:59 You’re not offloading the product vision to that person.
0:27:05 What you’re offloading is the day-to-day ground war of figuring out what is customer support
0:27:08 telling me, what is sales telling me, what is engineering telling me, what are customers
0:27:14 telling me, synthesizing, analyzing, prioritizing, sorting that data.
0:27:18 And obviously as a founder and visionary you have ultimate say, but you’re going to be
0:27:22 armed with so much more insight information that your intuition, which plays a big role
0:27:24 too, is just going to be that further enhanced.
0:27:29 As a visionary you’re going to have some special secret, some earned power that you have over
0:27:32 the lifetime of your experience, where you’re the domain expert in a problem set, you’re
0:27:33 going to know more about it.
0:27:35 Because you’ve gone through the idea maze, you’ve literally lived and breathed this thing,
0:27:39 you built the company, you started it because you literally have it seeping out of your
0:27:40 pores.
0:27:41 That’s right.
0:27:43 We had a recent podcast at Safi Bakal and he described how Steve Job had both the artists
0:27:44 and the soldiers.
0:27:47 And so not only did he have himself, but he had like Tim Cook and Johnny Ive.
0:27:51 But also when you think about the story of the iPhone, the app store was actually a result
0:27:56 of his team coming up with the point that, hey, you can’t just have Apple apps on this
0:27:58 if you want people to use this.
0:28:01 And so people and your product managers will come to you and they’ll, when they have conviction
0:28:04 on something and they have the data and they have the view, you’ll then be able to make
0:28:05 those bets.
0:28:06 Yeah.
0:28:08 And nobody would say that he wasn’t a product visionary just because he didn’t come up with
0:28:09 the app store.
0:28:12 On that note, just a probe on one bit, because I’ve always wondered about this.
0:28:17 There is a tension between this idea, like I hate this idea of the head and the hand.
0:28:20 You can’t have one person be the head and the other person be the hand.
0:28:21 How do you reconcile that bit?
0:28:25 Like how do you, I guess what I’m asking is how do you calibrate along this line of visionary
0:28:26 to manager?
0:28:30 So I think you want to know what you’re hiring for because there are product managers that
0:28:33 are much more analytical and there are product managers much more visionary and you might
0:28:35 need different kinds of people at different kinds of times.
0:28:39 So I think you have to be sort of self-aware and be really intellectually honest because
0:28:42 if you actually need someone who is more visionary, then you’re going to have to deal with the
0:28:46 fact that you’re going to be going to battle and sitting in a room and duking it out over
0:28:47 ideas.
0:28:51 It leads to a secondary insight, which is that if you’re a CEO of a company and you
0:28:55 do not trust that the information you’re getting from one of your leaders is what’s actually
0:28:57 happening on the ground, that’s a tremendous problem.
0:28:58 That’s a huge red flag.
0:28:59 Massive problem.
0:29:00 Fire and move on.
0:29:01 Fire.
0:29:02 Or it could be you if you’re just not a trusting person.
0:29:03 I think you have to work to resolve these things.
0:29:04 Yeah.
0:29:07 Like you don’t just cut and move on immediately, but you either have to work to understand what’s
0:29:10 actually, like do they understand what’s happening and are they able to communicate it to you
0:29:11 and the rest of your leadership team?
0:29:13 I always like to think of leadership teams.
0:29:17 It’s not just this like, oh, the head of sales reports to the CEO, the head of marketing reports
0:29:18 to the CEO.
0:29:22 And you have these like siloed, you know, sort of pairwise conversations.
0:29:25 So leadership team needs to be working together as a team and communicating with each other
0:29:30 because as a CEO, you don’t want to be, you know, interjecting and intervening in every
0:29:32 conversation, every decision.
0:29:35 And so you want to start to figure out like, are they collaborating?
0:29:37 Are they sharing each other’s experiences?
0:29:39 Do they understand what’s happening in each other’s businesses?
0:29:40 Are they meeting on their own?
0:29:44 I think as a CEO, you actually want your leadership team to meet independent of the CEO.
0:29:45 That’s actually really interesting and counterintuitive.
0:29:46 Yeah.
0:29:47 I think it’s really important.
0:29:51 And I think it does happen in a lot of high-performing teams very commonly, maybe not explicitly,
0:29:52 but it happens.
0:29:56 And then obviously, I think in some places you can do it explicitly and when it’s done
0:30:00 a productive and positive way, not because the CEO is a distraction, but ideally the
0:30:04 CEO is out doing something that’s of high value to the company.
0:30:07 But if you get to this place where you do not have confidence that you are getting the
0:30:12 best information from your leaders, if you don’t resolve that, then you have to find
0:30:13 someone who’s a better fit.
0:30:17 Whenever I hear when I talk to a CEO who’s having a tough time in the company and they’re
0:30:21 telling me about like what’s happening, I’m like, well, like, just tell me, like, do you
0:30:23 really believe that that is what’s happening?
0:30:26 And you either have to go deep and as a CEO, you do get these occasional bullets where
0:30:30 you can cause a little bit of like organizational stress to like go at three levels deep and
0:30:31 really figure it out.
0:30:34 And if you find out what’s happening, it’s not what you are being told, you gotta make
0:30:35 a change in leadership.
0:30:39 By the way, I should just say that all my lessons about leadership and management, I pretty
0:30:40 much learned the hard way.
0:30:43 So I’m just trying to help save other people from making the same mistakes I made.
0:30:44 Yeah.
0:30:46 Well, speaking of that, let’s talk about your stories.
0:30:49 So you’re the founder of a company called OpenDNS.
0:30:51 First of all, what is OpenDNS?
0:30:56 So OpenDNS is a cybersecurity service that delivers a faster and safer internet.
0:31:00 And we really innovated on a 25-year-old technology that used to be a cost center that nobody
0:31:01 wanted to innovate on.
0:31:04 We actually proved that you can actually build a business on top of this like thing that
0:31:05 used to be free if you make it better.
0:31:09 So speed was one part, but then the other part was security.
0:31:13 Let’s say you type in zamazon.com, your meaning to go to amazon.com, that could be a phishing
0:31:15 site trying to steal your credentials.
0:31:19 So we would say, hey, wait a minute, we know that from our, you know, tens of millions of
0:31:23 users, what you really meant to type in was amazon.com, so we’re going to show you a page
0:31:27 that says, hey, you typed in zamazon.com, we think it’s a fraudulent site.
0:31:29 Did you really mean to go to amazon.com?
0:31:30 That may help protect you from getting phished.
0:31:32 It was the first third-party dance provider.
0:31:35 In fact, when we started the company, some of the, like, the gray beards of the internet
0:31:40 who I respected tremendously, they told me, A, what I wanted to do wasn’t possible, and
0:31:43 B, even if it was possible, nobody would want it because they just would get it from their
0:31:44 ISP.
0:31:45 Oh my God, this reminds me of Mark with Netscape.
0:31:49 One of my favorite stories is I saw these old forums that he was on when he was proposing
0:31:50 like a more of a graphical user interface.
0:31:51 Oh, like the image tag.
0:31:52 Exactly.
0:31:55 And the thing that I thought was so funny is the people who are the established kind
0:31:58 of old fogies for lack of a better phrase, they don’t like the change ironically, even
0:32:00 though they were very revolutionary at the time.
0:32:03 So you mentioned a 25-year-old technology.
0:32:06 Why was that almost impossible to them?
0:32:10 So think of the DNS as like a phone book, except that what we wanted to do was not just give
0:32:13 the same phone book to everyone, but we wanted to give a custom phone book to every person
0:32:17 which meant, let’s say you typed in playboy.com, well, for some user over here, they may not
0:32:20 want content filtering, so they want the answer for playboy.com.
0:32:23 But maybe for someone who has small kids at home, they want a different answer.
0:32:27 And doing this at very high speed was thought to be impossible.
0:32:28 That’s fantastic.
0:32:31 But it turned out that it was possible, and we could do it faster than even if you had
0:32:32 no preferences in settings.
0:32:34 I have to ask you, how old were you when you had the insight that you wanted to build
0:32:35 open DNS?
0:32:40 So I had started a DNS company in college that did a different kind of DNS.
0:32:43 And through that, I had gotten super interested in cybersecurity.
0:32:48 And so I met an investor when I moved out to California, who had asked me basically why
0:32:50 I wasn’t doing more with my original company.
0:32:53 Then he and I ultimately came up with the idea for open DNS.
0:32:56 That original business model that I worked on with him was an advertising supported business
0:32:57 model.
0:33:01 We pivoted the business, and then in 2009, having the people that use our service be
0:33:04 the people that pay us for our service, was such a much better alignment of interest.
0:33:06 And so that journey took a long time.
0:33:10 And by the time we pivoted the business, it really was a different business than when
0:33:11 we started in.
0:33:15 Then when we sold it in 2015, the Cisco was really a full blown cybersecurity company.
0:33:16 Why did Cisco want it?
0:33:21 So I think if you looked at what happened between, let’s say 2007 and 2015, the iPhone
0:33:22 came out.
0:33:25 You had more and more people working from coffee shops that all had Wi-Fi.
0:33:29 You had workers working from the road, people using mobile devices.
0:33:34 So installing like Norton Antivirus or McAfee Antivirus on your desktop was no longer sufficient
0:33:35 security.
0:33:39 And so our service, Open DNS, was cybersecurity delivered as a service.
0:33:42 And it happened just intrinsically as a part of your internet connection.
0:33:44 So you didn’t have to have special software.
0:33:47 You didn’t have to install an appliance or a piece of hardware.
0:33:50 And so as people were working differently and the networks were becoming more ephemeral,
0:33:54 Cisco, which is a major cybersecurity company, such as the largest cybersecurity company,
0:33:57 wants to evolve to match that shifting IT landscape.
0:33:59 You mentioned the pivot a few times.
0:34:02 Tell me about that, because that’s such an overuse, one of those platitude words like
0:34:04 big P, little P, whatever.
0:34:07 And I know you mean it and actually what happened, but give me a little bit more texture around
0:34:08 the pivot.
0:34:09 What was that like?
0:34:10 The best time I never want to have again.
0:34:15 I mean, look, this might be its podcast onto its own, because there was an 11 month period
0:34:17 where I wasn’t even CEO of the company.
0:34:19 My original investor had fired me.
0:34:20 Oh my God.
0:34:21 I was CTO.
0:34:22 I didn’t know that.
0:34:23 I’ve been demoted.
0:34:24 CTO is awesome though.
0:34:25 I think CTO is the most powerful person in the company.
0:34:26 Not when you want to be CEO.
0:34:27 I guess that’s true.
0:34:30 We pivoted the business 2009.
0:34:33 What we thought was a consumer business actually turned out to just be a free business.
0:34:37 One day we actually got a call from a major oil and gas company that had been using us.
0:34:41 And we knew they were using us globally, like on oil rigs or their headquarters or other
0:34:42 distributed offices.
0:34:46 And so then they finally we got an email like, look, we need to have a support contract
0:34:49 like as a matter of our, just like corporate hygiene.
0:34:51 So like figure it out and like give us a quote.
0:34:54 We need to like have a way to call you if there’s a problem.
0:34:58 And so we went and got one of these like virtual phone numbers on the internet that would like
0:34:59 route to like an engineer’s phone number.
0:35:02 And if that person didn’t pick up about route to like the next engineer’s phone number,
0:35:04 if that person didn’t pick up about route to my phone.
0:35:05 Oh my God.
0:35:06 You were like the support desk.
0:35:07 Yeah.
0:35:11 It was like a tiered call system and went to like three people and we sent them a quote
0:35:13 for a hundred grand and they signed it immediately and returned it.
0:35:17 And they now went from like us making $2 a year and advertising, which we hated to paying
0:35:20 us a hundred grand for something we’re already doing and we get to turn off the ads.
0:35:23 You don’t need to be a rocket science to figure out like, wait a minute, maybe there’s something
0:35:24 here.
0:35:26 And we had two or three other companies that had asked previously for something like that.
0:35:29 So we went and sent them quotes and they all signed them and returned them.
0:35:32 Why do you think you didn’t know that this would be the business model up front?
0:35:33 Like why did you have to pivot?
0:35:35 Honestly, not to sound judgmental at all.
0:35:38 It seems obvious to me when you say it in hindsight.
0:35:39 Totally.
0:35:40 So I’m confused at why you didn’t see that coming up.
0:35:44 I think we were sort of enamored with this idea of keeping the whole internet safer.
0:35:46 And that meant going after individuals.
0:35:47 Idealistic.
0:35:51 We had partnerships with Nectar and D-Link and these people that sold consumer routers.
0:35:55 And so we sort of ignored the opportunity that’s right in front of our face.
0:35:58 But as soon as you realize like you’re not going to be able to raise money and like you
0:36:01 actually have to build a business, you start to like open your eyes a little bit.
0:36:02 And so we did that.
0:36:07 And then I hired Michelle Law, who actually spent seven years at Greylock to run BD for
0:36:08 us.
0:36:13 And ultimately she became my COO, a wonderful person and good friend.
0:36:18 And she basically had seen enterprise companies many times and so she realized as we wanted
0:36:20 to go enterprise that a bunch of the team had to change.
0:36:23 First of all, half the team just like didn’t care about building an enterprise software
0:36:24 company so they just quit.
0:36:28 Then like the other half of the team just like could not internalize that we can’t just
0:36:32 like change the UI overnight because it turns out some of our big customers had their own
0:36:34 manuals that they had built with screenshots of our product.
0:36:38 If we got a nasty email once from this major oil and gas company, they said like we have
0:36:42 all this training material and screenshots and videos we made and you just totally changed
0:36:43 your whole dashboard.
0:36:44 Yeah.
0:36:45 Like you can’t do that.
0:36:47 And you just have to learn how to manage those things.
0:36:49 And like then you do feature flags, which are things that are like common today.
0:36:54 But in 2009, like feature flagging things and feature flagging means that like some subset
0:36:58 of cohort of customers gets the access to a new feature or a new look and feel.
0:37:01 A lot of people use it for A/B testing to see if something works, but you can also use
0:37:05 it just to like keep certain customers on certain packages or on older features or older
0:37:06 looks and feel.
0:37:11 So it’s not like you still have one code base, but people have slightly different experiences.
0:37:13 And so we were starting to do those things.
0:37:16 Like we started implementing feature flags and things of that nature, but it meant that
0:37:21 over the course of about 12 to 18 months of the 30 people before the pivot, I think only
0:37:22 three were left at the end.
0:37:25 When did you go from CEO to CTO?
0:37:28 So right before all that happened, from most to 2008.
0:37:32 And then the only good thing that ever came out of the total global recession and economic
0:37:35 collapse was that my early investor needed cash.
0:37:39 And so we found two investors, and that’s actually when I first met Mark and Ben.
0:37:42 We found two investors to come in and buy out my early investor.
0:37:44 Those investors came in, started to rebuild the company.
0:37:49 But the biggest thing that’s fascinating to me is you came back as a CEO.
0:37:53 So what changed that you didn’t make this, I mean, cause you’re the same person, you
0:37:54 didn’t change overnight.
0:37:55 Right.
0:37:57 Like how did you pull that off?
0:37:58 Yeah.
0:38:02 Coming back as CEO the second time, after spending almost a year as CTO, one of the things I
0:38:06 saw when I wasn’t CEO was all these things that weren’t happening in the company.
0:38:07 Yeah.
0:38:08 That sort of been happening.
0:38:12 And of course I blamed the current CEO, but the reality was I actually was not doing them
0:38:13 either when I was CEO.
0:38:16 Sometimes you just have to have this like outside the glass box kind of view.
0:38:17 Yeah.
0:38:18 Wait a minute.
0:38:19 People don’t know what’s important.
0:38:20 Wait a minute.
0:38:22 We’re not making it clear what the priorities are.
0:38:23 Wait a minute.
0:38:26 We’re not firing these low performing people, but I wasn’t doing any of those things either.
0:38:28 And so that to me was very eye opening.
0:38:32 So when I came back as CEO, I think I was a much better listener.
0:38:35 I think I had this like belief maybe the first time I was CEO that I’m expected to have all
0:38:36 the answers.
0:38:37 Uh huh.
0:38:38 And it’s just not possible.
0:38:41 What is important as CEOs have to make decisions, and I think they have to be able to articulate
0:38:44 their decisions, but they don’t have to have all the answers.
0:38:46 That’s a really important point.
0:38:49 There’s actually a big difference between an answer and a decision.
0:38:50 Totally.
0:38:52 That’s actually something really to reflect on because I think most people conflate those
0:38:53 two things.
0:38:54 Totally.
0:38:56 And in fact, it turns out actually the opposite is true about having the answers.
0:39:00 In fact, I often tell CEOs because I like even before I joined Andrews Norwood’s people
0:39:01 call me for advice.
0:39:02 And that’s one of the things I really enjoy.
0:39:03 That’s why you’re VC now.
0:39:04 Yeah.
0:39:05 That’s why I’m a VC.
0:39:07 That’s one of the best parts about the job is like, I like being the first phone call
0:39:10 for a CEO when they’ve had a tough moment or they need help.
0:39:14 One of the things I often tell CEOs is like, you know, actually, like when you think about
0:39:17 the table of leaders around you, like there’s actually room around the table for one person
0:39:20 who has no idea what they’re doing, and that’s you, a CEO.
0:39:24 If you have the right leadership team, they’re adding intelligence for you.
0:39:28 You know, I had sort of gone from technical CEO to product CEO to sales CEO.
0:39:31 But my fault as a sales CEO is that I loved the dog and pony show.
0:39:32 I loved the pitch.
0:39:33 I hated the clothes.
0:39:34 Why is that?
0:39:35 That’s fascinating.
0:39:39 You know, I thought, you know, you could say that it was ego or ignorance or naivete.
0:39:42 Like, I didn’t like asking for the purchase order because first of all, I always thought
0:39:43 our pricing was low.
0:39:46 So if the customer and customers often like to negotiate.
0:39:49 Oh, you’re like ready to fight, like, fuck you, I want you to pay more.
0:39:50 Yeah.
0:39:51 So the customer goes, oh, $100,000.
0:39:52 I think it’s to be 70.
0:39:53 I’m like, fuck you.
0:39:54 It should be $140,000.
0:39:55 Like, I’m raising the price.
0:39:57 You’re the wrong guy to bring it the clothes.
0:39:58 Yeah.
0:40:00 My sales would be like, yeah, so you can’t negotiate with this customer because like,
0:40:02 you’re going to blow up this deal.
0:40:06 But also I didn’t like the uncomfortable, and I’m in a much different place now, obviously.
0:40:10 But like at that time and where I was as a CEO, I hated the negotiation.
0:40:11 I got uncomfortable.
0:40:16 Everyone, they taught me so much because there was only room around the table for me to not
0:40:17 really know all the answers.
0:40:23 I will often say that my CMO at OpenDNS, Jeff Samuels, I think of him not just as my CMO,
0:40:24 but as a mentor to me.
0:40:27 I mean, he taught me so much and actually I would say that about my VP of engineering,
0:40:29 my head of sales, my VP of sales.
0:40:32 I could take the inputs and use all that to make a decision.
0:40:35 And I felt very good about those decisions I made.
0:40:38 I think CEOs find it a huge relief when you tell them, you’re allowed to not know.
0:40:42 In fact, you kind of, if you have the best people, you’re going to know the least.
0:40:47 It is not uncommon for a CEO or a leader, a manager, this is just a good general life
0:40:48 advice.
0:40:51 It’s like, you know, you don’t want to be like the whiny person constantly like harping
0:40:52 on something.
0:40:56 But what I would say is like, you do sometimes need to present an idea more than once.
0:41:01 My old head of finance who ultimately became my best friend, used to always joke with me
0:41:04 that like he would just tell me everything he said twice because he knew the first time
0:41:05 I’d ignore him.
0:41:07 I think I have the same problem.
0:41:08 Yeah.
0:41:11 He would tell me some statistic about what’s happening with marketing spend or with hiring
0:41:12 or sales plans.
0:41:14 And I’d be like, oh, that’s not really a problem.
0:41:15 Like whatever.
0:41:16 Like, I don’t care.
0:41:17 You’re just like a crazy finance, like bean counter.
0:41:18 I don’t care.
0:41:22 But then like you’d come back like a week later and be like, hey, I have more data.
0:41:24 Like I did further analysis.
0:41:25 Like you ignored me.
0:41:26 But like, I know I’m right here.
0:41:27 And then I’d be like, oh, you’re right.
0:41:29 Why didn’t you tell me this last week?
0:41:30 Yeah.
0:41:34 So in that case, the CEO can get answers from all their management team and then make a
0:41:36 decision based on all the answers you’re hearing.
0:41:37 That’s right.
0:41:40 And I do think sometimes you do have to tell people more than once and that’s just a function
0:41:42 of how human beings operate.
0:41:45 So speaking of this telling people more than once and learning to listen, that was your
0:41:49 big shift between when you came back to be CEO and you kind of learned your lesson, so
0:41:50 to speak.
0:41:54 I honestly feel like that’s kind of a trite thing people say all the time, listen better.
0:42:01 I hate the design thinking mindset around empathy for the, and this scenario and this persona
0:42:03 and it’s just so, I can’t diagnose what’s off.
0:42:07 I think when you’re building a company, being empathetic, really it means understanding.
0:42:09 It doesn’t mean accommodating.
0:42:12 There were situations all the time, this happens all the time as a leader, like you may not
0:42:14 resolve that thing.
0:42:15 You can still understand it and be empathetic.
0:42:17 Like I can be like, yeah, that is terrible.
0:42:18 Like I understand what you’re saying.
0:42:19 I’m hearing you.
0:42:21 Honestly, that’s half the battle in relationships.
0:42:22 That’s right.
0:42:23 You don’t need an answer, 99% of it.
0:42:24 You just need to understand.
0:42:25 You should have someone to say, fuck, I feel for you.
0:42:26 That sucks.
0:42:27 And you’re already feeling like 80% better.
0:42:28 That’s right.
0:42:29 That’s right.
0:42:32 When I think about empathy, you really want to be a great listener.
0:42:36 And so actually a friend of mine, Wendy McNaughton, she does this all like New York Times thing
0:42:39 every other week where she really goes deep onto a topic.
0:42:40 She’s written these books.
0:42:41 I think she thinks of herself as an artist.
0:42:43 I think of her as an artist and entrepreneur.
0:42:47 One of the things that she taught me about a year or a year or two ago was like when
0:42:52 she’s trying to teach people like how to sort of be a really, really solid listener, is
0:42:56 that when someone’s talking to you, you’re like, tell me more about that.
0:42:57 That’s just a phrase.
0:42:58 Tell me more about that.
0:42:59 Five words.
0:43:00 So just make that your first question.
0:43:02 When you talk to a customer, oh, what’s going on?
0:43:05 Oh, we’re doing like annual planning, budget planning, tell me more about that.
0:43:07 How are you thinking about that?
0:43:08 What is happening?
0:43:10 What’s the frustrating part about annual planning?
0:43:11 Tell me more about that.
0:43:15 So what’s interesting to me about that is, to me, this is the difference between a focus
0:43:18 group and an ethnographer.
0:43:21 Focus groups and surveys are asking questions for things you already know to ask.
0:43:22 That’s right.
0:43:26 And an ethnographer is embedded in an organization or is sitting and is essentially just listening
0:43:29 to learn and observe and letting those patterns reveal things.
0:43:33 So deeper insights come out when you go down there, like tell me more about that path.
0:43:34 Exactly.
0:43:37 That’s when you get these flashes in your brain of like, wait, now I really understand
0:43:38 what’s happening.
0:43:40 It’s not that annual planning sucks.
0:43:43 It’s that you’re having budget issues that aren’t being resolved in a way that you need
0:43:46 or that maybe your tools you’re using to do annual planning or your way you communicate
0:43:50 and collaborate with your team or the way you work cross-functionally is not working.
0:43:51 Totally.
0:43:52 Totally.
0:43:55 I consider myself an ethnographer-esque editor because I want the context to know what I’m
0:43:57 not hearing, to really understand.
0:44:01 So it’s interesting because on the ethnographer side, I don’t think people know this about
0:44:05 you, but you started off your career or academic career because you actually started working
0:44:06 when you were like what?
0:44:07 Eighth grade?
0:44:08 What is that?
0:44:09 Like 14 or 15?
0:44:10 And that’s when you got your first W too, right?
0:44:11 Yeah.
0:44:13 I’ve had a 1099 or a W2 every year since eighth grade.
0:44:17 I worked at a mom and pop ISP in San Diego and I learned all about routing and networking.
0:44:20 I went to Washington University in St. Louis.
0:44:22 I applied to the School of Arts and Sciences.
0:44:26 When I went there to interview, they actually then had me interview with somebody in the
0:44:28 School of Engineering in the computer science department.
0:44:31 By the end of first semester of freshman year, I switched back to the School of Arts and
0:44:32 Sciences.
0:44:35 And the reason I switched is I took a class of Introduction to Human Evolution and I
0:44:36 just found it so fascinating.
0:44:39 Like I’ve always learned in my life, I do best of the things I really enjoyed working
0:44:40 on.
0:44:42 Like I have trouble doing things I don’t want to do.
0:44:43 Me too.
0:44:44 I’m the exact same way.
0:44:46 Yeah, sounds obvious, but like some people can actually just like will their way through
0:44:47 the hard stuff.
0:44:48 No, I can’t.
0:44:49 I have no energy.
0:44:50 I have zero.
0:44:51 Talk about returning energy.
0:44:52 I have like no energy to do the things.
0:44:53 Yeah.
0:44:54 I’m just like also like I’ll be okay if I just don’t do this.
0:44:55 Yeah.
0:44:56 But I really didn’t enjoy it.
0:44:58 So I learned how to optimize for the things that I like doing.
0:45:02 Anthropology, every book I read I thought was so interesting.
0:45:06 I learned about like how women enforce power hierarchy in South America in a way that we
0:45:07 don’t have elsewhere in the world.
0:45:10 I learned about like what happens in Southeast Asia around farming.
0:45:12 I learned about the Green Revolution in Africa.
0:45:16 But then I find that like in my life, I actually think about these things all the time.
0:45:19 That was back to me in my next question is, do you think it actually is useful in your
0:45:21 career as a technologist?
0:45:22 Absolutely.
0:45:24 I think about demographic transitions.
0:45:26 I think about when I read about like what’s happening in Japan.
0:45:29 It makes me think about how is that going to change my investing thesis?
0:45:31 And I think it comes up constantly.
0:45:34 It comes up both both tactically as you think about yourself in like leadership and organizational
0:45:35 dynamics.
0:45:37 It gives you an appreciation that there’s many perspectives in the world.
0:45:40 In fact, it gives you an appreciation that more perspectives are more better and you
0:45:42 want more.
0:45:46 So that is then a perfect way to close this episode.
0:45:52 So David Yulovic, he’s made a journey from anthropologist to technical CEO to product
0:45:55 CEO to sales CEO to go-to-market CEO.
0:45:56 And now investor.
0:45:58 Welcome to the A6NC podcast.
0:45:59 Thanks.

0:00:05 The content here is for informational purposes only, should not be taken as legal business
0:00:10 tax or investment advice, or be used to evaluate any investment or security and is not directed
0:00:15 at any investors or potential investors in any A16Z fund. For more details, please see
0:00:21 a16z.com/disclosures. Hi, welcome to the A16Z podcast. This is
0:00:27 Frank Chen. Today’s episode is called “Five Open Problems for the Blockchain Computer.”
0:00:32 It originally aired as a YouTube video. And you can watch all of our YouTube videos at
0:00:39 youtube.com/a16zvideos. Well, welcome to the A16Z YouTube channel. Today,
0:00:45 I’m here with Ali Yahya, our deal partner in the A16Z crypto team. And we’re going to
0:00:49 have a fun conversation. So here’s what we’re going to do. I’m going to pretend Ali to
0:00:54 be a Google software engineer or an Apple software engineer, right? So I’m somebody
0:00:59 who knows how to write software, has been doing it for a while. And then all of a sudden,
0:01:03 I saw my friends start to peel off and go to crypto startups. And I’m looking around
0:01:08 going, “What’s happening? Google’s a great place or Apple’s a great place. Why are people
0:01:13 leaving to go to crypto startups?” And maybe you can help me understand what’s causing
0:01:16 all these smart, talented people to head into crypto land.
0:01:17 I love it.
0:01:22 Fantastic. So maybe let’s just start with the world in which we live today, which is,
0:01:28 you know, I use my iPhone or my Android phone. I happen to use a Google phone, the Pixel.
0:01:35 I use Google Photos. I use Gmail. My carrier is T-Mobile. It’s sort of a centralized world.
0:01:39 And it works pretty well, right? Like, it’s pretty reliable. And Google gets all my photos
0:01:44 and my mail arrives when I want it. And so that’s not a bad world. Is crypto really
0:01:48 trying to overturn that world?
0:01:53 That world does work fine, but it’s not the frontier. So what I would say is the reason
0:02:00 that crypto is so exciting is because it offers a fundamentally new paradigm for computation
0:02:03 that has features that are completely novel and different from the features that enable
0:02:09 applications like social media, as it exists today, like sharing photos, like all of sort
0:02:14 of the centralized services that we know and love today. And so I think with every successive
0:02:22 wave of computation that we’ve seen throughout the history of computing, normally, the new
0:02:28 paradigm tends to suck at first and tends to be pretty bad at most things that the old
0:02:34 paradigm is very good at. But it happens to shine in one or two particular ways that enable
0:02:38 new applications that previously were just not possible to build. And so with, I mean,
0:02:43 I think one of the clearest examples is just the example of mobile phones enabling applications
0:02:49 like Uber, where applications like Instagram, by virtue of having a camera and a GPS bolted
0:02:54 onto the phone, that enable those kinds of behaviors that with a personal computer, you
0:02:57 couldn’t have possibly, couldn’t have possibly built.
0:03:00 Your PC didn’t know where you were necessarily, so it couldn’t enable Lyft.
0:03:04 Exactly. And it would have been also just deeply impractical for one to pull out one’s
0:03:05 laptop.
0:03:08 Hold on, let me get my PC.
0:03:16 And so with crypto, I think the key dimension along which these decentralized computers
0:03:22 that people are building in the world of crypto shine is that of trust. They provide this
0:03:32 new angle that previous computers didn’t have because previous computers are owned and operated
0:03:39 by individuals or by single entities like companies. And so you have to trust that individual,
0:03:43 you have to trust that company to actually run the software that they’re claiming that
0:03:47 they’re running and to actually do what they claim they will do with your data. And with,
0:03:51 we’re basically with the entire interaction between you and them. And so we trust Google
0:03:56 with our photos, we trust Google with our email, we trust Google with just about everything
0:04:01 that entails the kinds of interactions that we have with Google.
0:04:06 This new paradigm of computation is such that you now have a computational fabric that is
0:04:11 not owned and operated by any one person. This is the whole point of decentralization.
0:04:15 When people talk about decentralization in the world of crypto, they mean decentralization
0:04:23 of human control. Not decentralization of computing in a geographic sense. It’s not
0:04:28 decentralization in any other way that you may think. Like the key thing about crypto
0:04:34 is decentralization of human power and human control over systems and figuring out clever
0:04:41 ways to build a system such that it is self policing and such that it’s security and its
0:04:47 trust emerges bottom up from its participants and from individuals as opposed to top down
0:04:54 from like some trusted organization at the top that kind of enforces the rules.
0:05:00 Got it. So instead of trusting Google or Facebook or Apple, I can then trust the collective
0:05:05 of people who have contributed their computing, their storage, power, etc., etc., to deliver
0:05:12 the service that I’m consuming. And that’s the big innovation. And so why don’t we go
0:05:18 through the implications of that by sort of talking through, well, what will we need to
0:05:26 rebuild in crypto land, starting with compute, so that all of the applications that run on
0:05:31 top of this distributed computer sort of will have the power that you’re describing. So
0:05:35 let’s start with distributed compute. What do we need? So I guess maybe to set the stage,
0:05:40 if we think of compute today, we have computers that can perform certain monotransactions
0:05:46 per second. We have visa, which can clear so many financial transactions per second.
0:05:49 And then we compare those things with things like, well, how many Bitcoin transactions
0:05:54 can clear, how many Ethereum smart contracts can clear. So why don’t we talk about how
0:05:57 do we get distributed compute to really sing in crypto land?
0:06:04 Absolutely. So I think so much of the attention in crypto tends to be on this metric of transactions
0:06:08 per second. And I think we would argue that that’s the wrong, I mean, it’s not even the
0:06:13 right framing because we’re not talking about just a ledger that processes payments. We’re
0:06:19 talking about a computer, we’re talking about a decentralized fabric for general computation.
0:06:23 So the right metric is not really transactions per second, it’s really instructions per second.
0:06:28 How many, how many instructions of some computation can you process in any given period of time?
0:06:33 And that’s kind of what people know as, know as as throughput. And so that’s one of the
0:06:38 metrics for scalability that that matter when it comes to compute. The other one is the
0:06:42 latency to finality. It’s like, how do we know that the computation was done and that
0:06:49 it can no longer be reverted, that it can no longer, that its output was final and that
0:06:54 nothing can happen that could reverse it and have it be something different. You can trust
0:06:59 that that outcome is settled. So that’s latency to finality is how much time do you have to
0:07:05 wait before that happens. And people talk all the time about how Bitcoin has terrible latency
0:07:10 to finality, you have to wait 60 minutes before you can be reasonably sure that your
0:07:14 payment is final. So that’s another another access. And then the final one when we talk
0:07:22 about scalability of computation is what is the cost per instruction? How much do I have
0:07:26 to pay for that transaction? How much do I have to pay for just an arbitrary computation
0:07:31 on Ethereum or on some of the more general platforms for for computation? And the reason
0:07:35 obviously that all of this matters is because the kinds of applications that we want to
0:07:44 build will just require far greater scalability and also will require far lower cost to really
0:07:49 to really work. And I think it may be helpful to just exemplify what those applications
0:07:53 are. I think some of the some of the things that we we are seeing already in the world
0:07:59 of of Ethereum kind of the Ethereum ecosystem is maybe the richest so far in terms of actual
0:08:04 developer activity on top. So we’ve seen kind of the emergence of this parallel financial
0:08:12 world where you have things like like stable coins, which are price stable cryptocurrencies
0:08:18 that have some logic that modulate the supply of the of the token to keep it stable to some
0:08:23 external reference like the US dollar. And then on top of that, people build things like
0:08:27 lending platforms and they build they build things like derivatives platforms, they build
0:08:33 things like decentralized exchanges where you can exchange tokens or exchange crypto assets
0:08:38 without depending on some central exchange. So all of these things that’s like one one
0:08:43 example one trend that is already happening among among many other trends that we can
0:08:48 talk about other examples later review if you think it’s helpful. But all of that depends
0:08:59 on far greater throughput, far lower latency to finality and far lower cost per per instruction.
0:09:04 It’s already just with this initial activity, we’re already seeing the limits of the current
0:09:09 of the current technology. So it’s an open problem. How do we increase throughput? And
0:09:13 for that particular question, people people like what one of the things that matters the
0:09:19 most is the delay in propagation of messages in a distributed system. That’s what ends
0:09:28 up dominating the cost of that particular problem. So people talk about the block time
0:09:34 in crypto was like how much time do you have to wait before you can append a new block
0:09:39 to the blockchain and blocks usually contain computations they contain transactions. So
0:09:44 you can lower the amount of time that you have to wait for new blocks to come along,
0:09:50 then you will process more transactions and more computations per per unit time than you
0:09:55 would otherwise. And so propagating messages in the network is is a is the dominant factor
0:10:00 is that’s what makes it slow to sort of finalize a transaction. Exactly. And the reason for
0:10:04 this is that we are building a distributed system. And so if you think about it, what
0:10:09 is the difference between a distributed system and one that’s just centralized? And that
0:10:14 is that there’s distance between the different nodes that are participating in the system.
0:10:18 So the key difference is that now there’s this additional communication cost between
0:10:24 the different nodes in the system. And that cost is also is significant because it’s bounded
0:10:29 is it like the lower bound on it is the speed of light. You cannot get faster than the speed
0:10:35 of light. So so it provides it causes this this kind of lower bound as to how perform
0:10:40 it can possibly be. And you can only get so clever before you reach that that that kind
0:10:46 of lower bound. But it is the case that today we’re still far far from from that lower bound.
0:10:49 There’s still a lot of room for improvement. Yeah, I mean, people in general pretty impatient.
0:10:53 I remember when the chip and pin system started getting deployed here in the United States,
0:10:57 it was just a couple of years ago, right? And then the you didn’t start your credit
0:11:02 card, it would take like five seconds, right? And that was a lot slower than the swipe. And
0:11:06 people were like, this is never going to work. I’m not waiting five seconds for my credit
0:11:12 card to clear. And so maybe talk a little bit about, you know, sort of what is the propagation
0:11:17 delay today? And then what’s practical to get to given sort of speed of light limitations?
0:11:20 And then what’s the target? And how do we get there?
0:11:28 Yeah, for sure. So today, I mean, there’s this there’s enormous tension between, well,
0:11:33 to back up a little bit. So there are two things that matter here. One of them is how
0:11:38 much time does it take to send a message between two two points in space? But then there’s
0:11:46 also the problem of what what influence does the size of the message have on on that amount
0:11:51 of time? And so there’s basically the two angles here are latency and bandwidth, latency
0:11:55 being the amount of time that it takes to send a message bandwidth, being how much how
0:12:00 much data can you actually fit through the pipe per unit of per unit of time. So there’s
0:12:05 a tension in this space, you can see this reflected in in say the Bitcoin like block
0:12:14 size debate. Yeah, between sort of the throughput that you can get out of a network and the
0:12:19 propagation delay that that is caused by say increasing the block size. So in the case
0:12:24 of Bitcoin, people people were talking about doubling the block size from one megabyte to
0:12:29 two megabytes. So that would increase the throughput of the Bitcoin blockchain, because
0:12:35 now you can fit twice as many transactions, and the blocks will still come at a 10 minute
0:12:42 cadence. But that would increase the propagation delay for those blocks, which would cause
0:12:46 certain miners to no longer really be able to participate because they won’t get the
0:12:49 block in time. So they would have to they would eventually end up falling out, you’d
0:12:55 end up with a more centralized system. So we see here there’s a trade off between performance
0:12:59 and decentralization, assuming that you want to keep security constant, you don’t want
0:13:04 to suffer. You still need the trust, right, you can give up on the trust, you can’t allow
0:13:10 double spending, right. So I always thought that the reason say Bitcoin was slow was the
0:13:15 proof of work was so computationally demanding. Is that still the case, or is that is that
0:13:16 a solved problem?
0:13:21 So it’s a very, very good point. So we’ve been talking so far we’ve been talking about
0:13:29 the throughput of of instructions for a blockchain, which is one of the three different dimensions
0:13:35 for scalability of compute. The third one was the cost of instruction of an instruction
0:13:41 how much does it cost to have a transaction be processed. So the cost of proof of work
0:13:49 is what ends up driving the cost of an instruction so far, so so high. Rather than I mean, like
0:13:56 so there are, there are like various different lines of work. There’s the line of work that’s
0:14:01 trying to improve the propagation of messages and to make that more efficient. So there
0:14:07 are companies like blocks route, which are building like a kind of a content delivery
0:14:13 network which has advanced computer networking technology that allows miners to propagate
0:14:18 their blocks to other miners very efficiently. And so that’ll help with the propagation delay,
0:14:19 which will help with the throughput problem.
0:14:23 So the new generation CDN that is optimized for crypto.
0:14:27 And in fact, they call it a blockchain distribution network, a BDN.
0:14:28 Oh, got it.
0:14:31 And so that’s an interesting angle and that operates at layer zero is like the networking
0:14:36 layer below the blockchain layer. And it can help any blockchain project, any blockchain
0:14:42 project that builds on top of it will benefit from faster propagation of messages.
0:14:45 And the classic internet definitely needed this. Like it’s impossible to imagine the
0:14:50 internet without a CDN, right? You’d be waiting a lot longer for almost anything without that
0:14:54 layer of infrastructure. So that makes sense. So there’s sort of a CDN layer.
0:14:57 Yeah. And then, and then there are people who are working on this latency to finality
0:15:01 dimension, which we also talked about, which is how much time do you have to wait before
0:15:10 your message or your update, your computation is final. And so that is a consensus problem.
0:15:16 How do we agree that the update is final? How do we agree that something can no longer
0:15:23 be reversed? So proof of work is a probabilistic consensus algorithm in that there is always
0:15:28 some probability that whatever update to the ledger was performed could be reverted at
0:15:35 some point in time later. And the key aspect there is that the more time passes, the less
0:15:40 likely it becomes that that update gets reverted. But it’s always probabilistic. And this is
0:15:43 why you kind of have to wait 60 minutes before you know that it’s final, because that’s
0:15:48 the point at which that probability becomes so minimal that you can effectively trust
0:15:49 that it won’t be reverted.
0:15:54 But there is innovation in consensus algorithms that are better than that, that are not probabilistic
0:16:01 and that are actually deterministic and are final on a far shorter time span.
0:16:05 Yeah. And you need both, right? You need non probabilistic and you need fast, right?
0:16:10 So I, you know, everybody talks about the, the Ethereum contract things underlying things
0:16:16 like, hey, when you go rent an Airbnb, that lock will open because I know it’s you. There’s
0:16:20 a smart contract that governs the, oh, you’re allowed to stay here tonight. No one’s going
0:16:26 to wait there 60 minutes for that. And so is it feasible? Are we on a path to basically
0:16:31 enable use cases like that where like, I’ve got my smart key and I’m in front of the Airbnb
0:16:35 and like in seconds that thing is going to open because the contract clear, is that possible
0:16:38 or is that not quite possible yet? We don’t know the path.
0:16:43 I think, I mean, we do see, we do see a path. I think that, so given the improvements on
0:16:49 the networking layer with companies like BoxRoute, improvements on the consensus layer with companies
0:16:53 like DFINITY and Ethereum 2.0 and Cosmos and Polkadot, there’s like a large number of people
0:17:00 working at that level. And then finally, improvements on the cost per instruction. Similarly, proof
0:17:05 of stake and other consensus algorithms don’t use the expensive proof of work that the original
0:17:09 blockchain’s used. And so that can also come down as you can see a world where, where this
0:17:15 does come down to a degree that it becomes fairly practical for everyday use for things
0:17:23 like kind of like a quick lighthearted interactions between people or between people and machines.
0:17:29 And so I think that that is certainly possible and I think we’re on our way. But I think
0:17:35 it’s worth noting, decentralized systems will always be more expensive and less performant
0:17:41 than centralized ones. There’s just an inherent tradeoff there and there’s an inherent cost
0:17:47 to decentralizing a computer system. And so it won’t replace everything. There will
0:17:53 be applications that will always make sense for a centralized world, for a centralized
0:17:57 world. And there will be some applications for which decentralization very much does
0:18:03 make sense. And those are the ones where trust is the key differentiator, where trust is
0:18:08 the bottleneck to scale. That’s where decentralized systems will shine.
0:18:12 I want you to give me a couple of examples of sort of applications where trust is the
0:18:18 key as opposed to performance or cost or whatever. But before we do that, I want to talk about
0:18:23 this notion of proof of work, transitioning to proof of stake, because this is super important.
0:18:30 I read all the time that Bitcoin mining is consuming some known fraction of the world’s
0:18:35 electricity because the math is so hard to actually do one of these proof of works, right?
0:18:41 It has to go similar to public key cryptography. And so what’s happening here? Like, how do
0:18:45 we get on a path where we’re not consuming all the world’s electricity doing these proofs?
0:18:51 Yeah. So the key goal for crypto networks is to build trust in a way that is bottom-up
0:18:55 and that does not depend on some central authority. And so as a result, you have to figure out
0:19:01 a way to make the network be self-policing and to kind of have an incentive structure
0:19:07 that makes its members police one another in a way that the entire network kind of works
0:19:12 and sort of proceeds according to people’s expectations. So in a sense, you have to make
0:19:20 it the rational equilibrium to play by the rules of the game rather than to defect and
0:19:26 profit in some way that is against the rules and that kind of figures out a way to game
0:19:31 the system. And so in Bitcoin, one of the key ways this worked was through this proof
0:19:35 of work. So maybe talk a little bit about how did it work? Why did it consume so much
0:19:40 electricity? And then where are we going? So the key problem that crypto networks have
0:19:46 to solve is figuring out who gets to participate because there’s no one central party who is
0:19:50 able to decide who gets to participate and who doesn’t. That’s the entire point we want
0:19:56 to do away with that. And so proof of work does this by requiring every participant to
0:20:01 compute an expensive proof of work. It’s a computation that’s done on top of every block
0:20:05 that they want to add to the blockchain. So that’s an extrinsic resource that they have
0:20:11 to come across, they have to procure to be able to participate and it prevents any one
0:20:18 person from completely monopolizing the system and from having unilateral ability to modify
0:20:22 the underlying blockchain. That of course is very expensive because you have to come
0:20:27 across all of this computational power in order to participate. So proof of stake says
0:20:31 something different is instead of making the resource that you have to come across and
0:20:37 you have to procure be extrinsic to the system, why not make it something that’s intrinsic
0:20:44 to it, namely why not make it a crypto asset, why not make it a token that you have to own
0:20:50 in order to buy your participation in the system. So what proof of stake does is it
0:20:57 says if you own 2% of the tokens in the network, then by and large on average you’ll have 2%
0:21:01 of the say in what blocks get to make it onto the blockchain and which ones don’t. But now
0:21:09 because of the asset itself, the resource that you need to be in procession of in order
0:21:14 to participate, because it’s no longer extrinsic to the system, it’s no longer a resource that
0:21:19 is sort of a physical resource like electricity and rather it’s entirely virtual, now the
0:21:25 cost of actually participating in consensus and making the entire network work in real
0:21:31 terms comes down dramatically. And it’s just secure or at least theoretically can be made
0:21:37 just to secure and that’s a controversial statement but I will sort of stand by it.
0:21:42 But it’s less expensive and so from a cost per instruction, on a cost per instruction
0:21:47 basis it’ll be much more performant than a proof of work system.
0:21:51 So if I could restate that it sounds like in Bitcoin with its proof of work I had to
0:21:56 bring electricity, consume the electricity, do this hard math and that was how I sort
0:22:00 of entered the system and participated and my reward is a minor for burning all this
0:22:06 electricity as I get paid in Bitcoin. In proof of work I’m bringing basically tokens, sorry,
0:22:10 in proof of stake I’m bringing the tokens themselves, I’m not consuming electricity I’m just bringing
0:22:17 the tokens themselves and I’m by virtue of my ownership of the tokens I can participate
0:22:23 in a proof of stake that delivers the same trust properties as proof of work without
0:22:25 burning all the electricity.
0:22:26 Exactly.
0:22:31 Got it. Got it. So it sounds like all of these things need to come together for us to build
0:22:36 sort of distributed compute in this new world, right? We need the new CDNs, we need this transition
0:22:40 to things that look like proof of stake so we’re not consuming all this electricity.
0:22:46 Any other big innovations that need to happen in this space to bring the transaction cost
0:22:48 down and the transaction speed up?
0:22:54 I think those are the big ones. So we’re talking about the three pillars of computation, well
0:23:00 the three pillars of scalability of computation, there’s throughput, there’s latency and then
0:23:03 there’s a cost per instruction and so we kind of addressed all three, there’s companies
0:23:06 that are working in all three and I think these are very much still open problems and
0:23:12 there’s just a lot of green field for exploration and so I will tell you Google engineer, this
0:23:17 is where it’s exciting, this is where your skills as a sort of distributed systems engineer
0:23:23 or machine learning expert can kind of leverage those skills to figure out some of these open
0:23:24 problems.
0:23:29 Got it, so if I’m motivated by doing things like I want to create a better TCP/IP, I want
0:23:34 to create a better HTTPS and oh man I’m just too late to the party, like I arrived when
0:23:38 all those protocols were already settled, like you’re saying this space is for me because
0:23:42 a lot of the problems haven’t been settled yet, right? A thorny problem, unsettled, big
0:23:43 world impact.
0:23:48 Yeah, even if it’s been 10 years since the publishing of the Bitcoin white paper, this
0:23:55 is still very early days, because I think that it’s only recently that people have begun
0:24:00 to conceive of blockchains as computers as opposed to just payment systems, so the emergence
0:24:09 of Ethereum was in 2014 and it’s only really been five years or four years really of people
0:24:14 thinking of blockchains in this way and so it’s very early days, the space is very nascent
0:24:17 and there’s just a lot of work to do, great.
0:24:21 So perfect time, why don’t we sort of move on to part two, so we’ve talked about distributed
0:24:28 compute, let’s talk about distributed storage and so we started with the Google photos example,
0:24:34 I kind of trust Google to have all my storage, all my photos, but to do that they have huge
0:24:38 servers with lots of hard drives in them scattered around the world, it’s pretty expensive and
0:24:44 so if I was a startup trying to mount a frontal assault against that, I kind of only have
0:24:50 two choices, one is raise like a trillion dollars and try to duplicate their infrastructure,
0:24:58 put data centers everywhere, points of presence, or I could do what the distributed crypto community
0:25:02 is trying to do which is convince you to lend me a bit of your hard drive space.
0:25:09 The key reason that we need a decentralized layer of storage is because it itself will
0:25:14 be a foundational building block for this decentralized world computer that we’re talking
0:25:19 about and so in order for some of these applications that we’ve talked about that are not really
0:25:26 possible to build on top of a centralized architecture to really work, we need the full extent of
0:25:31 a computer that works in this way and so if we had a centralized storage layer instead
0:25:38 of a decentralized one then that would be the weakest link, it would dilute the promise
0:25:44 of the decentralized layer of computation if you don’t have all of the pieces themselves
0:25:49 being decentralized and so that’s why it’s important because we want to enable these
0:25:56 applications that kind of depend on decentralization for trust and it’s not so much to compete
0:26:02 head on with Amazon because the economics are different, as we said like decentralizing
0:26:08 a system always comes with the cost and so it won’t make sense for just storing your
0:26:14 photos if storing your photos is something that Amazon/Google can do and if there’s
0:26:19 no trust dimension to doing that, maybe if you care deeply about your photos not ever
0:26:24 being seen by anyone but yourself or by anyone but your close friends then maybe you can
0:26:28 imagine using a different kind of architecture, one that’s maybe more decentralized, but for
0:26:32 that kind of use case I imagine sort of the centralized data center model, it works very
0:26:38 well and they’re not paying the decentralization tax and so it’s always going to be cheaper
0:26:45 for them to store your photos but there are interesting opportunities, so for example
0:26:48 there’s a project called Filecoin, there are a number of others too that are working in
0:26:52 the same space, one of them is called SIA, another one is called StoreJ and they are
0:26:59 trying to build decentralized marketplaces for storage, so the idea is yes I can rent
0:27:05 some of your idle storage space on your laptop and pay you for that storage in Filecoin and
0:27:11 the reason that that’s now possible is because I can now trust that you will actually store
0:27:15 my files and you can trust that I will pay you for that storage even if we’re complete
0:27:20 strangers and reside across the world from one another because of the cryptographic guarantees
0:27:23 of the underlying protocol.
0:27:29 And so that’s important because previously without crypto and without blockchains that
0:27:34 would have been a very difficult interaction to coordinate and it would have been very hard
0:27:39 for us to establish trust from halfway across the world and make that exchange happen.
0:27:44 So it’s a marketplace that now emerges where previously it couldn’t have and it gives us
0:27:54 this property that no one controls this sort of layer of storage and we can use it in conjunction
0:27:58 with a computation layer to build applications that are fully decentralized and that are
0:28:07 unstoppable and kind of run in their own right and therefore command greater trust than
0:28:09 applications that are centralized.
0:28:13 So I remember before the crypto craze there were definitely startups that were trying
0:28:17 to do that do this exact thing which is sort of let’s share hard drive space.
0:28:22 I remember there were backup companies that basically say your price would be I’ll make
0:28:26 up a price $20 per gig per month but if you contribute your own hard drive space your
0:28:31 price is $10 per gig a month or whatever and making up those numbers but they never really
0:28:32 got to scale.
0:28:39 So what are the advantages of doing this with sort of a cryptographic protocol as their
0:28:43 intermediary as opposed to just hey there’s a company and there’s a service and there’s
0:28:48 a price chart and please participate right and we’re going to sort of transact value
0:28:50 and fiat currency.
0:28:55 Yeah, well I think that the key difference is that those companies were operating on
0:29:03 the assumption that the value at here is an economic that you actually that is kind of
0:29:12 like Uber you’ll tap into all of this unused storage space that previously was inaccessible
0:29:17 that you’ll offer that at a cheaper rate but I think that in the end because storage is
0:29:24 the most commoditized of computational resources and because there’s just so strong economies
0:29:31 of scale that benefit companies like Amazon as they build data centers the economic argument
0:29:32 just doesn’t doesn’t work.
0:29:37 So the reason that we need decentralized storage networks is not because they’re going to reduce
0:29:43 the price of storage by orders of magnitude at least for most kinds of files and for most
0:29:47 use cases I don’t believe that that’ll be the case.
0:29:53 The value proposition is that again we now no longer have this central entity that’s
0:29:57 controlling the storage on this network and so for the kinds of applications that depend
0:30:02 on that the kinds of applications that really cannot be built unless you have that you just
0:30:04 have no other option.
0:30:05 Yeah.
0:30:09 So you would pay the additional cost you pay a higher price for storing your files and
0:30:12 file coin because that matters.
0:30:17 Is there a privacy argument here which is the because it’s decentralized for instance
0:30:23 there’s nobody to give a government subpoena to to say I want to see your files.
0:30:29 I think privacy comes into it to some extent but I think it’s a little bit orthogonal because
0:30:33 you can imagine encrypting your files before storing them in a centralized service.
0:30:34 Yeah.
0:30:35 Fair.
0:30:40 So you there are ways of building privacy into into sort of existing centralized storage
0:30:41 networks.
0:30:42 Okay.
0:30:46 And then what are some of the challenging computer science things about building these
0:30:51 and so if there was proof of work for computation what are the proof of in the space.
0:30:52 Yeah.
0:30:55 Well the biggest one is trusting that the people who are claiming to be storing your
0:30:58 files actually are storing your files.
0:31:04 So there’s this line of work that’s been spearheaded by the people at Filecoin and by
0:31:11 Stanford’s cryptography lab, Dan Bonet’s lab and sort of people like Confisch and Benedict
0:31:18 Boone’s underneath him have done a lot of work on figuring out how to create cryptographic
0:31:20 proofs of retrievability.
0:31:26 How can I prove to you that I actually am storing the files that I’m claiming that I
0:31:27 am storing.
0:31:28 Right.
0:31:29 And it’s super interesting.
0:31:33 It’s extremely cutting edge and it’s basically at the heart of how you make a system like
0:31:34 this work.
0:31:35 Yeah.
0:31:39 So you basically have to catch the pretenders which is you don’t want somebody to be able
0:31:44 to say yes I’ll store your files and then not actually store them right because it would
0:31:49 be cheaper for them not to store them right and so these sort of proofs of retrievability
0:31:51 are basically ways to catch the pretenders.
0:31:52 Exactly.
0:31:53 Right.
0:31:54 And it’s sort of a mathematical fashion.
0:31:56 So yeah, you actually had a conversation with Ben Fisch on this.
0:32:01 So for people who are interested in exploring this topic further, there’s a couple YouTube
0:32:02 videos.
0:32:03 Awesome.
0:32:04 So, okay.
0:32:05 So we’ve talked about distributed compute.
0:32:07 We’ve talked about distributed storage.
0:32:10 I guess the third leg is now networking like what’s happening here.
0:32:14 Actually this kind of reminds me, do you remember the company Phone?
0:32:20 This is sort of back in 2006 and the idea was I could buy a Wi-Fi router from this company
0:32:23 called Phone and then I could sort of do one of two things with it.
0:32:29 One, I could sort of offer it in Linus mode which is I gave it away free Wi-Fi access
0:32:30 right.
0:32:34 Anybody that came to my house with my Wi-Fi router, you could access my Wi-Fi for free
0:32:40 and then in exchange I could access anybody else’s Wi-Fi access point for free.
0:32:44 So that’s mode one or I could be in bill mode and in bill mode basically I would say look
0:32:52 my Wi-Fi is available to you but you’re going to rent it for two bucks and in exchange for
0:32:56 that I would have to pay to access other people’s Wi-Fi.
0:33:01 So I could be in open source mode or I could be in rent seeking mode and it was this attempt
0:33:08 to basically create a distributed ISP out of millions and millions of wireless.
0:33:09 So it’s something wireless access points.
0:33:12 Is something similar going on in crypto today?
0:33:13 Absolutely.
0:33:19 So the difficulty with those efforts has often been just the problem of density and the problem
0:33:20 of incentives.
0:33:26 So how do you get enough people to offer up hardware that forwards packets and provides
0:33:32 bandwidth within a particular geographic region to make it make sense and to make it work
0:33:39 and to be at all competitive with sort of the more kind of centralized top-down internet
0:33:41 backbone infrastructure that we rely on.
0:33:46 And so there are a number of projects that are, it’s very early because this is actually
0:33:48 probably one of the hardest problems in this space to tackle.
0:33:52 How do you decentralize even the networking layer, the communication between different
0:33:58 nodes and to not have it depend on centralized internet infrastructure.
0:34:05 So people are talking about incentivized mesh networking protocols where you can earn cryptocurrency,
0:34:13 you can earn the asset that’s native to a particular protocol by setting up a router.
0:34:17 This router can be a normal router that just forwards packets but it could also be wireless
0:34:24 and provide a different layer of connectivity that otherwise, that essentially makes the
0:34:31 networking layer more robust and more resistant to censorship and perhaps even more performant
0:34:35 if you have just greater connectivity to the people you want to interact with.
0:34:40 So yeah, I think this is one of these problem areas that’s fairly far out because it kind
0:34:46 of depends on the other two building blocks and it has its unique challenges because now
0:34:49 we’re talking about bringing hardware into the picture.
0:34:51 That’s always a whole other kind of worms.
0:34:56 But it is very interesting and I think in the end, it’ll also be a piece of the puzzle.
0:35:03 So that’s one angle to it, it’s decentralizing networking and then the other angle is making
0:35:08 networking itself just more performant for the use cases of decentralization.
0:35:12 So we talked a little bit about the CDNs for blocks.
0:35:16 So that kind of falls into that category, into this category as well.
0:35:17 Got it.
0:35:20 So those are the key ingredients that you need to build a computer, right?
0:35:24 You need compute, you need network, you need storage and it looks like there’s sort of
0:35:27 efforts underway in all of these things.
0:35:32 Let’s assume for a second that time has gone by and protocols and sort of Darwinian fashion
0:35:38 have competed and a couple winners have emerged and these things look more like solved problems.
0:35:43 So now the exciting opportunity is, okay, now we can build killer apps on top of the
0:35:47 blockchain computer and so maybe talk to me about what is the community most excited about?
0:35:50 What kinds of apps are you going to build?
0:35:53 Because as you’ve been pointing out, it’s not going to be the straightforward replacements
0:35:55 for the things that we know and love today.
0:36:01 It’s not like instantly the replacement for Airbnb or Google Photos or Lyft because those
0:36:04 systems don’t have to pay the decentralization tax.
0:36:08 It’s probably going to be another class of apps at least to begin with.
0:36:12 That is the killer question and I think as with any new technology it is very difficult
0:36:16 to predict what applications will be the most impactful.
0:36:22 I think one reason to believe that the kind of innovation that we’ll see will be enormous
0:36:28 is that everything happens, all of the code that’s written in the space ends up being
0:36:29 open source.
0:36:31 And so as a result the ideas are out there.
0:36:36 People share their ideas with other teams, other teams sort of build on top of one another’s
0:36:37 ideas.
0:36:43 And so the kind of innovation that we’re likely to see is just combinatorial in nature and
0:36:49 likely more explosive and will accelerate more quickly than it has for previous waves
0:36:52 of computing and previous waves of technology.
0:36:59 We do have this kind of decentralized world computer that is a kind of computational fabric
0:37:04 on top of which applications can run and it is unified in that one application can easily
0:37:06 talk to another.
0:37:10 Then we have the possibility of composability of applications.
0:37:16 So not only do we have the sharing of ideas that are just available to people because
0:37:22 by virtue of being open source, but we also have the actual composability of running code.
0:37:25 Code that runs on top of this computational fabric that builds on top of the code that
0:37:27 other people have built.
0:37:32 And this kind of composability will just fuel the flame of combinatorial innovation even
0:37:33 further.
0:37:38 So I feel like the kinds of applications that we’ll see as a result are fundamentally
0:37:40 impossible to predict.
0:37:44 But I will say, I think the kinds of things that we’ve started to see, the kinds of applications
0:37:48 that seem to be working so far, and it’s still very early and they’re working only in kind
0:37:56 of niche, within niche communities, are ones where trust is the bottleneck to scale.
0:37:59 So I think the most obvious one began with Bitcoin.
0:38:08 It’s attempting to be money and the only way that you would trust a program that maintains
0:38:15 a ledger of tokens that claims that those tokens should be money is if that ledger isn’t
0:38:20 in the control of any one entity or any one individual.
0:38:27 I guess you would trust the central government, maybe, but you would not trust a company to
0:38:28 do that.
0:38:31 So you wouldn’t have been able to build Bitcoin on top of Amazon.
0:38:35 Bitcoin is like one example of an application that you can build and a bunch of the applications
0:38:41 that have worked so far in the Ethereum ecosystem primarily have been financial in nature, have
0:38:47 been things that build on top of that initial idea, so things like lending platforms, derivatives,
0:38:54 exchanges, things that depend on trust for them to really take off.
0:38:59 But we’ve also started to see other applications that benefit from this feature.
0:39:05 I think gaming is an interesting one where you can imagine taking the existing world
0:39:09 of gaming, you can imagine, for example, a world of Warcraft where people have significant
0:39:14 investment in their character and in the gear that they have and in the lives that they
0:39:22 live within these games, taken to a whole other level where you actually own your character
0:39:27 and you own the gear for your character and you can take your character and gear out of
0:39:33 the game and maybe into another game because you now have this interoperable trustworthy
0:39:36 fabric of computation that other developers can build on top of.
0:39:43 So that kind of investment in your personality and in your character in the game is unlike
0:39:44 what we’ve seen in gaming before.
0:39:48 So this could take gaming just to a whole other level and that could be a very interesting
0:39:49 set of applications.
0:39:53 But it very much depends on these three building blocks, we need scalability before gaming
0:39:55 can really take off.
0:40:00 And we’ve seen examples of this, I think CryptoKitties is one where people became very invested
0:40:08 in owning this digital collectible, which is something that is fundamentally new.
0:40:13 Never before would you be able to directly own something that’s digital is the first
0:40:14 time that that’s possible.
0:40:18 I love this idea of being able to take sort of a high level character that I’ve developed
0:40:23 in one game and moving it to another because, you know, look, essentially a high level character
0:40:27 in say, World of Warcraft is the ultimate proof of work, right, which is I had to do
0:40:31 a lot in order to get this character to be super high level.
0:40:34 And now I’m kind of stuck in World of Warcraft, which is great if I want to play more World
0:40:38 of Warcraft, but like it’d be awesome if I could take my proof of work and move it to
0:40:39 another system.
0:40:40 Absolutely.
0:40:41 Yeah.
0:40:46 Yeah, I mean, I think there’s a story about how Vitalik, part of Vitalik’s inspiration
0:40:51 for starting Ethereum is having, I’m not sure which game it was, but it was like some
0:40:58 gaming platform that revoked his ownership over, over like a key, a key item in the game.
0:40:59 There it is.
0:41:02 That made him like, made him like so, so mad, right.
0:41:06 This is the problem with centralization, right, which is you have a company operating
0:41:08 a game and they can do whatever they want with the game, right.
0:41:13 One change to the terms of service and all of a sudden your proof of work is basically,
0:41:14 it’s invalid.
0:41:15 Exactly.
0:41:16 Yeah.
0:41:17 That would make you mad.
0:41:20 Well, if you think about sort of trust being the key feature, I mean, there’s so many
0:41:26 sort of, you know, properties that we think about on the internet that are essentially
0:41:27 sort of brokers of trust, right.
0:41:33 So LinkedIn is sort of the trusted entity to manage your resume and present your resume.
0:41:37 And eBay is sort of the trusted marketplace where the sellers or Etsy is sort of the trusted
0:41:42 place where you sort of send money and expect stuff, right.
0:41:49 There’s Airbnb and Lyft, and so like trust seems like a super powerful primitive for
0:41:50 creating killer apps.
0:41:51 Yeah, definitely.
0:41:58 And I think the web 2.0 world has figured out how to bootstrap trust in a way that
0:42:05 depends on things like identity and reputation, where there’s social capital associated with
0:42:08 your track record on the internet.
0:42:16 So things like reviews on Yelp or things like reviews on or stars on Uber or number of likes
0:42:21 on Twitter and number of followers on just generally social media.
0:42:28 These things are, this is like the mechanism for trust that’s used in web 2.0.
0:42:30 And I think crypto is orthogonal to that.
0:42:36 Crypto today has no sense of identity, that people are pseudonymous, people can create
0:42:38 multiple addresses and pretend to be different people.
0:42:43 People can abandon identities that maybe have a bad reputation and move over to new identities
0:42:45 that don’t.
0:42:53 And the entire fabric of trust, therefore, depends not on social capital but rather on
0:42:54 financial incentives.
0:43:00 So it’s this orthogonal layer where you’re incentivized to behave honestly because there
0:43:03 is real money at stake.
0:43:08 And if you lie or if you behave in a way that’s not in accordance to the rules of the protocol,
0:43:10 then there’s something that you will lose as a result.
0:43:17 So there’s sort of financial capital and financial incentives as a way of bootstrapping trust
0:43:20 and then there’s social capital as a way of bootstrapping trust and I think that’s one
0:43:29 of the key differences between the web 2.0 world and the now web 3.0 crypto-enabled world.
0:43:34 And what will be very interesting to see is the two models coming together.
0:43:36 So that’s something to kind of look out for.
0:43:41 And it seems like Keybase is sort of an early attempt at that, which is on the one hand
0:43:46 you had all of these private keys that represented you in these cryptographic networks and on
0:43:50 the other hand you had sort of your Twitter profile and your LinkedIn profile and your
0:43:56 Facebook profile and Keybase sort of bridged them.
0:44:00 What other things do you think we will see in this space of sort of making identity more
0:44:01 seamless?
0:44:02 Yeah.
0:44:04 I think Keybase is a key one.
0:44:13 A key problem is how do you map a real human individual to a public key in a way that is
0:44:15 trustworthy and in a way that you can rely on.
0:44:21 So Keybase is this and I think it’s a very apropos example where you can use your existing
0:44:26 web 2.0 world identity to bootstrap your web 3.0 identity.
0:44:31 You can use your Twitter account and your Facebook account and your GitHub account and
0:44:39 your website and point them all to this cryptographic identity that you can then use to interact
0:44:43 with other people in the sort of crypto anonymous world.
0:44:47 And they can verify that that really is you because of the cryptographic assurances of
0:44:54 those connections between Twitter and so on and your public key.
0:44:55 You can take that further though.
0:45:00 I think once you do have identity in the crypto world and I think it is an unsolved problem,
0:45:05 Keybase is the first kind of attempt, but there’s still a lot to do there.
0:45:10 Once you have a solid layer of identity within crypto, that also doesn’t sacrifice privacy.
0:45:15 So it’s worth noting there’s a big trade-off there, like if you have strong identities
0:45:19 and you have less privacy and it’s kind of difficult to come to the right balance between
0:45:22 the two and it will vary per application.
0:45:26 But once you have a good system for that, then you can start building reputation systems.
0:45:30 You can even imagine like a page rank style algorithm for reputation.
0:45:41 If I trust Frank and Frank trusts Joe, then I kind of indirectly trust Joe.
0:45:46 And you can imagine kind of taking this to the whole other level to really enhance the
0:45:53 kind of trust that emerges from financial incentives with social capital and with reputation.
0:45:54 That’s very powerful.
0:45:58 There’s some of this even in things like the Facebook marketplace today, which is you see
0:46:02 somebody listing cheese or a bike or whatever and you’ll see, “Oh, this is a friend of Ali.”
0:46:06 And then that brings a level of trust to that transaction that wouldn’t otherwise exist.
0:46:07 Exactly.
0:46:12 And one of the reasons this is so important for crypto is that today, every interaction
0:46:16 in the world of crypto tends to be very transactional.
0:46:20 You don’t even know who you’re dealing with and so it really is about that one transaction.
0:46:27 It’s one-off and whenever there’s conflict, it’s a one-off prisoner/dilemma style game.
0:46:30 Whereas if you had identity and if you had reputation, you could turn all of those one-off
0:46:35 prisoner/dilemma style games into iterated prisoner/dilemma style games, which are far
0:46:36 easier to solve.
0:46:38 You have like the long view of relationships.
0:46:45 You can have a track record and rapport with the people that you interact with if you only
0:46:47 had that other layer.
0:46:50 So so many of the problems, the game theoretical problems that need to be solved for crypto
0:46:54 to work that are so hard to solve will become easier once you have this additional lever
0:46:55 to play with.
0:46:56 Yeah, that’s super interesting.
0:46:57 Right.
0:47:01 So every prisoner/dilemma type game is sort of assume perfect strangers go in and now
0:47:05 we have to sort of mathematically model what will happen, not knowing anything about them.
0:47:06 Exactly.
0:47:09 But if you threw me and you into a prisoner’s dilemma, right, like we’ll have much higher
0:47:14 fidelity predictions about what each other like, “I’m not going to squelch on you, Ali.
0:47:15 He’s a friend of mine.”
0:47:18 Especially if we know that we’re going to be in a similar kind of game in the future.
0:47:21 And it’s like if we cooperate now, then we’ll build rapport and then it’ll be easier for
0:47:24 us to cooperate in the future.
0:47:29 And if we cheat each other now, then we will kind of ruin that opportunity later on and
0:47:32 make it harder for us to cooperate down the line.
0:47:33 Super interesting.
0:47:38 So one thing before we go, I want to talk a little bit about governance, because today
0:47:43 it seems like there’s a lot of conversation in the tech community about, “Gee, maybe the
0:47:47 tech giants have gotten too big,” right, because with the stroke of a pen and one change in
0:47:52 terms of service, like all of a sudden, the rules of engagement or the winners and losers
0:47:55 in that environment are dramatically different.
0:48:00 In crypto land, the idea would be let’s not have one company which completely owns their
0:48:05 terms of service control that, there’s going to be sort of a decentralized community.
0:48:10 But we end up with some of the same questions, right, like who gets to change the terms of
0:48:11 service?
0:48:14 How do those changes come about, who proposes them?
0:48:19 So maybe talk to me a little bit about what’s happening in the community as we iterate on
0:48:20 systems of governance.
0:48:25 Yeah, you’re hitting at one of the most fundamental questions in this space, which is that if
0:48:32 you do build a system that is decentralized and that control over it does not rest with
0:48:36 any one individual, then there’s a question, well, how do you go about updating it?
0:48:39 How do you go about changing it in any meaningful way?
0:48:43 If it is going to be a complex system that adapts and evolves over time, this question
0:48:46 most certainly has to be answered in order for any of this to work.
0:48:53 So this is a question of sort of governance of protocols, and there are enormous number
0:48:57 of experiments that people are running, like different kind of approaches.
0:49:04 The canonical and sort of initial approach was out of Bitcoin, which is that essentially
0:49:07 you do just have to coordinate with all of the stakeholders, all of the people who are
0:49:14 running the Bitcoin node software in order to change the protocol.
0:49:18 And in this case, that would be that all miners, all of the people who are running the code
0:49:23 to mine Bitcoin, have to modify their software, and this is a human level process.
0:49:28 You have to call them up or you have to sort of issue an announcement saying that the protocol
0:49:31 is being upgraded and get that to work.
0:49:35 There are other approaches that people are exploring with that are more sort of they’re
0:49:38 formalized and are built into the protocol.
0:49:41 So there’s the idea of being able to vote with tokens.
0:49:50 So if I own a certain stake, a certain amount of the network, then I can use the tokens
0:49:55 that constitute that stake to vote in favor or against proposals that may be made by the
0:49:56 community.
0:50:01 It’s just another approach to decentralize governance that tries to lower the barrier
0:50:03 and tries to make it a little bit more seamless.
0:50:07 There’s an enormous set of challenges associated with that because there are possible attacks
0:50:10 where you can bribe people.
0:50:12 There is the issue of voter participation.
0:50:17 And all of the issues that you see in governance systems outside of the world of crypto, just
0:50:18 an offline government.
0:50:19 Offline?
0:50:20 Who’s going to the election?
0:50:21 How do they vote?
0:50:22 Exactly.
0:50:24 How do we prevent dead people from voting?
0:50:25 Exactly.
0:50:30 These problems have become replicated in crypto as well and they are therefore like fundamentally
0:50:33 difficult problems that have been unsolved for millennia.
0:50:39 So it’s not as if crypto will solve any of that, it’ll just have to figure out the right
0:50:44 mechanisms and the right structures to be good enough and to enable systems that are
0:50:51 decentralized to adapt and to change and evolve while striking a balance between sort of
0:50:53 evolvability and decentralization.
0:50:59 And actually, we did two podcasts specifically on this question, the question of governance
0:51:03 and crypto, that will go much, much deeper and talk about all of the challenges.
0:51:07 So if you’re interested in that topic, I recommend checking those out.
0:51:08 Perfect.
0:51:12 We’ll throw the links into the YouTube video so you can follow them easily.
0:51:15 Well, Ali, this has been super interesting.
0:51:20 There’s so many problems to be solved, there’s so many meeting computer science things to
0:51:25 be had, like how do you prove that I’m actually storing your photos instead of just pretending
0:51:28 to store your photos and collecting the money.
0:51:33 And so I guess the way I think about it is like if you have ever wished that you could
0:51:41 have been like a semiconductor engineer at Bell Labs in the 1950s or a PC enthusiast in
0:51:47 the 1970s and you were like, “I missed the 50s and then I missed the 70s.”
0:51:53 And then like if you wished you were at UIUC with Mark at the dawn of the internet in the
0:51:55 90s, like, “Look, here it is.
0:51:57 This is the new computing platform.
0:51:59 Here is your opportunity.
0:52:00 It’s not too late.”
0:52:05 And these are the times to exactly insert yourself into that conversation if sort of that’s
0:52:09 what you wish you had the opportunity to do is influence some of these protocols, these
0:52:12 incentive systems at the ground level.
0:52:13 Absolutely.
0:52:14 Awesome.
0:52:15 Fantastic.
0:52:18 So that’s it for this episode.
0:52:22 And if you liked what you saw, go ahead and subscribe to the list.
0:52:25 If you have comments, go ahead and leave them down below.
0:52:28 Maybe you could pick one thing that you were super excited about, like what problem do
0:52:32 you wish you could solve as an engineer.
0:52:33 And we will see you next episode.
0:52:42 [BLANK_AUDIO]

0:00:03 – Hi, and welcome to the A16Z podcast.
0:00:04 I’m Hannah.
0:00:06 Good data, bad data, there’s maybe no other area
0:00:09 where understanding what the evidence actually tells us
0:00:11 is harder than in health and parenting.
0:00:14 In this episode, economics professor Emily Oster,
0:00:16 author of “Expecting Better”
0:00:18 and the recently released “Cribsheet,”
0:00:21 a data-driven guide to better, more relaxed parenting,
0:00:23 does just that, looking at the science and the data
0:00:25 behind the studies we hear about
0:00:27 and make decisions based on in those worlds.
0:00:30 From whether to breastfeed your child to screen time
0:00:32 to sleep training, we talk about what it means
0:00:35 to make database decisions in these settings,
0:00:37 in diet and in health and in life,
0:00:39 like whether chia seeds are actually good for you
0:00:42 and how we can tell what’s real and what’s not.
0:00:44 We also talk about how guidelines and advice like this
0:00:47 gets formalized and accepted for better or for worse
0:00:49 and how they can or can’t be changed.
0:00:52 And finally, how the course of science itself
0:00:55 can be changed by how these studies are done.
0:00:57 – You describe yourself as teasing out causality
0:00:58 in health economics.
0:01:01 Can you give us a little primer on what exactly that means
0:01:02 and how you start going about doing that?
0:01:04 – So there are a lot of settings in health
0:01:06 and in all of those settings,
0:01:09 we have to figure out what does the evidence say.
0:01:11 And I think about some of them in this context of parenting,
0:01:14 but you can think about even questions like,
0:01:15 is it a good idea to eat eggs
0:01:17 or is it a good idea to take vitamins,
0:01:19 other kinds of health decisions.
0:01:20 And you can sort of think about there being
0:01:23 kind of two types of data you could bring to that.
0:01:25 One would be randomized data.
0:01:26 So you could run a randomized trial
0:01:29 in which half of the people got eggs
0:01:30 and half of the people didn’t.
0:01:31 And you followed them for 50 years
0:01:33 and you saw which of them died.
0:01:36 And that would be very compelling and convincing.
0:01:38 And when we have data like that, it’s really great.
0:01:40 – I mean, I kind of think of that as being the default.
0:01:42 No, is that not at all the standard?
0:01:45 – That is the gold standard, but it is not the default.
0:01:47 So many of the kinds of recommendations
0:01:48 that I look at in parenting,
0:01:50 but that you look at in general in health
0:01:52 are based on observational data,
0:01:54 which is the other kind where we compare people
0:01:56 who do one thing to people who do another thing
0:01:58 and we look at their outcomes.
0:02:00 And one of the ways in which the people differ
0:02:02 is on the thing that you’re studying,
0:02:06 but of course there are other ways that they may differ also.
0:02:06 – A million other ways.
0:02:08 – A million other ways, yes.
0:02:10 And data like that is really subject
0:02:12 to these kind of biases that the kind of people
0:02:13 who make one choice are different
0:02:16 from the kind of people who make another choice.
0:02:17 One of the things that’s very frustrating
0:02:19 in a lot of the health literature
0:02:21 is that there isn’t always that much effort
0:02:24 to improve the conclusions that we draw
0:02:25 from those kind of data.
0:02:27 – And we’re using that kind of approach
0:02:31 because of the inability to have long longitudinal studies,
0:02:33 or it does tend to be a shortcut.
0:02:35 – So I think it is both things.
0:02:38 So it is much easier, faster to write papers,
0:02:40 to produce research about that,
0:02:42 and it can be really useful for developing hypotheses.
0:02:43 – So it’s like a scratch pad almost.
0:02:46 – In the best case scenario, it’d be like a scratch pad.
0:02:48 Like let’s just look in the data
0:02:50 and see what kinds of things are associated
0:02:52 with good health or associated with good outcomes for kids.
0:02:54 And then we could imagine a next step
0:02:58 where you would analyze more rigorous gold standard.
0:02:59 And sometimes that happens.
0:03:01 So there’s one really nice example of the book
0:03:03 where this happens exactly like you would hope,
0:03:06 which is in studying the impact of peanut exposure
0:03:07 on peanut allergies.
0:03:10 So the first paper on that is written by a guy,
0:03:12 and what he did was he just compared Jewish kids
0:03:16 in the U.K. to Jewish kids in Israel,
0:03:18 and he saw that the kids in Israel
0:03:19 were less likely to be allergic to peanuts,
0:03:22 and he said that’s because they eat this peanut snack
0:03:24 when they’re being as a bomba.
0:03:28 And so then that’s like the hypothesis generation,
0:03:30 and then he went and did the thing you would really like,
0:03:32 would just say, okay, let’s run a randomized trial,
0:03:34 and let’s randomly give some kids early peanuts,
0:03:35 and some kids not.
0:03:37 And indeed, like he found that he was right.
0:03:39 So that’s like a great example
0:03:42 of like how you would hope that literature would evolve.
0:03:44 But in many of the kinds of health settings
0:03:47 we’re interested in, that you can’t do that,
0:03:49 or it is much harder to do that,
0:03:52 because the outcomes would take a long time to realize,
0:03:54 or it’s expensive, or it’s hard to manipulate
0:03:55 what people are doing.
0:03:57 And then we often end up relying
0:04:00 on these more bias sources of data
0:04:02 to draw our conclusions, not just as a scratch pad.
0:04:06 And I think that’s where we encounter problems.
0:04:08 – That’s where it gets murky,
0:04:10 and we never know whether we should eat eggs or not.
0:04:13 Yeah, and that’s exactly the area that you tend to focus on.
0:04:14 – Yeah, exactly.
0:04:16 I try to first see are there good pieces of data
0:04:19 that we can use, and then if we’re stuck with the data
0:04:20 that isn’t good, trying to figure out
0:04:25 which of the murky studies are better than others.
0:04:27 And what would you mean by better?
0:04:30 Well, it’s roughly like how good is this study
0:04:34 at controlling or adjusting for the differences across people?
0:04:38 – So you talk about kind of breaking it down
0:04:42 into both into the relationship between data and preference.
0:04:44 How do you factor in that in the healthcare system
0:04:46 where it’s so diverse, where preference
0:04:48 has such an incredible effect
0:04:51 and puts you into so many different possibilities?
0:04:53 – I think this is why in these spaces,
0:04:55 decision-making should be so personal.
0:04:59 We often run up in health and also in parenting
0:05:01 and all of these spaces into a place
0:05:04 where we’re telling people like there’s a right thing
0:05:06 there’s a right thing to do.
0:05:10 And I think that that can be problematic
0:05:12 because it doesn’t recognize this difference
0:05:14 in preferences across people.
0:05:16 – You have to basically accept the variety
0:05:17 in the system and then give a space
0:05:18 for preference in the decision-making.
0:05:20 – Yeah, but I think it’s exactly these preferences
0:05:22 that of course make it hard to learn
0:05:24 about these relationships in the data.
0:05:26 ‘Cause once you recognize that a lot of the reason
0:05:28 that some people choose to eat eggs
0:05:29 and some people choose to eat cocoa crispies
0:05:31 is that some people really like cocoa crispies
0:05:32 and some people really like eggs.
0:05:34 How can you ever learn about the impact of eggs
0:05:37 because we know there must be differences across people.
0:05:39 And I think that that becomes even more extreme
0:05:41 when we think about really important decisions
0:05:42 that people are making,
0:05:44 like the kinds of choices they make in parenting
0:05:45 or also in their diets.
0:05:48 – So can you walk us through one example like that
0:05:51 of where it was a really kind of murky gray area
0:05:53 and how you pull out the causality?
0:05:54 – The best example of this in the data
0:05:57 in the parenting space is probably in breastfeeding.
0:05:59 Let’s say you wanna know the impact of breastfeeding
0:06:00 on obesity in kids.
0:06:02 That’s a thing which you hear a lot.
0:06:06 Breastfeeding is a way to make your kid skinny and so on.
0:06:09 And so the basic way you might analyze that
0:06:12 is to compare kids who are breastfed to kids
0:06:13 who are not and look at their obesity
0:06:14 when they’re say seven or eight.
0:06:16 And indeed, if you do that,
0:06:17 you will find that the kids who are breastfed
0:06:20 are less likely to be obese than the kids who are not.
0:06:24 But you will also find that there’s all kinds of relationships
0:06:27 between obesity and income and obesity,
0:06:28 mother’s income and mother’s education
0:06:30 and other things about the family.
0:06:32 And those things correlate with breastfeeding
0:06:34 and they also correlate with obesity.
0:06:36 – So you can’t really pull apart this web.
0:06:38 – So it’s hard to pull apart the web.
0:06:39 So I would say this is an example
0:06:42 where the data is suggestive.
0:06:43 It would certainly be consistent
0:06:45 with an effect of breastfeeding on obesity,
0:06:48 but I think it doesn’t prove an effect.
0:06:50 And then you can sort of take the next step
0:06:52 and say, okay, well, do we have any data that’s better?
0:06:53 And in that example,
0:06:55 we do have one kind of randomized data.
0:06:57 But again, we run up against the limits
0:06:59 of all kinds of evidence.
0:07:02 So the randomized data on this question
0:07:05 is from a randomized trial that was run in Belarus
0:07:07 in the 1990s.
0:07:09 They randomly encourage some moms to breastfeed
0:07:09 and some moms not.
0:07:11 And so there’s a lot of good things
0:07:12 that we can learn from that.
0:07:14 – But such a specific place in time.
0:07:15 – Exactly, it’s so specific.
0:07:16 And you said like, well, you know,
0:07:21 how do I take that result to the Bay Area in 2019?
0:07:23 It’s a challenge.
0:07:24 Okay, well, is there any other within this space
0:07:28 of not randomized datas or anything that’s better?
0:07:29 And in that case, there is,
0:07:32 there are some studies that like compare siblings,
0:07:35 where you look at two kids born to the same siblings,
0:07:36 born to the same mom.
0:07:39 One of whom was breastfed and one of whom was not.
0:07:40 And then look at their obesity rates.
0:07:41 And when you do that,
0:07:43 you find there’s basically no impact.
0:07:46 So then you’re kind of holding constant like who’s the mom.
0:07:49 So if you’re a worry was that there are differences
0:07:51 across parents in their choices to breastfeed,
0:07:53 well now you’re looking at the same parent.
0:07:54 – Right, you’re normalizing.
0:07:55 – You’re normalizing.
0:07:57 And so you may think, oh, that’s great, perfect.
0:07:57 I’m totally done.
0:07:59 But of course you’re not,
0:08:02 this isn’t perfect because why did the mom choose
0:08:03 to breastfeed one kid and not the other?
0:08:04 – People are not choosing random.
0:08:05 – You had to see section one time,
0:08:07 you didn’t another time.
0:08:08 – If that were the reason that would be great, right?
0:08:11 If the reason were just like kind of worked one time,
0:08:12 didn’t work the other time.
0:08:14 If there was something that was effectively
0:08:16 a little bit random,
0:08:19 then that would be exactly the kind of variation
0:08:21 you’d wanna use.
0:08:23 But the thing you worry about is like one kid
0:08:25 is not doing well, is unhealthy.
0:08:26 So the mom chooses not to breastfeed
0:08:30 or chooses to breastfeed to try to make them healthier.
0:08:32 Those are the kind of things where there’s some other reason
0:08:34 that they’re choosing differences in breastfeeding
0:08:36 which has its own effect on the kid’s outcomes.
0:08:41 So you kind of like some of what I try to do in the book
0:08:43 is sort of like put all of these pieces together
0:08:46 and kind of like look at them
0:08:50 and think about them all as a sort of totality of evidence
0:08:52 and just think like how compelling is this altogether?
0:08:55 – It sounds almost like sifting, like using a sifter.
0:08:58 You take all this very murky data,
0:09:00 very variable from all sorts of different contexts
0:09:02 and like put it through the sifter of like
0:09:03 this kind of data, this kind of data
0:09:06 and then match it all up and say, okay, what do we have left?
0:09:08 And then therefore, and then hand that over and say,
0:09:11 and now you make the decision based on this.
0:09:12 – Based on this.
0:09:14 – Right, here’s kind of what we can be more or less
0:09:15 or less sure about.
0:09:17 – You talk a little bit about the idea
0:09:20 of constrained optimization as being very important.
0:09:23 Can you explain what that means and how that plays out?
0:09:25 – In economics, we think about people
0:09:27 optimizing their utility function.
0:09:28 The idea is that you have a bunch of things
0:09:30 that make you happy, that’s your utility.
0:09:33 They produce your utility and you want to make the choices
0:09:35 that are going to optimize your utility.
0:09:39 They’re going to give you the most amount of happiness points,
0:09:40 eudals, eudals.
0:09:44 It’s really, it’s a very warm and fuzzy.
0:09:46 – Yeah, I feel like I’m gonna go home and use that.
0:09:47 – Absolutely.
0:09:49 – Like you gave me some eudals today.
0:09:53 – But we also recognize that people have constraints.
0:09:54 In the absence of constraints,
0:09:58 like having money to buy things or time to do them,
0:10:00 people would just have an infinite amount of stuff.
0:10:02 That’s the thing that would make them the most happy.
0:10:04 And so, but when you’re actually making choices,
0:10:07 you’re constrained by either money or time.
0:10:09 And in the book, I talk a lot about this
0:10:12 in the context of time, that you’re as a parent,
0:10:15 you’re making choices, and you have some preferences
0:10:16 and things you would like to do,
0:10:19 but you are also facing some constraints.
0:10:22 – But is there, is information flow kind of what,
0:10:25 and the data itself a constraint in that regard?
0:10:27 Is that a, because it’s so piecemeal,
0:10:29 the information you get.
0:10:30 That feels almost totally random.
0:10:32 Like some media story picks up on something,
0:10:34 you tend, you know, some tidbit, you hear some,
0:10:36 unless you’re like systemically
0:10:38 studying a graduate seminar on parenting,
0:10:41 which none of us do, you know, then it is random.
0:10:44 – Yeah, and I think we wouldn’t necessarily think of that
0:10:46 as in constraints, because of course in our models,
0:10:49 people are fully informed about everything all the time.
0:10:52 That’s one of the great things about the models.
0:10:52 – But in real life?
0:10:55 – But in real life, yeah, I think people face constraints
0:10:58 associated with just not having all the information.
0:11:01 And, you know, also the fact that this,
0:11:04 these kind of information, like whipsaws over time,
0:11:07 that you know, you get one piece and then you kind of,
0:11:10 the next day there’s a different piece of information
0:11:12 and we have a tendency to kind of
0:11:15 a glom onto whatever is the most recent thing
0:11:16 that we have seen about this,
0:11:19 as opposed to what is the whole literature
0:11:21 over this whole period of time say.
0:11:24 – Right, you say, you have a great quote where you say,
0:11:26 in confronting the questions here,
0:11:28 we also have to confront the limits of the data
0:11:29 and the limits of all data.
0:11:30 There’s no perfect studies,
0:11:33 so there will always be some uncertainty about conclusions.
0:11:35 The only data we have will be problematic.
0:11:37 There will be a single not very good study
0:11:39 and all we can say is that this study
0:11:41 doesn’t support a relationship.
0:11:44 So it feels kind of hopeless.
0:11:46 I loved when you talked about the first three days
0:11:47 of when you brought Penelope home
0:11:50 and it really brought that back for me
0:11:52 as I was just this dark room
0:11:54 that you’re kind of alone making these decisions.
0:11:57 How do you even begin to see this data,
0:12:00 you know, as a decision making practice?
0:12:02 Like how does that translate?
0:12:04 – There are pieces where it’s easier,
0:12:08 where the data is better and makes it is clearer
0:12:10 about what you need to do or what the choices are.
0:12:12 You will be making many choices
0:12:17 without the benefit of evidence or data or very good data.
0:12:19 I think part of what makes some of this parenting so hard
0:12:21 is that for those of us who like, you know,
0:12:24 evidence and facts and it’s hard to accept,
0:12:26 I’m just going to have to make this decision
0:12:29 basically based on what I think is a good idea.
0:12:30 – Based on my gut.
0:12:31 – Based on my gut.
0:12:36 – And, you know, maybe based on my mom and, you know.
0:12:37 – Which is a sample size of one.
0:12:39 – A sample size of one.
0:12:41 And, you know, maybe if you have like a mother-in-law
0:12:43 and father-in-law, it’s like a sample size of two,
0:12:46 but that’s kind of, that’s kind of it.
0:12:48 And I think that that’s really scary,
0:12:50 especially when the choices seem so important.
0:12:52 – Yeah, I mean, but it feels like, you know,
0:12:54 that’s kind of at heart what you’re trying to do, right?
0:12:56 Is like to translate and to give tools
0:12:58 in this decision making place.
0:13:02 So how would you begin to systematize that?
0:13:06 I mean, is there a way to bridge that gap better
0:13:07 in the system?
0:13:09 – I think that it would be helpful
0:13:12 if more information was shared.
0:13:14 So I think a lot of these things,
0:13:17 there is a lot of information that is contained
0:13:21 in people’s experiences that we are not using
0:13:23 in our evidence production.
0:13:27 So in the book I talk about like the sleep schedule, right?
0:13:30 So you’re sort of told as a parent, like, oh, you know,
0:13:32 this is kind of roughly like around six weeks,
0:13:35 your kid will start sleeping like longer at night,
0:13:38 but there’s no, the information that’s sort of
0:13:42 typically conveyed to people is not a range.
0:13:44 It’s just like around six weeks-ish,
0:13:46 you know, that’ll start to happen.
0:13:48 But the truth is like, yeah, that’s kind of right,
0:13:51 but it’s, if you look at data on when that actually happens,
0:13:54 it’s pretty, it’s a pretty wide range.
0:13:57 And I think part of what is so stressful
0:14:00 about this early, these like early parts of parenting
0:14:03 are that it’s very hard to understand
0:14:06 whether what you’re experiencing is like normal.
0:14:08 And I think if you could understand, like, yeah,
0:14:11 most kids don’t do this thing at this time
0:14:13 or most parents have this experience or-
0:14:15 – The way the graph plots kind of.
0:14:16 – Yeah. – A little more broadly.
0:14:17 – Exactly.
0:14:19 What is, I think that would be, that would be super helpful.
0:14:21 And that’s a place where I can imagine,
0:14:23 you know, data collection helping, right?
0:14:27 You know, we have a much more of an ability at this point
0:14:30 to like get information about what is happening
0:14:33 with our kids, what’s happening with, you know, with our health.
0:14:35 There is a sense in which that could be helpful
0:14:39 in just setting some norms for the normal,
0:14:42 the standard variation across people.
0:14:45 – So looking at the variation and providing that
0:14:46 as like a piece of the information.
0:14:47 – As a piece of the information.
0:14:48 – Here’s also the variation on that.
0:14:49 – Yeah. – Yeah.
0:14:51 – Yeah, and I think that is kind of part of like
0:14:53 generating the uncertainty and sort of showing people
0:14:55 like what are the limits of the data, right?
0:14:58 That how sure are you that this should happen at this time?
0:15:00 – Not just how sure, but like what are some of the
0:15:02 like other ends of the curve?
0:15:04 I mean, that’s just information you just don’t get.
0:15:05 – Yeah, you just don’t get, right.
0:15:07 – So let’s zoom out a little bit
0:15:09 as somebody who lives in the world deeply of data
0:15:11 in the health system.
0:15:14 We’re in a time of enormous shift, right, for data.
0:15:17 Does the improvement, does our kind of the sea of data
0:15:20 and like better data, cleaner data, more granular data,
0:15:23 all that help this at all, this question?
0:15:28 – Yeah, and I think we are collecting so much data
0:15:32 on people, both sort of individual people
0:15:34 are collecting a lot of data about themselves.
0:15:37 Health systems are collecting a lot of data about people.
0:15:40 This data is like underutilized, I think.
0:15:41 We’re amassing pools of it,
0:15:43 but not in ways that are especially helpful.
0:15:45 So, you know, when I go to conferences
0:15:47 and people who work on healthcare,
0:15:48 like there’s a tremendous amount of data
0:15:50 that’s being used on health claims, right?
0:15:51 So if you sort of think about like,
0:15:53 what are some kinds of data that we have?
0:15:55 We have like health claims data, like payments,
0:15:56 everything that we’re,
0:15:58 where there’s an individual payment for it,
0:16:00 we’ll like see, we’ll see it.
0:16:02 There’s almost no work with medical records.
0:16:05 Even though every hospital, everybody’s using Epic,
0:16:08 you would think that that would make it straightforward
0:16:11 to have that data in a usable form, but it’s not.
0:16:12 And, but you know, at the same time,
0:16:16 the potential for sort of going beyond like,
0:16:18 here is all the tests that you ordered
0:16:21 into actually like, what happened with those tests?
0:16:23 And then what happened to this person later?
0:16:27 Like that data is not being mined in the way
0:16:30 that we could to try to look at some, you know,
0:16:33 at some of the kinds of outcomes that are a result.
0:16:36 – The causality that you would pull out afterwards.
0:16:37 – Yeah, absolutely.
0:16:39 You know, how can we improve our causality?
0:16:40 More data is helpful.
0:16:42 More information about people is helpful.
0:16:43 Being able to look at, you know,
0:16:45 the timing relationship between some treatment
0:16:47 and some outcome, those are all the kinds of things
0:16:50 that, you know, having better data would help us,
0:16:51 would help us do.
0:16:53 – Are there other areas where you start,
0:16:56 you are starting to see the data coalesce in a way
0:16:59 where you’re able to pull meaningful insights from it?
0:17:01 – So I think, yes, you know, when we have better data,
0:17:06 we can use better tools, even if we don’t have randomization.
0:17:09 A classic example in health is looking at the impacts
0:17:11 of like a really advanced neonatal care.
0:17:14 Like how cost effective is it to have like,
0:17:16 you know, kids in sort of getting like,
0:17:17 really extensive NICU care?
0:17:21 Like how effective is that in terms of improving survival
0:17:22 and how much does it cost?
0:17:23 – No, such a basic question.
0:17:27 – Such a basic question, and super hard to imagine analyzing
0:17:29 because of course, you know, babies that are very small
0:17:32 and are sick cost more but also have worse outcomes.
0:17:34 And so if you sort of looked at that,
0:17:36 you would be like, well, actually like spending more,
0:17:38 we’re not getting anything because those babies
0:17:41 are more likely to die than babies that are spending less.
0:17:46 We define very low-birthway babies as less than 1500 grams,
0:17:48 which means that the treatment that you get
0:17:51 if you’re a baby at 1500 and three grams
0:17:53 is very different than the treatment that you get
0:17:56 as a baby at 1497 grams, which is completely arbitrary.
0:17:59 I mean, the choice of 1500 grams has nothing to do with science.
0:18:00 – It’s like this line in the sand.
0:18:02 – That’s not a good way to set policy.
0:18:04 However, having set the policy like that,
0:18:07 you can then say, okay, well now we have some babies
0:18:08 that are almost exactly the same,
0:18:11 but the babies that are a little bit lighter
0:18:12 that are like 1497 grams
0:18:14 get all kinds of additional interventions
0:18:17 relative to the babies that are 1500 and three grams.
0:18:18 And when people have done that,
0:18:21 they see actually the babies at 1497 grams do better.
0:18:24 – So the line actually is beneficial in that way
0:18:26 because you’re defining these two groups very closely.
0:18:28 – Oh, interesting.
0:18:29 – Setting this line in this arbitrary way
0:18:31 lets you get at some causality.
0:18:32 – Even though not good for the babies.
0:18:34 – Sort of having done it good for research.
0:18:35 – Good for information.
0:18:36 Interesting.
0:18:38 What are some of the other tools?
0:18:39 Are there others in that list?
0:18:44 – So that’s an example called regression discontinuity,
0:18:47 that there’s some discontinuous change in policy
0:18:49 on either side of a cutoff.
0:18:52 And that has become a sort of part of a big toolkit
0:18:55 of things people are using more of.
0:18:57 The other is to look at sort of sharp changes
0:19:01 in policies at a time, at like a moment in time.
0:19:03 – Oh, so the same thing at the same time.
0:19:05 – Then there’s that and then there’s looking across
0:19:07 when different policies change differently
0:19:08 for different groups.
0:19:13 So, all of these things have become easier with more data
0:19:15 and become more possible with more data.
0:19:16 And I think that that has improved our inference
0:19:19 in some of these settings.
0:19:21 – I love that you talked a little bit about the experience
0:19:24 of doing your own data collection kind of in the wild
0:19:27 with this spreadsheet after Penelope was born,
0:19:27 which made me laugh so much
0:19:30 ’cause it was so much like my spreadsheet.
0:19:33 It was just so sad to think of like all these moms alone
0:19:35 in their bedrooms at night.
0:19:35 – I know, I know.
0:19:40 I mean, I think there’s been a lot more apps
0:19:44 since like we had that help, yeah.
0:19:48 But still, I love that you said
0:19:52 it gives the illusion of control, not control.
0:19:55 And in that particular, in these kinds of like data vacuums,
0:19:58 like if we’re not good at statistical analysis
0:20:01 or like pulling out causality from these murky areas,
0:20:03 like if we’re not Emily Oster basically,
0:20:05 how do you like, or even if you are,
0:20:08 how do you kind of stay on that line
0:20:10 of like the illusion of control
0:20:11 versus like actual knowledge
0:20:14 that like impacts real decision-making?
0:20:15 – No, I think it’s super hard
0:20:17 because the thing is the illusion of control
0:20:19 is a very powerful illusion.
0:20:20 – Very, yeah.
0:20:24 – And both empowering and dangerous in health context.
0:20:25 – Exactly.
0:20:26 Like we would, you know, you would sort of,
0:20:29 we like people to feel like they’re in control.
0:20:30 Some of the message of this book,
0:20:31 I think people have taken, not quite right,
0:20:33 but to say like, well, it doesn’t really matter
0:20:36 what choices you make, like all choices are good choices.
0:20:38 I think there’s, that’s not quite the right,
0:20:39 it’s not quite the message.
0:20:41 – That’s, I’m surprised that’s the message
0:20:42 that people take from this.
0:20:43 – Occasionally.
0:20:45 – There are a lot of different good choices
0:20:47 that you could make about parenting.
0:20:50 And so I think that there is a piece that like,
0:20:54 we maybe don’t need to be so like obsessive
0:20:55 about all of these.
0:20:55 – About one of those.
0:20:57 – About one of those, any one of those,
0:20:58 of those choices.
0:20:59 – What’s your point about range?
0:21:02 It’s like, well, let’s educate a little bit more
0:21:03 about like the spectrum of possibilities.
0:21:05 – Spectrum of good, of good choices.
0:21:06 – Yeah.
0:21:09 Another area I feel like where every other day
0:21:11 there’s a new study that says something different.
0:21:14 And it feels like there’s a plethora of studies
0:21:15 is screen time.
0:21:17 I’m just, I’m gonna put that out there right now.
0:21:18 I’m sorry.
0:21:21 Everybody, we’re gonna touch that third row.
0:21:22 – Three times.
0:21:24 – So can you walk us through,
0:21:28 like can you help guide us through some of that maze?
0:21:29 – So when I looked into screen time,
0:21:31 I had always thought about like screen time
0:21:32 is like bad.
0:21:35 Like it’s like, the question is, is it bad or not?
0:21:37 But actually there’s like a whole other side of this,
0:21:38 which is some people like screen time
0:21:40 is the way to make your kid smart.
0:21:42 Like you can, like your baby can learn from that.
0:21:42 – Okay.
0:21:44 So point number one is what does screen time
0:21:45 actually mean?
0:21:45 – Right.
0:21:46 – Which is a bunch of different stuff.
0:21:47 – Yeah.
0:21:48 And I think that’s part of the,
0:21:49 like that’s part of the problem with this is like,
0:21:51 when you say screen time, like what do you mean?
0:21:52 – Yeah.
0:21:54 – Do you mean like, you know, educational apps?
0:21:55 – Yeah.
0:21:56 – Do you mean…
0:21:57 – Do you mean Sesame Street?
0:21:57 – Sesame Street?
0:21:58 – Where you like jump in the shower?
0:21:59 – Yeah.
0:22:01 – Or yeah, and that point like while you jump in the shower,
0:22:03 like what is the other thing you’re going to be doing
0:22:04 with your time?
0:22:07 I think this is where all of these recommendations
0:22:11 seem to assume that the alternative use of your,
0:22:13 like if your kid wasn’t watching Sesame Street,
0:22:15 you would be like on the floor,
0:22:16 like playing puzzles with them
0:22:18 and like super engaged with them.
0:22:19 Which like, maybe is true.
0:22:21 – Taking them to the zoo and like having them touch
0:22:22 different textures of animal skins
0:22:25 or whatever like sensory development, yeah.
0:22:26 – Yeah, which like is great stuff
0:22:28 that you should definitely do with your kid.
0:22:30 But some of the time when, you know,
0:22:32 when our kids are watching TV,
0:22:34 it’s cause like we, it’s,
0:22:35 that maybe isn’t the thing
0:22:36 that you would otherwise be doing.
0:22:37 – Yeah.
0:22:39 – You could be like purring healthy vegetables
0:22:40 to like feed them well.
0:22:42 – Yeah, exactly.
0:22:43 I’m sure that’s what we’re all doing.
0:22:47 – Or maybe watching a little reality TV
0:22:49 for five minutes while you fold laundry.
0:22:51 – You look a little bit of called a midwife,
0:22:53 you know, got a little bit of, yeah.
0:22:54 – The problem with screen time is that the evidence
0:22:56 is very, is very poor.
0:22:58 – Can you just break up like why the evidence
0:22:59 is so poor?
0:23:00 Because this does seem like an area
0:23:02 where there should have been time
0:23:04 for that kind of gold standard randomized study
0:23:05 that to develop.
0:23:07 No, what is the evidence problem?
0:23:09 – So the, I think the evidence problem is twofold.
0:23:11 One, it’s actually not a super easy thing
0:23:14 to run a randomized trial on
0:23:15 because these are choices
0:23:17 that people are thinking a lot about.
0:23:19 And, you know, think about something like an iPad.
0:23:20 Like, do you want to be involved
0:23:23 in a randomized trial of whether you’re a kid?
0:23:24 – Oh, there’s too much intention,
0:23:25 too much at stake.
0:23:26 – Too much attention, exactly.
0:23:28 – Too much like lifestyle stuff.
0:23:30 Some people have been able to use like the introduction
0:23:33 of TV, which was sort of had some random features
0:23:34 to like look at the impacts of TV.
0:23:36 And that evidence is sort of reassuring
0:23:37 and suggested TV is okay.
0:23:38 But of course it’s very old.
0:23:40 It’s like from the fifties.
0:23:42 – A whole different way of consuming everything.
0:23:43 – Yeah.
0:23:44 – And I think the other thing is,
0:23:46 the other problem with the sort of current,
0:23:48 answering the current questions people want,
0:23:50 like what about iPads, what about apps, you know,
0:23:52 is that they just haven’t been around long enough.
0:23:54 So a lot of the kinds of outcomes you would want to know
0:23:58 that even things like short run, like test scores,
0:24:02 you know, I got the first iPad when my daughter was born.
0:24:03 Like that was like one,
0:24:04 and I remember getting in being like,
0:24:06 this is never going to catch on.
0:24:09 This is why I’m not in tech.
0:24:10 I was like, who would use this?
0:24:12 – I mean, while your daughter’s like swiping.
0:24:14 – I mean, while you’re just like, okay.
0:24:17 But you know, now she’s in second grade.
0:24:20 Like that’s kind of the earliest that you could kind of
0:24:22 imagine getting some kind of,
0:24:24 what we’d measured test scores or something like that.
0:24:26 But even, you know, she didn’t use the iPad anywhere
0:24:30 near as like facile away as my four year old, right?
0:24:32 This is evolving so quickly
0:24:35 that any kind of even slightly longer term outcomes
0:24:37 are really hard to imagine measuring,
0:24:40 let alone, you know, absent a randomized trial,
0:24:42 the, like if you weren’t able to randomize this,
0:24:44 which I think you won’t be able to,
0:24:48 the amount of time kids spend on these screens
0:24:52 is really wrapped up with other features of their household.
0:24:54 – Yeah, okay, so you have the definitions.
0:24:59 You have the time and the speed at which things are changing.
0:25:01 And then you have the willingness for people
0:25:03 to actually like engage and change
0:25:05 or doing things differently.
0:25:07 And then so all of that leads to what kind of,
0:25:10 so what do the studies actually tend to look like
0:25:12 in this space that we draw conclusions from?
0:25:14 – So actually there’s almost nothing
0:25:16 about iPads or phones.
0:25:18 – That seems so contrary to like
0:25:20 what the media is saying every five minutes.
0:25:22 – Yeah, so there’s tons of studies on TV,
0:25:24 which compare kids who watch more and less TV.
0:25:26 And, you know, you can, but most of that, again,
0:25:28 is sort of studies that are like based on data
0:25:31 where before people were watching TV on these screens,
0:25:35 maybe TV is TV, and you know, you can imagine
0:25:36 that that would be kind of similar.
0:25:39 But things like these apps, these just like no studies,
0:25:41 you know, or there’ll be, there’s like,
0:25:45 I think there’s one like abstract from a conference.
0:25:47 This is not a paper, there’s like answering comments
0:25:49 where it was just like we have some kids
0:25:51 and we like compare the kids who like spend,
0:25:52 like the babies who spend more time
0:25:54 watching their parents’ phones.
0:25:57 And then they like do worse, they’re like, look worse.
0:25:59 – But it’s like, it’s pathetic, it’s sad.
0:26:01 – It’s a terrible piece of evidence.
0:26:04 – So is this an area in which you just go with your gut?
0:26:07 – I mean, I try to generate a fancy version of go
0:26:10 with your gut, which is called Bayesian updating.
0:26:13 And so I basically try to say, look, you know,
0:26:16 I mean, we want to step back and think about
0:26:18 what are the places of uncertainty?
0:26:21 Logic would tell you, you know, your kid is awake
0:26:24 for what it is like 13 hours a day, 12 hours a day.
0:26:27 If your two year old is spending seven of those 12 hours
0:26:30 playing on the iPad, then there’s a lot of things
0:26:31 that they are not doing.
0:26:32 That’s probably not good.
0:26:36 On the other hand, you know, if your kid is spending
0:26:40 20 minutes every three days, it’s very hard to imagine
0:26:41 how that could be bad.
0:26:44 – So just thinking about it purely in times of like,
0:26:46 time allotted to any one activity, basically.
0:26:48 And then I think once you do that, then you’re sort of like,
0:26:50 okay, but you know, there are things that we’re uncertain
0:26:52 about, you know, what if my kid watches an hour of TV
0:26:54 every day or spends an hour on a screen every day?
0:26:56 Like is that too much, is the limit?
0:26:59 If we sort of accept like five minutes a day is fine.
0:27:00 Seven hours a day is too much.
0:27:03 Like is the limit at an hour, is the limit at two hours?
0:27:06 You know, and I think the truth is what we will find
0:27:08 if we end up in for doing any studies like this,
0:27:10 is that it depends a lot what other things
0:27:13 they would be doing with their time.
0:27:15 – Wouldn’t it also depend so much on the child?
0:27:19 – Some children need, you know, learn in a kind of way
0:27:21 that lends itself to this technology.
0:27:24 Some children need other kinds of learning, you know.
0:27:25 It’s highly individual.
0:27:27 – Yeah, I mean, I think this gets into the problem
0:27:29 with studying older kids in general,
0:27:30 that just like there’s so much,
0:27:31 there’s so many differences across kids.
0:27:33 It’s hard to even think about how you would structure
0:27:34 a study to learn about them.
0:27:37 Nevermind actually, like using evidence that exists.
0:27:40 – It’s really interesting because the last time we went
0:27:43 to take my daughter for her annual checkup,
0:27:44 or maybe it was my son, I can’t even remember it.
0:27:46 It’s so different from the first days
0:27:47 of those early spreadsheet where now I’m like,
0:27:49 did I even get it on which one is that?
0:27:51 – Yeah, exactly.
0:27:54 Anyways, the doctor said very concretely,
0:27:57 two hours, two hours max, within any date.
0:27:58 But it was really interesting to me
0:28:01 that it was such a specific line in the sand.
0:28:04 And now I’m thinking about how that information
0:28:06 would even get into that,
0:28:08 to percolate down to that level of like the system
0:28:10 and get kind of fossilized into the system,
0:28:12 so that that recommendation is being passed on to parents.
0:28:14 Like how does that happen with these studies?
0:28:18 How do they translate to that level of advice?
0:28:22 – Yeah, I think what happens is like organizations
0:28:23 like the American Academy of Pediatrics,
0:28:26 they bring people together to basically talk
0:28:28 about the conversation we just had,
0:28:30 which was like, okay, let’s agree,
0:28:31 sort of like we don’t know that much about this,
0:28:34 like five minutes seems fine, seven hours is too much.
0:28:36 These are like smart people who see kids a lot,
0:28:38 who presumably are using some knowledge
0:28:41 that they have about kids to pick some number.
0:28:43 But the answer is like, you could pick
0:28:44 a lot of different numbers.
0:28:47 We sort of say this and then it becomes like this rule.
0:28:49 And people have some impression that it comes
0:28:52 from some piece of evidence as opposed to sort of like,
0:28:58 you know, a synthesis of expert opinion or something,
0:29:01 which is really what it’s from.
0:29:04 – You also work specifically on certain health recommendations.
0:29:08 So how they change over time and how we stick to them.
0:29:10 You wrote a paper on behavioral feedback.
0:29:13 And then you talk about how those individual choices
0:29:15 might in fact be changing the science itself.
0:29:17 Can you talk about what that means
0:29:18 and how that might be happening?
0:29:20 – I was thinking about exactly this issue of like,
0:29:21 okay, we just make some recommendation
0:29:24 and sometimes those recommendations are kind of arbitrary,
0:29:25 but then they go out in the world
0:29:27 and people respond to them. – Take on lives of their own.
0:29:28 – Exactly.
0:29:30 And so like a sort of a good example,
0:29:32 vitamin E is supplements.
0:29:35 Like in the early 90s, there were a couple of studies
0:29:37 which suggested that like they are good for your health,
0:29:38 that like prevent cancer.
0:29:40 And so then there was like a recommendation,
0:29:42 like people should take vitamin E.
0:29:45 And then we had to ask a question like what,
0:29:47 like who takes vitamin E after that?
0:29:51 And one of the concerns is the kind of people
0:29:53 who would adopt these new recommendations,
0:29:55 like who listens to their doctor.
0:29:59 It is people who are probably,
0:30:01 maybe they’re more educated, maybe they’re richer,
0:30:05 but like above all, they are interested in their health.
0:30:07 So they are taking vitamin E, so they avoid cancer,
0:30:09 but they’re also exercising, so they avoid cancer.
0:30:11 And they’re eating vegetables, so they avoid cancer.
0:30:12 We use, call them selected.
0:30:14 These people are like positively selected
0:30:16 on other health things.
0:30:18 And so indeed you can see in the data
0:30:21 that the people who start taking vitamin E
0:30:22 after this recommendation changes
0:30:25 are kind of also exercising and not smoking
0:30:27 and doing all kinds of other stuff.
0:30:30 Well, why is that like interesting or problematic?
0:30:34 Well, later we’re gonna go back to the data
0:30:36 ’cause that’s like the way science works.
0:30:38 But now the people who take vitamin E
0:30:42 are even more different than they were before, right?
0:30:43 So now these people are like.
0:30:44 So you’ve added another layer.
0:30:45 You’ve added another layer.
0:30:47 So in fact, you can see that in the data.
0:30:49 You can see that basically before these recommendations
0:30:52 changed, there was sort of a small relationship
0:30:56 between taking vitamin E and like subsequent mortality rates.
0:30:58 But after the recommendations change,
0:31:01 you see like a very large relationship
0:31:03 between vitamin E and mortality rates.
0:31:06 And so it looks like basically ends up looking like vitamin E
0:31:08 like is really great for you.
0:31:09 – Has this big impact.
0:31:11 – But of course that’s because at least,
0:31:13 it seems like it must be at least in part
0:31:16 because the people who adopt vitamin E
0:31:21 are the people who are also doing these other things.
0:31:23 – So what does that mean then?
0:31:24 It feels like such a loss.
0:31:26 Like how does one ever- – It’s so depressing.
0:31:28 – Yes. (laughs)
0:31:31 How would one ever develop like a recommendation
0:31:33 based on what we think we know.
0:31:34 – I know.
0:31:36 – And untangle it from like-
0:31:39 – So this paper is very destructive in some sense.
0:31:42 – Other than saying like it probably doesn’t matter
0:31:44 if you take vitamin E, so that’s like news you can use.
0:31:46 You can take that home with you.
0:31:50 But I mean, I think it does more or less just highlight
0:31:54 some of the inherent and very deep limitations
0:31:58 with our ability to learn about some of these effects,
0:31:59 particularly when they’re small.
0:32:01 – Is this basically part of the sort of crisis
0:32:02 of reproducibility?
0:32:04 – I think it’s not unrelated.
0:32:06 So I often think about this idea of p-hacking,
0:32:10 which refers to the idea that you keep running your studies
0:32:12 until you get a significant result.
0:32:16 There’s a bunch of people interested in this process
0:32:19 of like how science evolves
0:32:24 and the ways in which the evolution of science
0:32:27 influences the science itself or the incentives
0:32:31 for research influence how science works.
0:32:35 And I think it’s particularly hard to draw conclusions
0:32:39 in these spaces like diet or these health behaviors
0:32:41 where the honest truth is probably a lot
0:32:43 of these effects are very small.
0:32:45 So if you ask the question like,
0:32:47 what is the effect of chia seeds on your health?
0:32:49 My dad is like really into chia seeds.
0:32:50 – That was a thing.
0:32:51 There was a moment.
0:32:53 – Well, he’s still in that moment.
0:32:53 He’s still in there.
0:32:56 And what is the effect of those on your health?
0:32:58 The actual effect is probably about zero.
0:33:00 Maybe it’s not exactly zero,
0:33:01 but it’s almost certainly about zero.
0:33:03 – But are there sometimes secret sleeper?
0:33:05 Like, whoa, they’re actually might,
0:33:08 the only way to find out is to do these things.
0:33:09 – Yeah, yeah.
0:33:10 And so maybe there are some secrets.
0:33:12 – Like maybe kale really is magic.
0:33:14 – Maybe it is, but it’s probably not.
0:33:18 I spent a lot of time with these diet data
0:33:21 and there’s these sort of like dietary patterns
0:33:23 like the Mediterranean diet,
0:33:26 which do seem to have some sort of vague support
0:33:30 in the evidence, but I would be extremely surprised
0:33:33 if we ever turn up like the one single thing.
0:33:35 – One magical food.
0:33:36 – So the point is it’s the pattern.
0:33:38 – It’s the pattern and it’s all the other things
0:33:39 that you’re doing, right?
0:33:42 If you smoke three packs a day and you never exercise,
0:33:45 but you eat some kale, that’s not gonna help you.
0:33:46 – Yeah, yeah.
0:33:47 – The kale’s not gonna help.
0:33:50 – What about when you really do need to affect change?
0:33:54 What are the ways in which these guidelines
0:33:57 can shift over time with kind of new sources of information
0:33:58 or data and statistics?
0:33:59 Like what’s the positive?
0:34:02 How does that actually play out in the right manner?
0:34:05 – Yeah, so I think there are times in which we,
0:34:08 the change in evidence is so big
0:34:12 and so like compelling that we can get changes,
0:34:15 best practices in obstetrics,
0:34:19 like how do you deliver, reach baby, as an example,
0:34:21 they change, like those changed over time
0:34:25 because there was like one very big well-recognized study
0:34:28 that everybody agreed like this is now the state of the art.
0:34:30 – And it happens fast at that point?
0:34:31 – And then it happens pretty fast.
0:34:32 It doesn’t happen immediately.
0:34:33 Like you might have thought that those kind of changes
0:34:35 could be like immediately affected
0:34:36 and I think that they’re not,
0:34:39 but they do happen over time.
0:34:44 Those examples really rely on there being like a cohort
0:34:49 of sort of like experts who are all reading the guidelines
0:34:51 and sort of seeing that they changed
0:34:56 and then themselves are sort of doing this all the time.
0:34:59 I think part of what’s hard in the broader health behavior
0:35:01 space where it’s people who need to make the choices,
0:35:03 not physicians.
0:35:06 – Yes, when it’s in the home and those dark bedrooms.
0:35:08 – It’s like that’s much harder to get people
0:35:10 to change their behavior in those spaces.
0:35:12 – It’s not these pediatric guidelines.
0:35:13 Those are not effective.
0:35:14 – Yeah, I do not think those are effective.
0:35:16 Or I think we don’t see any evidence in the data
0:35:18 that those are effective at moving these,
0:35:20 at least in these kind of spaces.
0:35:21 – So what’s the answer?
0:35:23 How do we positively affect change
0:35:25 and gather these insights and have smart people
0:35:26 making good recommendations?
0:35:29 – So I mean, I think one answer is media attention.
0:35:32 The kind of few times when we see very large spikes
0:35:35 and changes, they actually seem to correspond
0:35:36 with some media coverage.
0:35:39 On the flip side, like media can often be very bad.
0:35:41 Some of these big changes in these expert things
0:35:44 were kind of resulted from media coverage,
0:35:46 which was really like sensationalist
0:35:48 and like totally inappropriate.
0:35:50 And, you know, it wasn’t like a very nice,
0:35:53 like New York, New York time story about some study.
0:35:56 It was like a sensationalist 2020.
0:35:57 – About what?
0:36:00 – About, this is about vacuum extraction,
0:36:02 which is a way of pulling the baby out
0:36:04 and has gone down a lot over time.
0:36:05 And it was like the sort of sensationalist
0:36:08 like John Stossel 2020 episode
0:36:09 about how it could hurt your baby,
0:36:11 which caused like big productions.
0:36:12 – Interesting.
0:36:14 – Yeah, you know that like.
0:36:15 – But the science was there.
0:36:17 – The science, yeah, was there.
0:36:19 I mean, he overstated the science,
0:36:21 but it was, it was probably there.
0:36:23 – So it’s almost like a random confluence
0:36:24 of like when the science is there
0:36:26 and the media hits it the right way,
0:36:27 and then we see change?
0:36:28 – Yeah.
0:36:29 (laughing)
0:36:30 – It’s okay. – That’s something
0:36:31 to hope for.
0:36:34 – Yeah, that doesn’t feel like we can plan so much for that.
0:36:38 – You also study when we are resistant to change.
0:36:40 You looked specifically at diabetes,
0:36:42 people I think who had been diagnosed with diabetes
0:36:44 and then whether or not their behavior changed,
0:36:46 even given a certain amount of information.
0:36:49 So what do you see there about our resistance to change
0:36:52 even with the right kinds of information?
0:36:54 – I mean, I think one of the big challenges
0:36:56 in the health space at the moment
0:36:57 is that like so much of the,
0:37:00 so many of the health problems that we have in the US
0:37:03 are like problems associated with behavior,
0:37:06 just the fundamental fact that like people do not eat great
0:37:09 and we have a lot of morbidity
0:37:11 and expense associated with that.
0:37:14 And I think there is often a lot of emphasis
0:37:16 on the idea like if we just get the information out,
0:37:19 if people just understood vegetables were good for them,
0:37:20 hey, what are you vegetables?
0:37:21 – Doesn’t happen. – That’s not true.
0:37:24 I think, and so this paper is about sort of looking
0:37:26 at something where kind of a pretty extreme thing
0:37:28 happens to people, like they are diagnosed with diabetes
0:37:31 and we can see what happens to their diet.
0:37:33 And the answer is, it improves a tiny amount.
0:37:36 – Even with a real come to Jesus moment.
0:37:38 – Exactly, and a lot of new information.
0:37:41 – Right, and monitoring, follow-up, right?
0:37:42 I mean, you’re diagnosed with diabetes,
0:37:44 like you have to take medicine every day,
0:37:46 you got to go to the doctor like get a tester,
0:37:47 you know, test your insulin,
0:37:48 at least for some period of time.
0:37:51 So this isn’t like something where you can just forget
0:37:54 that it happened and even then the changes in diet,
0:37:56 you know, they’re there, but they’re really small.
0:38:00 They’re like, you know, like one less soda week or something.
0:38:01 – Oh gosh. – Like really,
0:38:02 like really small.
0:38:03 – And how are you noticing these?
0:38:06 – We’re inferring information on diagnosis
0:38:09 from people’s purchases of testing products
0:38:11 and then following their grocery purchases.
0:38:13 So this is like an example of using,
0:38:15 you know, a different kind of data.
0:38:16 So not health data in this case.
0:38:19 It’s actually like Nielsen data.
0:38:21 So Nielsen data on what people buy,
0:38:24 but then, you know, using some like machine learning techniques
0:38:27 to try to figure out from the kinds of things people buy,
0:38:29 when were they diagnosed with diabetes,
0:38:31 and then looking at their diets over time.
0:38:33 – So is the answer that there has to be
0:38:36 some sensational story that talks about like–
0:38:38 – I mean, I think there, I’m not even sure that would help.
0:38:40 I think part of the problem is people really like
0:38:42 the diets that they’re comfortable with.
0:38:45 Like diet is like such a habit formation thing,
0:38:48 and you know, people are willing to make
0:38:52 important health sacrifices to maintain the diet
0:38:53 that they like.
0:38:55 We get into some of these questions of preferences,
0:38:57 like, and you know, if people,
0:38:59 if that is the choice that people wanna make,
0:39:03 like should we be trying to intervene with policy?
0:39:05 Like let’s say everybody had all the information,
0:39:07 they knew that they shouldn’t drink so much soda
0:39:08 and that they should lose weight,
0:39:10 but they still chose not to.
0:39:13 Like do we wanna develop policies that affect that?
0:39:13 I’m not sure.
0:39:15 – Yeah, maybe that’s just free will.
0:39:16 – Yeah, maybe that’s just free will.
0:39:18 And it comes up in the parenting stuff too.
0:39:20 Like, you know, how much do we wanna be externally
0:39:23 controlling the choices people make with their kids,
0:39:25 even if we don’t think that they’re the right choices.
0:39:28 – But I do think there’s a segment of people
0:39:30 who want to make the change, but the gravity,
0:39:32 you know, because of the information,
0:39:33 but the gravity of the habit is so much
0:39:36 that it’s hard to know where to go about it.
0:39:39 – I guess I would say, where do you see this data going?
0:39:42 Like if you had your fantasy for where you want
0:39:44 the kind of data and the way that we see this data evolving
0:39:47 and the way that you see that kind of percolating out
0:39:50 to the public, I mean, in terms of being sort of a translator
0:39:51 and providing people the tools,
0:39:54 like what do you wanna see in terms of the way the system
0:39:57 responds to or integrates this data in the future?
0:40:00 – Yeah, I mean, I think the big message of the book
0:40:03 is in some sense that you should use the data
0:40:07 to make yourself confident and happy in your choices.
0:40:10 I think so much of what is hard about parenting
0:40:14 is that in the moment you are not often confident
0:40:17 in your choices, and then when somebody asks you,
0:40:20 like, why did you do that, then you feel bad, right?
0:40:23 And I think that there’s a sense in which sort of
0:40:24 looking at the data, but then confronting like,
0:40:27 well, we don’t know, but you’d be like, okay, I made this choice.
0:40:28 You know, I decided to let my kids watch an hour
0:40:31 of TV every day, because like I thought about it
0:40:33 and I thought there wasn’t any data,
0:40:35 and like that’s the choice that I made,
0:40:37 that sort of that confidence is like important
0:40:40 for being happy, and if we could sort of like
0:40:44 move in that direction, I think that would be good.
0:40:46 – It reminds me a lot of what one of my good friends,
0:40:47 Brandy said to me when I was in the trenches
0:40:49 of like babyhood and having a lot of anxieties
0:40:52 around all these hot button issues, breastfeeding,
0:40:53 sleep time, like all of it.
0:40:55 She had been through it, her kids were in college,
0:40:57 and she was like, let me give you a piece of advice.
0:40:59 Be wrong, but be wrong with confidence.
0:41:00 – Yes.
0:41:01 – Just be wrong with confidence.
0:41:02 That’s all that matters.
0:41:03 – Yeah. – Yeah.
0:41:04 – No, exactly.
0:41:05 – I love confidence, I love that.
0:41:07 Yes, I am wrong with confidence so frequently.
0:41:08 – Yes, and actually it turns out to be right.
0:41:09 – Like it turns out that it’s fine.
0:41:10 – The truth is there’s a lot of good options.
0:41:11 – A lot of good options.
0:41:12 – Yeah. – Thank you so much
0:41:14 for joining us on the A16Z podcast.
0:41:15 – Thank you for having me.

0:00:06 Hi, everyone. Welcome to the A6NC podcast. I’m Sonal, and today Mark and I are doing another one
0:00:10 of our book author episodes. We’re interviewing Annie Duke, who’s a professional poker player and
0:00:17 World Series champ and is the author of Thinking in Bets, which is just out in paperback today.
0:00:21 The subtitle of the book is Making Smarter Decisions When You Don’t Have All the Facts,
0:00:25 which actually applies to startups and companies of all sizes and ages, quite frankly. I mean,
0:00:30 basically any business or new product line operating under conditions of great uncertainty,
0:00:34 which I’d argue is my definition of a startup and innovation. So that will be the frame for
0:00:39 this episode. Annie is also working on her next book right now and founded HowIDecide.org,
0:00:43 which brings together various stakeholders to create a national education movement around
0:00:48 decision education, empowering students to also be better decision makers. So anyway,
0:00:51 Mark and I interview her about all sorts of things in and beyond her book,
0:00:55 going from investing to business to life. But Annie begins with a thought experiment,
0:00:58 even though neither of us really know that much about football.
0:01:02 So what I’d love to do is kind of throw a thought experiment at you guys so that we can
0:01:06 have a discussion about this. So I know you guys don’t know a lot about football,
0:01:09 but this one’s pretty easy. You’re going to be able to feel this one, which is do this thought
0:01:16 experiment. Pete Carroll calls for Marshawn Lynch to actually run the ball.
0:01:18 So we’re betting on someone who we know is really good.
0:01:21 Well, they’re all really good, but we’re betting on the play that everybody’s expected.
0:01:25 This is the default. This is the assumed irrational thing to do.
0:01:30 Right. So he has Russell Wilson handed off to Marshawn Lynch. Marshawn Lynch goes to barrel
0:01:35 through the line. He fails. Now they call the time out. So now they stop the clock,
0:01:39 they get another play now, and they hand the ball off to Marshawn Lynch,
0:01:45 what everybody expects. Marshawn Lynch again, attempts to get through that line and he fails.
0:01:52 End of game, Patriots win. My question to you is, are the headlines the next day
0:01:58 that worst call in Super Bowl history? Is Chris Collins we’re saying, I can’t believe the call,
0:02:04 I can’t believe the call, or is he saying something more like, that’s why the Patriots are so good,
0:02:09 their line is so great. That’s the Patriots line that we’ve come to see this whole season.
0:02:15 This will seal Belichick’s place in history. It would have all been about the Patriots.
0:02:21 So let’s sort of divide things into, we can either say the outcomes are due to skill or luck,
0:02:27 and luck in this particular case is going to be anything that has nothing to do with Pete Carroll.
0:02:31 And we can agree that the Patriots line doesn’t have anything to do with Pete Carroll. Belichick
0:02:34 doesn’t have anything to do with Pete Carroll. Tom Brady doesn’t have anything to do with Pete
0:02:38 Carroll as they’re sealing their fifth Super Bowl victory. So what we can see is there’s two
0:02:44 different routes to failure here. One route to failure, you get resulting. And basically what
0:02:50 resulting is, is that retrospectively, once you have the outcome of a decision, once there’s a
0:02:55 result, it’s really, really hard to work backwards from that single outcome to try to figure out
0:02:59 what the decision quality is. This is just very hard for us to do. They say, oh my gosh, the outcome
0:03:05 was so bad. This is clearly, I’m going to put that right into the skill bucket. This is because of
0:03:10 Pete Carroll’s own doing. But in the other case, they’re like, oh, you know, there’s uncertainty.
0:03:15 What could you do? Weird, right? Yeah. Okay. So you can kind of take that and you can say, aha,
0:03:21 now we can sort of understand some things. Like, for example, people have complained for a very
0:03:28 long time that in the NFL, they have been very, very slow to adopt what the analytics say that
0:03:32 you should be adopting, right? And even though now we’ve got some movement on like fourth down
0:03:36 calls and when are you going for two point conversions and things like that, there’s still
0:03:40 nowhere close to where they’re supposed to be. So they don’t make the plays corresponding to
0:03:45 the statistical probabilities? No. In fact, the analytics show that if you’re on your own one
0:03:51 yard line and it’s fourth down, you should go for it, no matter what. The reason for that is if you
0:03:54 kick it, you’re only going to be able to kick to midfield. So the other team is basically almost
0:03:59 guaranteed three points anyway. So you’re supposed to just try to get the, try to get the yards.
0:04:03 Like, when have you ever seen a team on their own one yard line on fourth down be like, yeah,
0:04:08 let’s go for it. That does not happen. Okay. So we know that they’ve been like super slow
0:04:12 to do what the analytics say is, is correct. And so you sit here and you go, well, why is that?
0:04:18 And that thought experiment really tells you why, because we’re all human beings. We all
0:04:23 understand that there are certain times when we don’t allow uncertainty to bubble up to the surface
0:04:29 as the explanation. And there are certain times then we do. And it seems to be that we do when we
0:04:35 have this kind of consensus around the decision, there’s other ways we get there. And so, okay,
0:04:39 if I’m a human decision maker, I’m going to choose the path where I don’t get yelled at.
0:04:45 Yeah, exactly. So basically we can kind of walk back and we can say, are we allowing the uncertainty
0:04:49 to bubble to the surface? And this is going to be the first step to kind of understanding what
0:04:55 really slows innovation down, what really slows adoption of, of what we might know is good decision
0:04:58 making, because we have conflicting interests, right, making the best decision for the long run,
0:05:03 or making the best decision to keep us out of a room where we’re getting judged or
0:05:07 yelled at or possibly fired. So can I, let me propose the framework that I used to think about
0:05:12 this and see if you agree with it. So it’d be a two by two, a two by two grid and it’s consensus
0:05:17 versus non-consensus and it’s right versus wrong. And the way we think about it, at least in our
0:05:24 business is basically consensus, right is fine. Consensus, non-consensus right is fine. In fact,
0:05:29 generally you get called a genius. Consensus wrong is fine because you just, you know,
0:05:32 it’s just the same mistake everybody else made. You all agree, right, it was wrong.
0:05:36 Non-consensus wrong is really bad. It’s horrible. It’s radioactively bad.
0:05:40 Right. And so, and then, and then as a consequence of that, and maybe this gets to the innovation
0:05:44 stuff that you’ll be talking about, but as a consequence of that, there are only two scripts
0:05:49 for talking about people operating in, in the non-consensus directions. One script is they’re
0:05:54 a genius because it went right and the other is they’re a complete moron because it went wrong. Is
0:05:58 that, does that map? That’s, that’s exactly, that’s exactly right. And I think that the problem
0:06:04 here is that what is right and wrong mean in your two by two, wrong and right is really this,
0:06:08 just to turn out well or not. Yeah, okay. And this is where we really get into this problem
0:06:13 because now what people are doing is they’re trying to swat the outcomes away and they understand,
0:06:19 just as you said, that on that consensus wrong, you will have like a cloak of invisibility over
0:06:24 you. Like, you don’t have to deal with it. Right. So, let’s think about other things besides
0:06:30 consensus. So, consensus is one way to do that, especially when you have like complicated cost
0:06:34 benefit analyses going into it. I don’t think that people, when they’re getting in a car,
0:06:41 are actually doing any kind of calculation about what the cost benefit analysis is to their own
0:06:47 productivity versus the danger of something very bad happening to them. Like, well, as a society,
0:06:50 someone’s done this calculation, we’ve all kind of done this together. And so therefore,
0:06:54 like getting in a car is totally fine. I’m going to do that. And nobody second guesses anybody.
0:06:57 Somebody dies in a car crash. You don’t say, wow, what a moron for getting in a car.
0:07:03 No. Another way that we can get there is through transparency. So, if the decision is pretty
0:07:09 transparent, another way to get there is status quo. So, like a good status quo example that I
0:07:14 like to give because everybody can understand it is, you have to get to a plane and you’re with
0:07:21 your significant other in the car and you go the usual route. So, you go your usual route.
0:07:26 Like, you go literally, this is the route that you’ve always gone and there’s some sort of accident,
0:07:30 there’s bad traffic, you missed the plane and you’re mostly probably comforting each other
0:07:35 in the car. It’s like, what could we do? But then you get in the car and you announce to
0:07:41 your significant other, I’ve got a great shortcut. So, let’s take the shortcut to the airport. And
0:07:46 there’s same accident, whatever, horrible traffic, you missed the flight. And that’s like that status
0:07:50 quo versus non-status quo decision. Right. You’re going against what’s familiar and comfortable.
0:07:56 Exactly. If we go back to the car example, when you look at what the reaction is to a pedestrian
0:08:02 dying because of an autonomous vehicle versus because of a human, we’re very, very harsh with
0:08:07 algorithms. For example, if you get in a car accident and you happen to hit a pedestrian,
0:08:12 I can say something like, well, Mark didn’t intend to do that. Because I think that I understand
0:08:17 your mind is not such a black box to me. So, I feel like I have some insight into what your
0:08:23 decision might be and so more allowing some of the uncertainty to bubble up there. But if this
0:08:29 black box algorithm makes the decision, now all of a sudden, I’m like, get these cars off the road.
0:08:33 Never mind that the human mind is a black box itself. Of course. But we have some sort of
0:08:37 illusion that I understand sort of what’s going on in there, just like I have an illusion that I
0:08:40 understand what’s going on in my own brain. And you can actually see this in some of the
0:08:46 language around crashes on Wall Street, too, when you have a crash that comes from human
0:08:50 beings selling. People say things like, the market went down today. When it’s algorithms,
0:08:56 they say it’s a flash crash. So now they’re sort of pointing out, this is clearly in the
0:08:59 skilled category. It’s the algorithm’s fault. We should really have a discussion about algorithmic
0:09:04 trading and whether there should be allowed. When obviously the mechanism for the market
0:09:08 going down is the same either way. So now if we understand that, so exactly your matrix,
0:09:12 now we can say, well, okay, human beings understand what’s going to get them in the room.
0:09:19 And pretty much anybody who’s living and breathing in the top levels of business at this point is
0:09:22 going to tell you, process, process, process. I don’t care about your outcomes, process, process,
0:09:27 process. But then the only time they ever have like an all hands on deck meeting is when something
0:09:31 goes wrong. Like let’s say that you’re in a real estate investing group. And so you invest in a
0:09:37 particular property based on your model. And the appraisal comes in 10% lower than what you
0:09:42 expected. Like everybody’s in a room, right? You’re all having a discussion. You’re all examining
0:09:46 the model. You’re trying to figure out, but what happens when the appraisal comes in 10% higher
0:09:50 than expected? Is everyone in the room going, what happened here? Now there is the obvious
0:09:54 reality, which is like, we don’t get paid in process. We get paid in outcomes. Booker players,
0:09:58 you don’t get paid in process, you get paid in outcome. And so there is an incentive alignment.
0:10:02 It’s not completely emotional. There’s also an actual, there’s a real component to it.
0:10:08 Yeah. So two things. One is you have to make it very clear to the people who work for you that
0:10:13 you understand that outcomes will come from good process. That’s number one. And then number two,
0:10:19 what you have to do is try to align the fact that as human beings, we tend to be outcome driven
0:10:28 to what you want in terms of getting an individual’s risk to align with the enterprise risk.
0:10:31 Because otherwise you’re going to get the CYA behavior. And the other thing is that we want
0:10:35 to understand if we have the right assessment of risk. So one of the big problems with the
0:10:39 appraisal coming in 10% too high there could be that your model’s correct. It could be that you
0:10:44 could have just a tail result, but it certainly is a trigger for you to go look and say, was there
0:10:48 risk in this decision that we didn’t know was there? And it’s really important for deploying
0:10:54 resources. I have a question about translating this to say non-investing context. So if in the
0:11:01 example of Mark’s Matrix, even if it’s a non-consensus wrong, you are staking money
0:11:06 that you are responsible for. In most companies, people do not have that kind of skin in the game.
0:11:12 So how do you drive accountability in a process-driven environment that the results actually
0:11:17 do matter? You want people to be accountable yet not overly focused on the outcome? How do you
0:11:23 calibrate that? So let’s think about how can we create balance across three dimensions
0:11:26 that makes it so that the outcome you care about is the quality of the forecast.
0:11:33 So first of all, obviously this demands that you have people making forecasts. You have to stay
0:11:38 in advance. Here’s what I think. This is my model of the world here where all the places are going
0:11:45 to fall. So this is what I think. So now you stated that and the weather the outcome is “good”
0:11:50 or “bad” is how close are you to whatever that forecast is. So now it’s not just like,
0:11:55 oh, you won to it or you lost to it, it was your forecast good. So that’s piece number one is make
0:12:00 sure that you’re trying to be as equal across quality as you can and focus more on forecast
0:12:04 quality as opposed to like traditionally what we would think of as outcome quality.
0:12:12 So now the second piece is directional. So when we have a bad outcome and everybody gets in the room,
0:12:16 when was the last time that someone suggested, “Well, you know, we really should have lost more
0:12:24 here.” Like nobody’s saying that. But sometimes that’s true. Sometimes if you examine it, you’ll
0:12:29 find out that you didn’t have a big enough position. It turned out, okay, well maybe we should have
0:12:36 actually lost more. So you want to ask both up, down, and orthogonal. So could we have lost less?
0:12:42 Should we have lost more? And then the question of should we have been in this position at all?
0:12:46 So Inventure Capital, after a company works and exits, they say it sells for a lot of money,
0:12:51 you do often say, “God, I wish we had invested more money.” You never, ever, ever, ever,
0:12:55 I have never heard anybody say on a loss we should have invested more money.
0:12:59 See, I wouldn’t be great if someone said that. Like wouldn’t you love for someone to come up and
0:13:03 say that to you? That would make you so happy. And what would be the logic of why they should say
0:13:06 that? I still don’t get the point. Exactly. Why does that matter? I don’t really understand that.
0:13:11 So let’s, can I just, like simple in a poker example. So let’s say that I get involved in a
0:13:20 hand with you and I have some idea about how you play. And I have decided that you are somebody
0:13:26 that if I, if I bet X, you will continue to play with me. Let’s say this is a spot where I know
0:13:32 that I have the best hand. But if I bet X plus C that you will fold. So if I go above X that I’m
0:13:36 not going to be able to keep you coming along with me, but if I bet X or below that you will. So I
0:13:43 bet X you call, but you call really fast in a way that makes me realize, oh, I could have actually
0:13:48 bet X plus C. You hit a very lucky card on the end and I happened to lose the pot. I should have
0:13:52 maximized at the point that I was a mathematical favorite. Your model of me was wrong, which is
0:13:56 a learning independent of the winner, the loss. Exactly. So you need to be exploring those questions
0:14:01 in a real honest way. Because it has to do with how you size future bets. This is exactly like a
0:14:05 company betting on a product line. Correct. And then like ticking, like, you know, what the next
0:14:09 product line is going to be, and then not having had the information that would then drive a better
0:14:12 decision-making process around that. Right. So think about the learning loss that’s happening,
0:14:16 because we’re not exploring that. The negative direction is, and now you should do this on
0:14:22 wins as well. So if you do ever discuss a win, you always think like, how could I press? How
0:14:25 could I have won more? How could I have made this even better? How could I do this again in the
0:14:29 future? Should we have won less? We oversized the bet and then got bailed out by a fluke.
0:14:33 We should have actually had less in it. And sometimes not at all, because sometimes
0:14:37 the reasons that we invested turned out to be orthogonal to the reasons that it
0:14:41 actually ended up playing out in the way that it was. And so had we had that information,
0:14:45 we actually wouldn’t have bet on this at all, because it was completely orthogonal. Like,
0:14:50 we totally had this wrong. It just turned out that we ended up winning. And that can happen.
0:14:54 Obviously, that happens in poker all the time. But what does that communicate to the people on
0:14:59 your team? Good, bad, I don’t care. I care about our model. I want to know that we’re
0:15:03 modeling the world well and that we’re thinking about how do we incorporate the things that we
0:15:09 learn? Because we can generally think about stuff we know and stuff we don’t know. There’s
0:15:13 stuff we don’t know we know, obviously. So we don’t worry about that, because we don’t know
0:15:18 we don’t know it. But then there’s stuff we could know and stuff we can’t know. It’s things like
0:15:22 the size of the universe or the thoughts of others. Or what the outcome will actually be.
0:15:28 We don’t know that. I have a question about this, though. What is a time frame for that forecast?
0:15:32 So let’s say you have a model of the world, a model of a technology, how it’s going to adopt,
0:15:38 how it’s going to play out. In some cases, there are companies that can take years to get traction.
0:15:42 You want to get your customers very early to figure that out, right? So you can get that data.
0:15:48 But how much time do you give? How do you size that time frame for the forecast? So you’re not
0:15:52 constantly updating with every customer data point. And so you’re also giving it enough time for your
0:15:57 model, your plan, your forecast to play out. You have to think about very clearly in advance,
0:16:03 what’s my time horizon? How long do I need for this to play out? But also, don’t just do this
0:16:06 for the big decisions, because there’s things that you can forecast for tomorrow as well,
0:16:11 so that you end up bringing it into just the way that people think. And then once you’ve decided,
0:16:15 okay, this is the time horizon of my forecast. And you would want to be thinking about what
0:16:22 are forecasts we make for a year, two years, five years for the specific decision to play out.
0:16:27 And then just make sure that you talk in advance at what point you’ll revisit the forecast.
0:16:31 So you want to think in advance, what are the things that would have to be true
0:16:36 for me to be willing to come in and actually revisit this forecast? Because otherwise, you can
0:16:39 start, as you just said, you can turn into super bad. You like to leave in the wind.
0:16:44 Exactly, because then you’re one bad customer and you suddenly over-rotate on that when,
0:16:49 in fact, it could have been not leaving the thing. So if you include that in your forecast,
0:16:52 here are the circumstances under which we would come in and check on our model.
0:16:57 Then you’ve already got that in advance. So that’s actually creating constraints
0:17:01 around the reactivity, which is helpful. Two questions on practical implementation of the
0:17:05 theory. So what I’m finding is more and more people understand the logic of what you’re describing,
0:17:09 because people are getting exposed to these ideas and kind of expanding in importance.
0:17:12 And so more and more people intellectually understand this stuff. But there’s two kind of,
0:17:16 I don’t know, so-called emotion-driven warps or something that people just really have a hard
0:17:21 time with. So one is, you understand this could be true of investors, CEO, product line manager,
0:17:24 in a company, kind of anybody, in one of these domains, which is you can’t get the
0:17:27 non-consensus right results unless you’re willing to take the damage,
0:17:32 the risk on the non-consensus wrong results. But people cannot cope with the non-consensus
0:17:37 wrong outcome. They just emotionally cannot handle it. And they would like to think that
0:17:39 they can. And they intellectually understand that they should be able to. But as you say,
0:17:44 when they’re in the room, it’s such a traumatizing experience that it’s touching the hot stove,
0:17:48 they will do anything in the future to avoid that. Is that just a… And so one interpretation would
0:17:52 be, that’s just simply flat out human nature. And so to some extent, the intellectual understanding
0:17:56 of here doesn’t actually matter that much because there’s an emotional override. And so that would
0:18:00 be a pessimistic view on our ability as a species to learn these lessons. Or do you have a more
0:18:04 optimistic view of that? I’m going to be both pessimistic and optimistic at the same time.
0:18:09 So let me explain why. Because I think that if you move this a little bit, it’s a huge difference.
0:18:13 You sort of have two tasks that you want to take. One is, how much can you move the individual to
0:18:19 sort of train this kind of thinking for them? And that means naturally, they’re thinking in
0:18:24 forecasts a little bit more, that when they do have those kinds of reactions, which naturally
0:18:30 everybody will, they write the ship more quickly so that they can learn the lessons more quickly.
0:18:35 Right? I mean, I actually just had this happen. I turned in a draft of my next book, the first
0:18:38 part of my next book to my editor, and I just got the worst comments I’ve ever gotten back.
0:18:42 And I had a really bad 24 hours. But after 24 hours, I was like, you know what, she’s right.
0:18:48 Now, I still had a really bad 24 hours. And I’m the like, give me negative feedback, like Queen,
0:18:52 because I’m a human being. But I got to it fast. Like, I sort of got through it pretty quickly
0:18:57 after this. I mean, I, you know, the, you know, on the phone with my agent saying, I’m standing
0:19:01 my ground. This is ridiculous. And then he got a text the next day being like, no, she’s right.
0:19:05 And then I rewrote it. And you know what? It’s so much better for having been rewritten. And now
0:19:10 I can get to a place of gratitude for having the negative feedback. But I still had the really
0:19:15 bad day. So it’s okay. It doesn’t go away. Right. Yeah. And it’s okay. Like, we’re all human.
0:19:22 Like, we’re not robots. So number one is like, how much are you getting the individuals to say,
0:19:27 okay, I improved 2%. That’s so amazing for my decision making and my learning going forward.
0:19:32 And then the second through line is, what are you doing to not make it worse?
0:19:37 Because obviously, for a long time, people like to talk about I’m results oriented.
0:19:39 I mean, it’s like the worst sentence that could come out of somebody’s mouth.
0:19:43 Why is that the worst? I’ve heard that a lot. Because you’re letting people know that all you
0:19:47 care about is like, did you win or lose? That’s fantastic. Be results oriented to all you want.
0:19:52 You should pay by the piece. You will get much faster work. But the minute that you’re asking
0:19:56 people to do intellectual work, results oriented is like the worst thing that you could say to
0:20:01 somebody. So I think that we need to take responsibility. And the people in our orbit,
0:20:06 we can make sure at minimum that we aren’t making it worse. And I think that that’s,
0:20:10 so that’s pessimistic and optimistic. I don’t think anyone’s making a full reversal here.
0:20:13 So the second question then goes to the societal aspect of this.
0:20:18 And so we’ll talk about the role of the storytellers or as they’re sometimes known,
0:20:22 the journalists and the editors and the editors and the publishers. And so the very
0:20:26 first reporter I ever met when I was a kid, this is Jared Sandberg at the Wall Street Journal.
0:20:30 The internet was first emerging. Like there were no stories in the press about the internet.
0:20:33 And I used to say, like, there’s all this interesting stuff happening. Why am I not
0:20:36 reading about any of it in these newspapers? And he’s like, well, because
0:20:39 the story of something is happening is not an interesting story. He said,
0:20:42 there are only two stories that sell newspapers. He said, one is, oh, the glory of it.
0:20:45 And the other is, oh, the shame of it. And basically he said, it’s conflict. So it’s
0:20:48 either something wonderful has happened or something horrible has happened. Like those
0:20:51 are the two stories. And then you think about business journalism as kind of our domain.
0:20:54 You got to think about it. You’re like, those are the only two profiles of a CEO
0:20:57 or a founder you’ll ever read. It’s just like, what a super genius for doing something,
0:21:01 presumably not consensus and right, or what a moron. Like what a hopeless idiot for doing
0:21:05 something, not consensus and wrong. And so, and so since I’ve become more aware of this,
0:21:09 like it’s actually gotten, it’s gotten very hard for me to actually read any of the coverage
0:21:12 of the people I know, because it’s like the people who got not consensus, right,
0:21:15 they’re being lavished with too much praise. And the people who got not consensus wrong,
0:21:19 they’re being damaged for all kinds of reasons. The traits are actually the same in a lot of
0:21:25 cases. And so I guess as a consequence, like if you read the coverage, it really reinforces this
0:21:30 bias of being results oriented. And it’s like, it’s not our fault that people don’t want to
0:21:34 read a story that says, well, he tried something and it didn’t work this time, right?
0:21:34 Yes, exactly.
0:21:35 And so is there a…
0:21:38 But it was mathematically pretty good. If we go back to Pete Carroll,
0:21:43 this is a pretty great case. If we think about options theory, that just quickly the past preserved
0:21:47 the option for two run plays. So if you want to get three tries at the end zone instead of two,
0:21:51 for strictly for clock management reasons, you pass first.
0:21:54 Right. And that’s not going to kick off ESPN Sports Center that night. And so optimistic or
0:22:00 pessimistic that the narrative, the public narrative on these topics will ever move.
0:22:08 I’m super, super pessimistic on the societal level, but I’m optimistic on if we’re educating
0:22:13 people better, that we can equip them better for this. So I’m really focused on how do we
0:22:19 make sure that we’re equipping people to be able to parse those narratives in a way that’s more
0:22:27 rational. And particularly, now there’s so much information. And it’s all about the framing
0:22:33 and the storytelling. And it’s particularly driven by what’s the interaction of your own
0:22:36 point of view. We could think about it as partisan point of view, for example, versus
0:22:40 the point of view of the communicator of the information and how is that interacting with
0:22:44 each other, in terms of how critically are you viewing the information, for example.
0:22:50 I think this is another really big piece of the pie and somewhat actually related to the question
0:22:53 about journalism, which is that third dimension of the space. So we talked about two dimension,
0:22:57 which is sort of outcome quality, and how are you allowing that you’re exploring both
0:23:02 downside and upside outcomes in a way that’s really looking at forecast. How are you thinking
0:23:07 directionally, so that you’re more directionally neutral. But then the other piece of the puzzle
0:23:13 is how are you treating omissions versus commissions? So one of the things that we know
0:23:19 with this issue of resulting is, here’s a really great way to make sure that nobody ever
0:23:26 results on you. Don’t do anything. If I just don’t ever make a decision, I’m never going to
0:23:31 be in that room with everybody yelling at me for the stupid decision I made, because I had a bad
0:23:36 outcome. But we know that not making a decision is making a decision. We just don’t think about it
0:23:40 that way. And it doesn’t have to just be about investing. You can have a shadow of your own
0:23:44 personal decision. So, you know, it’s really interesting. I remember I was giving somebody
0:23:51 advice who was like 23. And so obviously, you know, newly out of college had been in this position
0:23:56 for a year and was really, really unhappy in the position. And he was asking me like, I don’t know
0:24:01 what to do. I don’t know if I should change jobs. And I said, well, you know, so I did all the tricks,
0:24:04 you know, time traveling. And so I was like, well, okay, imagine it’s a year from now. Do you
0:24:09 think you’re going to be happy in this job? No. Okay, well, maybe you should go and choose this
0:24:14 other, maybe you should go and try to find another position. And this is what he said to me. And this,
0:24:18 I think, shows you how much people don’t realize that the thing you’re already doing, the status
0:24:23 quo thing, choosing to stay in that really is a decision. So he said to me, but if I go and find
0:24:28 another position, and then I have to spend another year, which I just spent trying to learn the ins and
0:24:33 outs of the company. And it turns out that I’m not happy there. I’ll have wasted my time. And I said
0:24:38 to him, okay, well, let’s think about this, though, the job you’re in, which is a choice to stay in,
0:24:44 you’ve now told me it’s 100% in a year that you will be sad. Then if you go to the new job,
0:24:49 yes, of course, it’s more volatile. But at least you’ve opened your, you’ve opened the range of
0:24:54 outcomes up, but he didn’t want to do it because it doesn’t feel like staying where he was, didn’t
0:24:59 feel like somehow he was choosing it. So that he felt like if he went to the other place, he
0:25:04 ended up sad that somehow that would be his fault in a bad decision. So profound. In my case,
0:25:08 this is my beginning a little too personal, but in my case, it was a decision that I didn’t know
0:25:13 I had made to not have kids. And it’s still an option, but it’s probably not going to happen.
0:25:18 And my therapist kind of told me that my not deciding was a choice. And I was like so blown
0:25:23 away by that, that it had then allowed me to then examine what was going on there in that
0:25:28 framework in order to not do that for other arenas in my life, where I might actually
0:25:32 want something, or maybe I don’t, but at least it’s a choice that there’s intentionality behind
0:25:36 it. Well, I appreciate you sharing. I mean, I really want to thank you for that because I think
0:25:41 that people, first of all, should be sharing this kind of stuff so that people feel like they can
0:25:45 talk about these kinds of things. Number one, and number two, in my book, I’ve got all these
0:25:49 examples in there of like, how are you making choices about raising your kids when it feels so
0:25:52 consequential? You do decisions for other people. Right. And you’re trying to decide like,
0:25:56 should I have kids or shouldn’t I have kids? Or this school or that school or where am I supposed
0:26:03 to live? And the thing that I try to get across is, we can talk about investing like I’m putting
0:26:09 money into some kind of financial instrument, but we all have resources that we’re investing.
0:26:14 That’s right. It’s our time, your energy, your heart. It could be whatever, your friendships,
0:26:19 your relationships. So you’re deploying resources and like for the kind of decision that you’re
0:26:24 talking about, it’s like, if you choose to have children, you’re choosing to deploy
0:26:29 certain resources with some expected return, some of it good, some of it bad. And if you’re
0:26:34 choosing not to have children, that’s a different deployment of your resources toward other things.
0:26:38 And you need to know that there are limits. Everything isn’t a zero-sum game.
0:26:43 No. But approaching the world and the fact that evolution has approached the world as a zero-sum
0:26:48 game and our toolkit makes it a zero-sum game, means that we need to still view everything
0:26:52 as a zero-sum game when it comes to those trade-offs and resources. Because you are losing
0:26:56 something every time, even in a non-zero game. Right. So I don’t feel like the world is a zero-sum
0:27:01 game in terms of like, most of the activities that you and I would engage on, we can both win too.
0:27:06 But it’s a zero-sum game to go back to your therapist. It’s a zero-sum game between you
0:27:10 and the other versions of yourself that you don’t choose. Exactly. Or an organization and
0:27:14 the other versions of itself it doesn’t choose. Exactly. So there’s a set of possible futures
0:27:20 that result from not making a decision as well. So on an individual decision, let’s put things
0:27:26 into three categories. Clear misses, near misses, and hits. There’s some that would just be a clear
0:27:30 miss, throw them out. And there’s some that I’m going to sort of really agonize over and I’m going
0:27:36 to think about it and I’m going to do a lot of analysis on it. And so the ones which become a
0:27:42 yes go into the hit category. And the other one is a near miss. I came close. What happens with
0:27:48 those near misses is they just go away. So what I realized is that on any given decision, let’s
0:27:52 take an investment decision. If I went to you or you came to me and said, well, tell me what’s
0:27:57 happening with the companies that you have under consideration. On a single decision,
0:28:02 when I explain to you why I didn’t invest in a company, it’s going to sound incredibly reasonable
0:28:07 to you. So you’ll only be able to see in the aggregate, if you look across many of those
0:28:13 decisions, that I tend to be having this bias toward missing. Towards saying, you know what,
0:28:18 we’re not going to do it so that I don’t want to stick my neck out. Now this for you is incredibly
0:28:21 hard to spot because you do have to see it in the aggregate because I’m going to be able to
0:28:27 tell you a very good story on any individual decision. So the way to combat that and again,
0:28:31 get people to think about what we really care around here as forecast, not really outcomes,
0:28:36 is actually to keep a shadow book. The anti portfolio should contain basically all of your
0:28:41 near misses, but then you have to take a sample of the clear misses as well, which nobody ever looks
0:28:46 at because the near misses tend to be a little in your periphery. If they happen to be big hits.
0:28:50 So here’s the problem. So the good news is bad news. So the good news is we have actually done
0:28:54 this. And so we call it the shadow portfolio. Awesome. And the way that we do it is we make
0:28:59 the investment. We take the other equivalent deal of that vintage of that size that we almost did,
0:29:02 but didn’t do. We put that in the shadow portfolio and we’re trying to do kind of
0:29:07 apples to apples comparison. In finance theory terms, the shadow portfolio may well outperform
0:29:11 the real portfolio. And in finance terms, that’s because the shadow portfolio may be higher variance,
0:29:16 higher volatility, higher risk, and therefore higher return. Because the fear is the ones that
0:29:20 are hitting are the ones that are less, they’re less spiky, they’re less volatile, they’re less
0:29:25 risky. Right. So what’s wonderful about that, when you decide not to invest in a company,
0:29:29 you actually model out why that’s in there. It’s often, by the way, it’s often a single flaw that
0:29:34 we’ve identified. Yeah. Like it’s just like, oh, we would do it except for X. Right. Where X looks
0:29:37 like something that’s like potentially existentially bad. Right. And then that’s just
0:29:41 written in there. And so you know that. So, and then just make sure people, like those ones that
0:29:45 people are just rejecting out of hand. That’s my question. So we never do that. But let me ask
0:29:48 you how to do that though. So that’s what we don’t do. And as you’re describing,
0:29:51 I’m like, of course we should do that. I’m trying to think of how we would do that. Because the
0:29:57 problem is we reject 99 for everyone we do. Yeah. So you just literally, it’s a sample. You just
0:30:00 take a random sample. A random sample. Okay. I mean, as long as it’s just sort of being kept
0:30:06 in view a little bit, because what that does is it basically just asks as pushing against your model.
0:30:10 You’re just sort of getting people to have the right kind of discussion.
0:30:15 So all of that communicates to the people around you, like, I care about your model.
0:30:19 So let me ask you a different question on, because you talk about sort of groups of decisions.
0:30:22 So the other question, portfolios of decisions. So the other question is, early on in the firm,
0:30:25 I happen to have this discussion with a friend of mine. And he basically looked at me and he’s like,
0:30:29 you’re thinking about this all wrong. You were thinking about this as a decision. You’re thinking
0:30:32 about investor not. He said, that’s totally the wrong way to think about this. You should be thinking
0:30:38 about this is, is this one of the 20 investments of this kind, of this class size that you’re
0:30:42 going to put in your portfolio. When you’re evaluating an opportunity,
0:30:47 you are kind of definitionally talking about that opportunity. But it’s very hard to
0:30:51 abstract that question from the broader concept of a portfolio or a basket.
0:30:54 Yeah. What I would suggest there is actually just doing some time traveling that as people
0:30:58 are really down in the weeds to say, let’s imagine it’s a year from now, and what does the portfolio
0:31:04 look like of these investments of this kind. So I’m a big promoter of time traveling, of just
0:31:08 making sure that you’re always asking that question, what does this look like in a year?
0:31:11 What does this look like in five years? Are we happy? Are we sad?
0:31:15 If we imagine that we have this, what percentage of this do we think will have failed?
0:31:19 We understand that any one of these individual ones could have failed. So let’s remember that.
0:31:24 And I think that that really allows you to sort of get out of what feels like the biggest decision
0:31:29 on earth because that’s the decision you have to be making and be able to see it in the context of
0:31:35 kind of all of what’s going on. It’s fantastic. One of the most powerful things my therapist
0:31:39 gave me, and it was such a simple construct. It was sort of like doing certain things today is
0:31:45 like stealing from my future self. Oh, it blew me. It blew my mind. So beautiful.
0:31:49 It’s so beautiful. And it seems so like, you know, hokey, like personal self help you. But
0:31:55 actually I had never thought of because we were on a continuum by making discreet individuals
0:31:59 like Sonal in the past, Sonal today, Sonal this woman in the future I haven’t met yet.
0:32:06 Wow. Like the idea of stealing from her was like I… That’s really a lovely way to put it.
0:32:10 Yeah, she is. I have an amazing therapy. I like talking publicly about therapy because
0:32:15 I like to stick on it. No, I’m very, very open about like, let’s not hide it. It’s totally fine.
0:32:19 There’s no fucking reason to hide it. I totally agree. Some of the ways that we deal with this
0:32:23 is actually prospectively employing really good decision hygiene, which involves a couple of
0:32:28 things. One is some of this good time traveling that we talked about where you’re really imagining
0:32:32 what is this going to look like in the future so that that’s metabolized into the decision.
0:32:39 Two is making sure that you have pushback once there’s consensus reached. Great. Now let’s go
0:32:44 disagree with each other. Then the next thing is in terms of the consensus problem is to make
0:32:51 sure that you’re eliciting as much input not in a room with other people. So, you know,
0:32:55 when somebody has a deal they want to bring to everybody that goes to the people individually,
0:32:58 they have to sort of write their thoughts about it individually. And then it comes into the
0:33:01 room after that. As opposed to the pile on effect that just happened. As opposed to the pile on
0:33:06 effect. And that reduces the sort of effects of consensus anyway. So now this is how you then come
0:33:11 up with basically what your forecast of the future is that then is absolutely memorialized because
0:33:15 that memorializing of it acts as the prophylactic. First of all, it gives you your forecast, which
0:33:19 is what you’re trying to push against anyway. You’re trying to change the attitude to be that
0:33:24 the forecast is the outcome that we care about. And it acts as a prophylactic for those emotional
0:33:28 issues, right? Which is now you, it’s like, okay, well, we all talked about this and we had our
0:33:33 red team over here and we had a good steel man going on. And we kind of really thought about
0:33:39 why we were wrong. We questioned if somebody, you know, let’s, if somebody has the outside view,
0:33:45 what would this really look like to them by eliciting the information individually. We were
0:33:51 less likely to be in the inside view anyway. We’ve done all of that good hygiene. And then that acts
0:33:56 as a way to, to protect yourself against these kinds of issues in the first place. Again,
0:34:02 you’re going to have a bad 24 hours. I’m just like, for sure. But you can get out of it more
0:34:07 quickly, more often and get to a place where you can say, okay, moving on to the next decision,
0:34:11 how do I, how do I improve this going forward? Yeah. So building on that, but returning real
0:34:16 quick to my optimism pessimism question, if society is not going to move on these issues,
0:34:19 but we can move as individuals. So one form of optimism would be more of us can move as
0:34:23 individuals. The other form of optimism could be there will just always be room in these
0:34:26 probabilistic domains for the rare individual who’s actually able to think about this stuff
0:34:30 correctly. Like there will always be an edge. There will always be certain people who are
0:34:34 like much better at poker than everybody else. There will. Oh, I think that’s for sure. Okay.
0:34:38 Because most people simply, most people just simply can’t or won’t get there. Like a few
0:34:42 people in every domain might be able to take the time and have the discipline of willpower to kind
0:34:46 of get all the way there. But most people can’t or won’t. I think that in some, in some ways, maybe
0:34:51 that, that’s okay. Like, I mean, I sort of think about from an evolutionary standpoint, that kind
0:34:55 of thinking was selected for, for a reason, right? Like it’s better for survival, likely better for
0:34:59 happiness. You mean the conventional, conventional wisdom? Yeah. Don’t touch the burn stove twice.
0:35:03 Yeah. Or run away when you hear rustling in the leaves. Don’t sit around and say, well, it’s a
0:35:06 probabilistic world. I have to figure out how often is that a lion that’s going to come eat me?
0:35:09 Most people shouldn’t be playing in the World Series of Poker. I have people come up to me all
0:35:14 the time and be like, Oh, you know, I play poker, but it’s just a home game. You know, and I’m like,
0:35:17 what are you saying? Just a home game. Like there are different purposes to poker. Like
0:35:21 you probably have a great time doing that. And it brings you a tremendous amount of enjoyment.
0:35:24 And you don’t have an interest in becoming a professional poker player and why just be
0:35:30 proud of that. I think that that’s amazing. Like I play tennis. I’m not saying, Oh, but you know,
0:35:36 I’m just playing week, you know, I’m just playing in like USTA, like 3.5. Like I’m really happy with
0:35:42 my tennis. I think it’s great. So I think we need to remember that like people have different things
0:35:48 that they love. And this kind of thinking, I think that I would love it if we could spread it more.
0:35:52 But of course, there are going to be some people who are going to be ending up in this
0:35:56 category more than others. And that’s okay. Like not everybody has to think like this. I think
0:36:00 it’s all right. So one of the things I get asked all the time is like, well, we can’t really do
0:36:06 this because people expect us to be confident in our choices. Don’t confuse confidence and certainty.
0:36:12 So I can express a lot of uncertainty and still convey confidence. Ready? I’m weighing these
0:36:16 three options, A, B and C. I’ve really done the analysis. Here’s the analysis. And this is what
0:36:22 I think. I think that option A is going to work out 60% of the time. Option B is going to work
0:36:27 out 25% of the time. And option C is going to work out 15% of the time. So option A is the clear
0:36:33 winner. Now, I just expressed so much uncertainty in that sentence. But also a lot of confidence.
0:36:37 But also a lot of confidence. I’ve done my analysis. This is my forecast. And all that I
0:36:42 ever ask people to do when they do that is make sure that they ask a question before they bank
0:36:46 the decision, which is, is there some piece of information that I could find out that would
0:36:51 reverse my decision that would actually cause, not that would make it go from 60 to 57. I don’t
0:36:55 care modulating so much. I care that you’re going to actually change. And your point is that organizations
0:36:59 can then bake that into their process. And not just in the forecasting, but in arriving to that
0:37:04 decision. So that then the next time they get to it right or wrong, they make a better decision.
0:37:10 And if the answer is yes, go find it. Or sometimes the answer is yes, but the cost is too high. It
0:37:15 could be time. It could be opportunity costs, whatever. Exactly. So then you just don’t. And
0:37:18 then you would say, well, then you all recognize as a group, we knew that if we found this out,
0:37:22 it would change our decision. But we’ve agreed that it would, the cost was too high. And so we
0:37:25 didn’t. So then if it reveals itself afterwards, you’re not sad. Well, you’ve talked a lot about
0:37:29 how people should use confidence intervals and communicating, which I love because we’re both
0:37:37 ex-PhD psychology people, neither are finished. So I love that idea. One thing that I struggle with,
0:37:40 though, is again, in the organizational context, like if you’re trying to translate this to a
0:37:46 big group of people, not just one-on-one or small group decisions, how do you communicate a confidence
0:37:52 interval and all the variables in it in an efficient kind of compressed way? Like honestly,
0:37:57 part of communication and organizations is emails and quick decisions. And yes, you can have all
0:38:03 the process behind the outcome. But how do you then convey that even though the people were not
0:38:08 part of that room of that discussion? I think that there’s a simpler way to express uncertainty,
0:38:12 which is using percentages. Now, obviously, sometimes you can only come up with a range.
0:38:19 But for example, if I’m talking to my editor, and this is very quick in an email, I’ll say,
0:38:24 you’ll have the draft by Friday, 83% of the time. By Monday, you’ll have it 97% of the time.
0:38:27 Those are inclusive, right? That’s another way of doing a confidence interval, but without
0:38:32 making it so wonky. Without making it so wonky. So I’m just letting her know, most of the time,
0:38:37 you’re going to get it on Friday, but I’m building, like my kid gets sick or I have trouble with a
0:38:42 particular section of the draft or whatever it is, and I set the expectations for that way.
0:38:45 That’s fantastic. I mean, we’ve been trying to do forecasting even for like timelines for
0:38:49 podcasts, editing and episodes. And I feel frustrated because I have like a set of frameworks,
0:38:55 like if there’s accents, if there’s more than two voices, if there’s, you know, a complexing,
0:38:59 room tone, like interaction, feedback, sound effects. I know all the factors that can go into
0:39:05 my model, but I don’t know how to put a confidence interval in our pipeline spreadsheet for, you
0:39:09 know, all the content that’s coming out. Yeah. So one way to do it is think about what’s the range,
0:39:13 what’s the earliest that I could get it, and you put a percentage on that. And then you think about
0:39:17 the latest day, they’re going to get it. And you put a percentage on that. And so now,
0:39:23 what’s wonderful about that is that it’s a few things. One is I’ve set the expectations properly
0:39:28 now so that I’m not getting, you know, yell that on Friday, like where the hell’s the draft.
0:39:32 Exactly. And I think that, and a lot of what happens is that because we’re sort of, we think
0:39:37 that we have to give a certain answer. It ends up, boy, who cried wolf, right? So that if I’m
0:39:44 telling her, I’m going to get it on Friday and, you know, 25% of the time, 25% of the time I’m
0:39:50 late, she just starts to not put much stock in what I’ve said anyway. So that’s number one.
0:39:54 Number two is what happens is that you really kind of infect other people with this in a good
0:39:59 way, where you get them, it just moves them off of that black and white thinking. So like,
0:40:02 I love that. One of the things that I love thinking about, and this is the difference
0:40:09 between a deadline or kind of giving this range, is that I think that we ask ourselves,
0:40:16 am I sure? And other people, are you sure way too often that it’s a terrible question to ask
0:40:20 somebody because the only answer is yes or no. So what should we be asking? How sure are you?
0:40:25 How sure are you? I have a quick question for you on this because earlier you mentioned uncertainty.
0:40:30 How do you as an organization build that uncertainty in by default? So first of all,
0:40:34 we obviously talked a little bit about time traveling and the usefulness of time traveling.
0:40:39 So one thing that I like to think about is not overvalue the decision that’s right at hand,
0:40:45 the things that are right sitting in front of us. So you can kind of think about it like,
0:40:50 how are you going to figure out the best path? As you think about what your goals are and obviously
0:40:54 the goal that you want to reach is going to sort of define for you what the best path is.
0:40:58 If you’re standing at the bottom of a mountain that you want to summit, let’s call the summit your
0:41:02 goal, all you can really see is the base of the mountain. So as you’re doing your planning,
0:41:06 you’re really worried about how do I get the next little bit, right? How do I start?
0:41:10 But if you’re at the top of the mountain having a tanger goal, now you can look at the whole
0:41:14 landscape. You get this beautiful view of the whole landscape and now you can really see what
0:41:18 the best path looks like. And so we want to do this not just physically like standing up on a
0:41:23 mountain, but we want to figure out a cognitive way to get there. And that’s to do this really good
0:41:28 time traveling. And you do this through back casting and premortem. And now let’s look backwards
0:41:33 instead of forwards to try to figure out, this is now the headline. Let me think about why that
0:41:37 happened. So you could think about this like as a simple like weight loss goal. I want to lose a
0:41:41 certain amount of weight within the next six months. It’s the end of the six months. I’ve lost
0:41:47 that weight. What happened? You know, I went to the gym. I avoided bread. I didn’t eat any sweets.
0:41:53 I made sure that, you know, whatever. So you now have this list. Then in pairing with that,
0:41:59 you want to do a premortem, which is I didn’t get to the top of the mountain. I failed to lose
0:42:03 the weight. I failed to do whatever it is. And then all the things you can do to counter program
0:42:07 against that. Exactly. Because that’s going to reveal really different things. It’s going to reveal
0:42:13 some things that are just sort of luck, right? Let me think, can I do something to reduce the
0:42:17 influence of luck there? Then there’s going to be some things that have to do with your
0:42:22 decisions. Like I went into the break room every day and there were donuts there. And so I couldn’t
0:42:26 resist them. So now you can think about how do I counter that, right? How can I bring other people
0:42:30 into the process and that kind of thing? And then there’s stuff that’s just, you can figure out,
0:42:33 it’s just out of your control. It turned out out of slow metabolism. And now what happens is that
0:42:36 you’re just much less reactive and you’re much more nimble because you’ve gotten a whole view of
0:42:40 the landscape and you’ve gotten a view of the good part of the landscape and the bad part of the
0:42:46 landscape. But I’m sure as he told you, people are very low to do these premortems because I think
0:42:52 that the imagining of failure feels so much like failure that people are like, no, and you should
0:42:56 posit, you know, positive visualization. I mean, even in brainstorming meetings, everyone’s like,
0:43:00 don’t dump on an idea. But the exact point is you have to dump on an idea and kill the winnowing
0:43:06 of options. No. As part of the process, you should be then premorteming it. Exactly. There’s
0:43:11 wonderful research by Gabrielle Adinchin that I really recommend that people see that the references
0:43:18 are in my book. In across domains, what she’s found is that when people do this sort of positive
0:43:22 fantasizing, the chances that they actually complete the goal are just lower than if people
0:43:27 do this negative fantasizing. And then there’s research that shows that when people do this
0:43:32 time travel and this backwards thinking that increases identifying reasons for success or
0:43:39 failure by about 30%, you’re just more likely to see what’s in your way. So like, for example,
0:43:44 she did like one of the simple studies was she asked people who were in college,
0:43:49 you know, who do you have a crush on that you haven’t talked to yet? She had one group who,
0:43:53 you know, it was all positive fantasies. So like, oh, I’m going to meet them and I’m going to ask
0:43:56 them out on a date and it’s going to be great and then we’re going to live happily ever after and
0:44:01 whatever. And then she had another group that engaged in negative fantasizing. What if I asked
0:44:06 them out and they say no? Like they said no and I was really embarrassed and so on and so forth.
0:44:12 And then she revisited them like four months later to see which group had actually gone out on a date
0:44:16 with the person that they had a crush on and the ones that did the negative fantasizing were much
0:44:20 more likely to have gone out on the date. It’s fantastic. Yeah. So one of the things that I
0:44:25 say is like, look, when we’re in teams to your point, we tend to sort of view people as naysayers,
0:44:32 right? But we don’t want to think of them as downers. So I suggest divide those up into two
0:44:37 processes. Have the group individually do a back cast, have the group individually write a narrative
0:44:42 about a pre-mortem. And what that does is when you’re now doing a pre-mortem, it changes the
0:44:47 rules of the game where being a good team player is now actually identifying the ways that you fail.
0:44:51 I love what you said because it’s like having two modes as a way of getting into these two
0:44:55 mindsets. Right. Where you’re not stopping people from feeling like they’re a team player. And I
0:44:59 think that that’s the issue. As you said, it’s like, don’t sit there and like, you know, crap on
0:45:04 my goal. Well, because what are they really saying? You’re not being a team player. So change the
0:45:09 rules of the game. You had this line in your book about how regret is an unproductive. The issue is
0:45:14 that it comes after the fact, not before. So the one thing that I don’t want people to do is think
0:45:18 about how they feel right after the outcome. Because I think that then you’re going to overweight
0:45:25 regret. So you want to think about regret before you make the decision. You have to get it within
0:45:29 the right timeframe. What we want to do instead is write in the moment of the outcome when you’re
0:45:34 feeling really sad. You can stop and say, am I going to care about this in a year? Think about
0:45:39 yourself as a happiness stock. And so if we can sort of get that more 10,000 foot view on our own
0:45:45 happiness and we think about ourselves as we’re investing in our own stock, our own happiness
0:45:50 stock, we can get to that regret question a lot better. You don’t need to improve that much
0:45:57 to get really big dividends. You make thousands of decisions a day. If you can get a little better
0:46:04 at this stuff, if you can just, you know, de-bias a little bit, think more probabilistically,
0:46:10 really sort of wrap your arms around uncertainty to free yourself up from sort of the emotional
0:46:15 impact of outcomes. A little bit is going to have such a huge effect on your future decision making.
0:46:19 Well, that’s amazing, Annie. Thank you so much for joining the A6NZ podcast.
0:46:21 -Thank you very much. -Yes, thank you.