0:00:05 The content here is for informational purposes only, should not be taken as legal business
0:00:10 tax or investment advice or be used to evaluate any investment or security and is not directed
0:00:15 at any investors or potential investors in any A16Z fund. For more details, please see
0:00:21 A16Z.com/disclosures. Hi, and welcome to the A16Z podcast. I’m Lauren Murrow, and today
0:00:26 we’re talking about when fintech meets social. It’s a trend that’s evident on both ends
0:00:30 of the spectrum, whether that’s people divulging their crushing levels of debt on Instagram
0:00:35 and Twitter or bragging about their credit scores and stock trades. In this hallway-style
0:00:39 conversation with fintech general partner Anish Acharya and Darcy Kulikin, a partner on the
0:00:46 consumer tech team, we discuss why this holy grail of social plus fintech is both so challenging
0:00:50 and potentially so rewarding. We’ll cover which products and companies are taking advantage
0:00:55 of it, how it’s being driven by various subcultures online, and why this shift is happening now,
0:00:59 which is where this conversation begins. The first voice you’ll hear is Anish, followed
0:01:02 by Darcy. So the fact that people are actually talking
0:01:07 publicly about their debt is a new behavior. In the past, spending was public, but debt
0:01:12 was private. And for the first time, debt is starting to become a public conversation.
0:01:17 What’s new is that this generation is living in a completely different socioeconomic context.
0:01:21 That’s not flighty millennials and zoomers or whatever. That’s a completely different
0:01:25 financial world that they’re growing up in, and that’s driving a different set of conversations.
0:01:29 You see certain categories that people are now talking about that they didn’t talk about
0:01:33 before. Salary is something that a certain generation is much more comfortable talking
0:01:38 about. Student debt is a category that people are much more comfortable talking about. Trading
0:01:42 is a category that people are much more comfortable talking about. Across the spectrum, you see
0:01:48 sharing on social of financial stuff going up. You see it on Twitter, you see it on Facebook,
0:01:51 you see it in blogs. There’s a bunch of pockets.
0:01:57 Why do you think this shift is happening? I think it’s driven by a few factors. One
0:02:02 is generational. Every generation’s relationship with sharing and every generation’s relationship
0:02:06 with money is different. So what boomers did versus what Gen X did versus what millennials
0:02:11 do versus what Gen Z does is different. And I think you see this macro trend around increasing
0:02:15 sharing. And that’s driven by historical changes. That’s driven by the financial crisis.
0:02:20 Exactly. They have to take nontraditional paths to achieve financial progress and dreams.
0:02:25 For a long, long time, buying a home was not only the American dream, but something you
0:02:29 achieved through the traditional financial system. So everyone had a mortgage. Today
0:02:33 mortgages are less accessible than they’ve ever been. Will you talk to your peer set
0:02:37 about how am I ever going to buy a home? And that’s really the catalyst behind many of
0:02:39 these things. And I think you see that also with the massive
0:02:44 increase in student debt over the last 10, 15 years. It’s reaching unsustainable levels
0:02:48 and that’s forcing a conversation. And then that breaks down the stigma about talking
0:02:51 about student debt. And then once you break the stigma, then it’s like, hold on and everything
0:02:55 comes flooding to the forefront. We’ve talked about how money is inherently
0:03:01 private. Do you not think that that is becoming less so? There’s a generational piece of it.
0:03:05 Then yes, we’re sharing more of our lives in general. And then there’s a political angle
0:03:09 to it, this idea of radical transparency to affect change. So that’s why we’re posting
0:03:13 more about student debt, about medical debt, about our salaries.
0:03:19 Usually there is a long-term trend line towards sharing more rather than sharing less. But
0:03:24 you see it happening at the category level and to a certain extent at the subculture
0:03:28 level. So take student debt as one category. When people start talking about it, then everybody
0:03:34 feels empowered to talk about it. And I think you need catalysts for walls to come down around
0:03:38 certain categories like the student debt crisis, a financial crisis. There’s a lot of external
0:03:42 events that have led to some of these things come down, but it’s happening inch by inch
0:03:46 and category by category. And the question is, what pieces are going mainstream?
0:03:52 I think the hacker mindset has pushed outside of software and into finance. There is always
0:03:57 a small number of people who are excited about hacking their money, but now that’s becoming
0:04:02 a more mainstream concept. So the idea of being someone who arbitrages rewards across
0:04:07 credit cards used to be a pretty niche edge thing. And now more and more people are doing
0:04:10 it to the point where a lot of card companies are having to pull rewards back because there’s
0:04:14 a points guy and a million other sites that tells you how to actually hack the system.
0:04:19 And credit scores are very similar. It’s not a destiny. It’s a game, or it’s at least closer
0:04:23 to a game than a destiny. And more people are talking about the ways that you play it.
0:04:28 When I say it’s a game, I say that in a hopeful way, but not in a dismissive way in terms of
0:04:29 the importance of it.
0:04:32 It also goes to what are the things people like to do on social? And three of the core
0:04:37 functions are bragging, complaining, and rubbernecking. And I just think you’ve seen where social and
0:04:42 finance intersect, kind of coalescing around those three use cases as well.
0:04:45 At the end of the day, social and finance, a lot of it is just content, and it’s content
0:04:49 that’s anchored around some financial transaction, but it’s still just content. And so the usual
0:04:53 rules of social apply. Another way to think about it is, when you’re building something
0:04:58 in social plus finance, you have an interaction layer, and you have a transaction layer. And
0:05:02 the interaction layer is built around the emotional and cognitive pieces. And that is
0:05:06 content creation, that is messaging, that is all these social things that we see pop
0:05:11 up that appeal to these cognitive and emotional levers. And then you have a transactional
0:05:15 layer, which is whatever your actual financial transaction is. And that’s generally much
0:05:21 more of a functional use case. The magic in social plus finance happens when the transactional
0:05:25 piece and the interactive piece are mutually reinforcing, and that’s where the flywheel
0:05:29 on social plus finance really starts to spin aggressively.
0:05:33 Can you give me some examples of particular products in which you’ve seen this magic happen?
0:05:38 The easiest example is probably Venmo back in the day, where you had messaging apps,
0:05:42 money transfer apps like PayPal existed, and Chat existed, but the idea that you could
0:05:46 attach your transaction to an emoji just made the transaction easier, it made the emoji
0:05:51 more fun, it made the whole thing more self-reinforcing. It’s a really challenging problem to be able
0:05:53 to do that, but when you do do it, it’s magic.
0:05:58 I actually think that those products are fascinating. I still like to scroll through the global
0:06:03 feed on Venmo, which now is capped at the last 50 transactions, but it’s just so fascinating
0:06:07 to see all of these people all over the country sending each other money. There’s something
0:06:13 that is just vicariously thrilling about it, and because money does touch all of us, and
0:06:17 it’s so private, and the products that can start to invert that, I think they just touch
0:06:21 a nerve in an interesting way. And by the way, it doesn’t have to only be online. So there’s
0:06:27 a couple of interesting offline examples, SoFi, which is really in the business of refinancing
0:06:34 mispriced student debt, built this whole community of Henry’s, high earning, not rich yet, did
0:06:40 a ton of parties and events, and made it feel special to be a SoFi member, and really they
0:06:45 were a lender. So I think they’ve actually, at least in the early days, had a lot of success
0:06:51 combining the two. I imagine it’s less successful as Capital One is now opening coffee shops
0:06:56 where you can hang out and get coffee and do your banking, I guess, and it’s easy to
0:07:00 dismiss that as clumsy, but I do think that they’re trying to touch the same nerve.
0:07:06 There’s also this long legacy of companies starting out at the nexus of social and fintech,
0:07:10 and then eventually moving one way or the other generally towards the fintech transactional
0:07:14 layer. So a lot of people who can build either features or community in the early days and
0:07:19 really use it as a way to bootstrap their product, but then over time it migrates more
0:07:23 towards a transactional fintech product rather than a truly social product as well.
0:07:24 What are some of those examples?
0:07:28 SoFi is a great example. It’s functionally a lender, which is not a multiplayer social
0:07:31 game, but they were able to build this early community, which is able to get them a lot
0:07:35 of traction. You look at Wealthfront. Before it transitioned into Wealthfront, I think it
0:07:40 started as Ka-ching, which was a social fintech product. If you look Robinhood originally,
0:07:45 it was a much more social product, then became a much more transactional product. Prosper
0:07:48 started as a much more social product, and then became more of a peer-to-peer lending
0:07:53 platform. A lot of these things start social and are able to bootstrap in their early days
0:07:57 off of some of those networks. Then you end up at a decision point where you try to thread
0:08:01 this needle and continue down the social plus finance angle, or do you move into a more
0:08:05 single-player fintech product? I think a lot of the more successful fintech companies have
0:08:07 started social, but then eventually transitioned.
0:08:09 Why are they making that transition?
0:08:10 It’s hard.
0:08:14 Well, let’s talk about it. What is so hard about social fintech?
0:08:19 The most direct manifestation of social plus fintech is we have messaging, plus we have
0:08:24 payments or some other shared accounts, shared ledgers, some other joint accounts, et cetera.
0:08:29 I think that is very difficult for a number of reasons because money is so private. People
0:08:35 are less likely to send invites to each other and bootstrap a social product in the way
0:08:39 that you would bootstrap other social products.
0:08:43 I think there’s a lot of other examples, though, where the experience may not directly represent
0:08:48 social plus money, but it very much plays to that. I think the example Darcy brought up
0:08:52 is great, which is Robinhood. There’s been a ton of talk about how Robinhood is doomed
0:08:57 because others have cut fees and adopted their business model, but in truth, Robinhood is
0:09:02 a game, and it’s a game that people like to talk about. It works because it feels like
0:09:06 adulting when you actually have a stock portfolio, not because active trading is something that’s
0:09:12 smart for almost anyone to do. I really see it as addressing a different consumer need
0:09:17 than Schwab is addressing, and it’s really not threatened as much by players like Schwab.
0:09:23 That’s an example where the fintech product is addressing a social consumer need, but
0:09:27 at first blush, it may not appear to be the combination of social plus money.
0:09:32 Some of these products are really tapping into the trend for its gamification. Do you
0:09:36 think more products will go that route and design around that impulse that we have?
0:09:41 I think the thing you will likely see is that these social plus fintech products will actually
0:09:45 come much more from the consumer side of things. I think there’s some things like Robinhood
0:09:50 where you are able to build both fintech and community, and it comes from the fintech side
0:09:53 of things. Another encouraging angle to it is the things that are coming from the social
0:09:57 side. Whether it’s a bunch of the chat apps that now have wallets and payments installed
0:10:00 in them, or even something as weird as Fortnite, which is technically a game, but they have
0:10:05 V-Bucks, and they have these economies built into them. It’ll be fascinating to see what
0:10:09 happens with those types of products, because that could be the actual place where we see
0:10:10 social plus money take off.
0:10:14 I do think, by the way, there’s been a bunch of past attempts, which maybe seem naive at
0:10:19 the time, but now just seemed like bad timing. Blippi is a famous example of this where it
0:10:23 tweeted everything that you bought, linked your credit card, and every time you swiped
0:10:29 it, it tweeted. Okay, there’s obvious reasons why that might not be a good idea, and yet
0:10:30 I think the fact that…
0:10:31 It’s too soon.
0:10:34 Hey, look, the fact that Dave Ramsey exists, and people are talking about debt and spending,
0:10:39 and there’s the nugget of truth in all these things. As Mark says, it’s rarely that the
0:10:42 idea is wrong. It’s usually that the timing is.
0:10:46 One of the interesting things about this category of company is that if you just take a step
0:10:50 back and you’re looking for a broader consumer trends, you can often look at little emergent
0:10:53 behaviors that are happening somewhere on the internet and try to figure out, is that
0:10:56 going to actually go into the mainstream at some point?
0:11:00 One of the interesting and challenging things about social plus fintech is that so much
0:11:04 of it is driven by norms, so much of it is driven around what’s taboo and what’s stigmatized,
0:11:09 and that actually exists at the subculture level. You can grow up in the same town at
0:11:13 the same age, and if you grow up on one side of town, your norms around money and sharing
0:11:16 are very different from the person on the other side of town.
0:11:20 That leads to a lot of very distinct subcultures within different pockets on the internet.
0:11:24 One of the more entertaining one is All Street Bets on Reddit, where people are posting some
0:11:29 mix of fake and real trades and explosions and everything like that, and so then you
0:11:32 can look at these things and say, “Oh, here’s this crazy emergent behavior that’s happening.
0:11:36 I think this is going to go mainstream.” In some cases, it will, or in some cases, it
0:11:40 is just part of that subculture because the norms and taboos will never translate into
0:11:46 the mainstream, but when those stigmas fall, then everything happens and everybody runs
0:11:48 for the entrance at that point.
0:11:52 Yeah, it is interesting. If you think about crypto, so this crypto as a computing platform,
0:11:57 which is how we talk about it a lot internally, but then there’s also the sort of socio-political,
0:12:02 perhaps anarchist thread of crypto. I think the historical example of that was mostly
0:12:03 gold, though at the end is taboo.
0:12:07 Nobody was screencapping their Boolean collection and sharing it on Twitter.
0:12:13 Well, depending on what Facebook group you’re in. I think, again, there is a past precedent,
0:12:17 but you’re right. There’s a functional aspect of hedging against things that may go badly
0:12:22 wrong in the future, and then there’s a cognitive, emotional, socio-political to your point, Lauren.
0:12:25 Crypto is fascinating because it’s a subculture that has a totally different relationship with
0:12:30 transparency and anonymity and all of these different dimensions and just changing the
0:12:36 form factor of value from a dollar to some sort of token has freed an entire segment
0:12:39 of people to talk about it and have a different relationship with it. It’s one of the most
0:12:45 entertaining parts of social is what’s happening in crypto. Again, the concept of crypto versus
0:12:49 concept of money created a psychological shift in some people that then made the norms around
0:12:50 it much different.
0:12:54 So, you’re saying there are these subgroups, little niche categories, but it’s difficult
0:12:57 to build a business around them until they reach that tipping point?
0:13:01 I actually think you can build great businesses around some of these subcultures. There’s
0:13:04 a lot of these quote-unquote niche, but they can be massive niches, right? Like Wall Street
0:13:06 Bets has like 800,000 members.
0:13:10 People always want to talk about how they’re making money. It’s having debt that’s always
0:13:15 been private. So, the hardest problem in terms of social and money is having people talk
0:13:20 about their debt, which is why people don’t want to have a relationship with their lender
0:13:23 or talk in too much detail about certainly their credit card debt because they feel bad
0:13:28 about it. They feel like it reflects poorly on them. Now, it’s just checking Insta right
0:13:35 now and there’s 675,000 posts for #debtfreejourney. This has become a public conversation and a
0:13:39 lot of it is happening on Instagram and I think that’s the hardest problem, the hardest
0:13:43 segment to actually unlock. So, I actually think we’re pretty far ahead right now.
0:13:47 Well, and to your point, Wall Street Bets is not just about, “I made a bunch of money.
0:13:50 It’s also people posting, “Shit, I just lost a bunch of money.”
0:13:54 Though, the subtext is, “Look at all the swagger I’ve got. I can lose all this money and it’s
0:13:55 all good.”
0:13:56 Yeah.
0:13:57 Not always.
0:14:03 Fair. Where this gets a lot more interesting is looking beyond social media and social
0:14:08 networks and starting to talk about how this stuff drives an emergence that are products
0:14:12 and how products are designed. Lauren and I both talked about this, which is the concept
0:14:17 that as a product, you can create value in a functional way, which is, “Hey, my credit
0:14:23 score was X and now it’s X plus Y.” You can create value in a cognitive way, which is,
0:14:27 “Hey, I now better understand my credit score,” or you can create value in an emotional way,
0:14:32 which is, “I feel better about my credit score and my financial situation.” Historically,
0:14:36 most products have been designed with a complete focus on the functional and now we’re seeing
0:14:40 the next generation of not just fintech but consumer products think more about cognitive
0:14:41 and emotional.
0:14:46 There’s also more offline examples than we’re all typically aware of. So, one I learned
0:14:52 about over the last few years is called RASCAS, which is Rotating Savings and Credit Associations,
0:14:56 which are these offline communities that are managed in mostly immigrant communities managed
0:15:01 by an individual where everyone contributes, let’s say, $1,000 a month. And then each month,
0:15:06 if there are 10 members, one member receives $10,000. And typically, these are folks in
0:15:11 your community. You might meet them at church and it’s really hard to save $10,000. It’s
0:15:15 a lot easier to contribute $1,000 a month. And then when you receive the lump sum, there’s
0:15:20 always some big thing you want to do with the $10,000. There’s tons of examples of these
0:15:25 micro communities has not yet successfully been brought online. So, not everything is
0:15:29 sort of starting from zero when it comes to digital products.
0:15:33 And those ones are interesting because there’s a different iteration of those in every single
0:15:37 culture and every single country. And it is this robust offline behavior. And the question
0:15:42 is how do you bring it online and how do you bring it online in a way that is culturally
0:15:47 specific enough that it reflects the norms of that culture but also in a way that’s scalable?
0:15:52 So there’s the example of Rosca’s in a lot of communities all over the world. And then
0:15:57 I think if you look at the flip of that, what’s the extreme San Francisco version, it’s a
0:16:01 lot of people here do things like invest in restaurants. Why would you ever invest in
0:16:05 a restaurant? You’re probably not going to get your money back. There’s no liquidity.
0:16:09 At best, it’s sort of like cool to tell your friends maybe that you’re an investor there.
0:16:12 Maybe you skip a sort of reservation. It goes to emotional versus transactional.
0:16:13 Totally.
0:16:15 It’s not a transactional piece. It’s the emotion of the finance.
0:16:21 Exactly. But the proof point of actually investing in something versus just frequenting something
0:16:26 is very different. People want to participate. They want to express these preferences and
0:16:27 money is the strongest way to do so.
0:16:31 Well, and another example of something that’s inherently social. You’re investing in something
0:16:34 that then has a built-in social network in the investment.
0:16:38 There’s also this amazing trend around fractional ownership. So there’s a category of companies
0:16:44 that includes Rally Road and Otis and Mythic. They will take some asset, be it a classic
0:16:51 car, be it some culturally significant item, be it a magic card, be it a case of wine.
0:16:53 There’s a different version of all of this. And they will take that asset and they’ll
0:16:59 functionally securitize it. And then you, as a user, can purchase shares of that asset.
0:17:03 And then in some cases, depending on the kind of investment that you make, you get certain
0:17:07 levels of access or swag or other things that are associated with ownership.
0:17:13 So on the one hand, you actually have a piece of equity, a share in something that is theoretically
0:17:17 valuable because it’s actually a hard asset that has value. On the other side, you have
0:17:23 the status of owner within this piece that is of value in a more emotional sense, which
0:17:28 you’re investing in cultural pieces, which may or may not be a good financial investment
0:17:32 but from emotional cognitive side can be really, really rewarding. So I think that’s another
0:17:35 version where this idea of social plus fintech is taking off.
0:17:38 I love this example. You know, we’ve talked about this internally as perhaps a future
0:17:43 of museums and I think that vision is really interesting and it’s a much more emotional
0:17:44 than rational.
0:17:48 What’s the potential there? Are there areas where you see opportunity in some of these
0:17:50 niche groups?
0:17:54 I think social and finance is like the Holy Grail, right? The social version of most products
0:17:59 is the best version of most products. Engagement is higher, retention is higher, customer acquisition
0:18:03 costs go down. All these things that most consumer fintech companies struggle with are
0:18:07 solved by building the social product. It’s the extent that you can get something that
0:18:12 threads that needle between social and fintech. It’s amazing. It’s magical. It’s this incredible
0:18:16 thing that happens when it actually happens. It’s really hard to do, but when it does happen,
0:18:17 it’s phenomenal.
0:18:21 I think the biggest opportunity comes from finding the emergent behavior within niche
0:18:27 groups at the social level, at the community level, and then figuring out how fintech or
0:18:32 transactional layers into or on top of that. The saying is every company is eventually going
0:18:36 to become a fintech company, and I think that is probably the direction in which it goes
0:18:41 and which you have weird social behavior that has some ability to layer transaction inside
0:18:44 of it, and then that’s how social plus money takes off.
0:18:49 In my mind, the most direct way to start seeing this play out is just having more fintech products
0:18:54 address emotional needs as well as functional and cognitive needs. There’s some fintech products
0:19:00 like Joy, an app where you rate every transaction on how it made you feel. The goal of the game,
0:19:03 of course, is to only spend money on things that make you feel good, which is interesting.
0:19:07 I think that’s a product that’s completely designed around a set of emotional needs with
0:19:12 perhaps a set of functional outcomes as a happy side effect. I think there’s probably
0:19:15 a middle ground where a lot of products that are focused on helping you buy your first
0:19:21 home or reduce your debt or invest in stocks can actually start to design for these emotional
0:19:25 needs when it comes to money. That’s how we actually start to see this achieve scale.
0:19:29 Are there companies right now that you see making strides in that direction?
0:19:33 I think an example of a company that’s really gotten this right is Credit Karma, and granted
0:19:38 I was at Credit Karma. But if you look at the tone of the emails, if you look at the
0:19:44 ads that are on TV, if you look at the way the product is positioned, it plays as much
0:19:50 to curiosity and taking some of the heaviness out of credit. I think that’s been a really
0:19:55 successful strategy for them. I think this is a company that’s gotten it right when it
0:19:59 comes to how you talk to your customer about these otherwise really heavy things.
0:20:03 As people share more, it becomes less intimidating.
0:20:07 Or if you can see yourself relative to other people, that’s the other way that Credit Karma
0:20:11 works. It’s like, I know where I stand relative to other people, and maybe it makes me stressed
0:20:14 or maybe it makes me feel more comfortable, but at least there’s some level of transparency.
0:20:19 Right. There’s some freedom in that transparency that perhaps is driving customer acquisition.
0:20:24 That’s right. In terms of the products that have not worked, I think the product category
0:20:29 that hasn’t really seen success is personal financial management tools. There’s two reasons.
0:20:34 The first is that there’s a very small number of people who are super excited about budgeting
0:20:38 and trying every budgeting app, which is why when a lot of these products launch, they
0:20:42 get great growth in their first 18 to 24 months. You can get a couple of million users who
0:20:47 are really engaged. That’s not actually representative of the wider market where most people hate
0:20:51 budgeting. It’s not just because it’s a pain to keep a budget. It’s because it’s mostly
0:20:57 bad news. I look at a lot of these PFM and budgeting apps like calorie counting apps.
0:21:01 Just mostly makes you feel bad, and it’s easier to uninstall the app than it is to actually
0:21:05 stick with the budget or the diet. I think that’s a great example of a product category
0:21:10 that despite the fact that there’s real functional value there, it hasn’t taken off because it
0:21:14 didn’t address the emotional challenge that the consumer is facing.
0:21:19 I think another category that has not worked super well is ones that are designed to be
0:21:24 social but only transactional. I think there’s been this long history of people trying to
0:21:29 get people to be more public about what their portfolio is, and then other people can invest
0:21:35 based off of that portfolio, and it benefits the portfolio manager who’s sharing it. That’s
0:21:41 one where it is almost purely transactional relationship with purely financial incentives.
0:21:44 I think there’s been a lot of attempts at that. As far as I’m aware, none of them have
0:21:48 really taken off, but I think that’s another category where when you just stick within one
0:21:53 kind of bucket just within the transactional side, it’s really hard to layer social into
0:21:54 that.
0:21:58 So if we agree that social meets Vintec is really hard to do, but I’ve also heard you
0:22:03 both say it is the holy grail. Why is that? What makes it so powerful if we can get there?
0:22:07 I think first of all, if you just look at it, the most narrow lens is just from a core
0:22:13 business perspective. Stickiness, cross-out, acquisition, all of these things that are
0:22:18 super hard problems for most Vintec companies become dramatically easier if there’s a strong
0:22:24 social layer. So that’s the most narrow lens. And then I think the broadest lens is just
0:22:30 ending this dynamic where we’re alone together. Everyone’s in a dark room feeling bad about
0:22:34 their money with everyone else in that same dark room, and I think if you can turn the
0:22:40 light on, then all of a sudden, it is an opportunity to uplift everyone a little bit and normalize
0:22:44 the situation that folks are in. I think that with, especially we talked about both the
0:22:50 good side of Insta, but Insta is also a very public place to talk about your spending,
0:22:55 and I think that that drives a perverse set of expectations around what’s normal, and
0:22:56 we should try to change that.
0:23:00 Yes. There’s multiple levels to why social plus money is this holy grail. Another lens
0:23:07 is it broadens the solution space within which, as a founder, you can operate because now you’re
0:23:12 not just on the transactional level or you’re not just on the emotional and cognitive level,
0:23:17 you’re now across all three. If you actually have financial or social plus Vintec or whatever
0:23:23 it is, so you can now design things that have some combination of those three levers, which,
0:23:26 if you’re competing against a purely transactional thing or you’re competing against a purely
0:23:31 emotional thing, you now just have more factors that you can operate across. The flip side
0:23:37 to that is it’s combinatorially more complicated to do, but if you do do it, you’re in a class
0:23:38 of your own.
0:23:40 Thank you for joining us on the A16Z podcast.
0:23:41 Thanks, Lauren.
0:23:42 Thanks, Darcy.
0:23:43 Thanks, Anish.
0:23:43 Cheers.
0:23:53 [BLANK_AUDIO]

0:00:05 Hi everyone, welcome to the A6NZ podcast, I’m Sonal.
0:00:08 So one of the recurring themes we talk a lot about on this podcast is how software changes
0:00:11 organizations and vice versa.
0:00:16 More broadly, it’s really about how companies of all kinds innovate with the org structures
0:00:18 and tools that they have.
0:00:22 And today’s episode, a rerun of a very popular episode from a couple years ago, draws on
0:00:28 actual research and data from one of the largest large-scale studies of software and organizational
0:00:30 performance out there.
0:00:34 Joining me in this conversation are two of the authors of the book Accelerate, the Science
0:00:39 of Lean Software and DevOps by Nicole Forsgren, Jez Humble, and Jean Kim.
0:00:43 We have the first two authors, so Nicole, who did her PhD research trying to answer
0:00:48 the lucid, eternal questions around how to measure software performance in orgs, especially
0:00:52 given past debates, around does IT matter?
0:00:57 She was the co-founder and CEO of Dora, which put out the annual State of DevOps report.
0:01:02 Dora was acquired by Google Cloud a little over a year ago, and she will soon be joining
0:01:05 GitHub as VP of Research and Strategy.
0:01:10 And then we also have Jez Humble, who was CTO at Dora, is currently in developer relations
0:01:17 at Google Cloud and is also the co-author of the books The DevOps Handbook, Lean Enterprise,
0:01:18 and Continuous Delivery.
0:01:23 In the conversation that follows, Nicole and Jez share their findings about high-performing
0:01:27 companies, even those that may not think they’re tech companies, and answer my questions about
0:01:32 whether there’s an ideal org type for this kind of innovation, whether it’s the size
0:01:36 of the organization, the software architecture they use, their culture or people, and where
0:01:39 the role of software and IT lives within that.
0:01:44 But first, we begin by talking briefly about the history of DevOps and where that fits
0:01:48 in the broader landscape of related software movements.
0:01:50 So I started as a software engineer at IBM.
0:01:54 I did hardware and software performance, and then I took a bit of a detour into academia
0:01:59 because I wanted to understand how to really measure and look at performance that would
0:02:04 be generalizable to several teams in predictable ways and in predictive ways.
0:02:10 And so I was looking at and investigating how to develop and deliver software in ways
0:02:15 that were impactful to individuals, teams, and organizations.
0:02:20 And then I pivoted back into industry because I realized this movement had gained so much
0:02:26 momentum and so much traction, and industry was desperate to really understand what types
0:02:30 of things are really driving performance outcomes and excellence.
0:02:32 And what do you mean by this movement?
0:02:38 This movement that now we call DevOps, so the ability to leverage software to deliver
0:02:44 value to customers, to organizations, to stakeholders.
0:02:47 And I think from a historical point of view, the best way to think about DevOps, it’s
0:02:53 a bunch of people who had to solve this problem of how do we build large distributed systems
0:02:59 that were secure and scalable and be able to change them really rapidly and evolve them.
0:03:02 And no one had had that problem before, certainly at the scale of companies like Amazon and
0:03:03 Google.
0:03:07 And that really is where the DevOps movement came from, trying to solve that problem.
0:03:11 And you can make an analogy to what Agile was about since the kind of software crisis
0:03:17 of the 1960s and people trying to build these defense systems at large scale, the invention
0:03:24 of software engineering as a field, Margaret Hamilton, her work at MIT on the Apollo program.
0:03:28 What happened in the decades after that was everything became kind of encased in concrete
0:03:33 in these very complex processes, this is how you develop software.
0:03:36 And Agile was kind of a reaction to that, saying we can develop software much more
0:03:40 quickly with much smaller teams in a much more lightweight way.
0:03:43 So we didn’t call it DevOps back then, but it’s also more Agile.
0:03:45 Can you guys break down the taxonomy for a moment?
0:03:48 Because when I think of DevOps, I think of it in the context of the containerization
0:03:51 of code and virtualization.
0:03:56 I think of it in the context of microservices and being able to do modular teams around
0:03:57 different things.
0:03:59 There’s an organizational element, there’s a software element, there’s an infrastructure
0:04:03 component, like paint the big picture for me of those building blocks and how they all
0:04:04 kind of fit together.
0:04:07 Well, I can give you a very personal story, which was my first show after college was
0:04:12 in 2000 in London, working at a startup where I was one of two technical people in the startup.
0:04:17 And I would deploy to production by FTP and code from my laptop directly into production.
0:04:21 And if I wanted to roll back, I’d say, “Hey, Johnny, can you FTP your copy of this file
0:04:22 to production?”
0:04:23 And that was our rollback process.
0:04:27 And then I went to work in consultancy where we were on these huge teams and deploying
0:04:30 to production, there was a whole team with a Gantt chart which put together the plan
0:04:31 to deploy to production.
0:04:32 And I’m like, this is crazy.
0:04:36 Unfortunately, I was working with a bunch of other people who also thought it was crazy.
0:04:40 And then we came up with these ideas around deployment automation and scripting and stuff
0:04:41 like that.
0:04:44 And suddenly we saw the same ideas had popped up everywhere, basically.
0:04:48 I mean, it’s realising that if you’re working in a large complex organisation, Agile’s going
0:04:54 to hit a brick wall because unlike the things we were building in the ’60s, product development
0:04:56 means that things are changing and evolving all the time.
0:04:58 So it’s not good enough to get to production the first time.
0:05:00 You’ve got to be able to keep getting there on and on.
0:05:01 And that really is where DevOps comes in.
0:05:05 It’s like, well, Agile, we’ve got a way to build and evolve products, but how do we keep
0:05:10 deploying to production and running the systems in production in a stable, reliable way, particularly
0:05:12 in the distributed context?
0:05:16 So if I phrase it another way, sometimes there’s a joke that says day one is short and day
0:05:17 two is long.
0:05:18 What does that mean?
0:05:19 Right.
0:05:20 So day one is when we create all these–
0:05:21 That’s by the way sad that you have to explain the joke to me.
0:05:22 No, it’s–
0:05:26 No, which is great, though, because so day one is when we create all of these systems.
0:05:28 And day two is when we deploy to production.
0:05:33 We have to deploy and maintain forever and ever and ever and ever.
0:05:35 So day two is an infinite day.
0:05:36 Right, exactly.
0:05:37 Yeah.
0:05:38 First successful product.
0:05:39 Hopefully.
0:05:41 We hope that day two is really, really long.
0:05:45 And we’re fond of saying Agile doesn’t scale.
0:05:48 And sometimes I’ll say this, and people shoot laser beams out of their eyes.
0:05:50 But when we think about it, Agile was meant for development.
0:05:53 Just like Jez said, it speeds up development.
0:05:58 But then you have to hand it over and especially infrastructure and IT operations.
0:05:59 What happens when we get there?
0:06:02 So DevOps was sort of born out of this movement.
0:06:06 And it was originally called Agile System Administration.
0:06:10 And so then DevOps sort of came out of development and operations.
0:06:14 And it’s not just DevOps, but if we think about it, that’s sort of the bookends of
0:06:15 this entire process.
0:06:17 Well, it’s actually like day one and day two combined into one phrase.
0:06:19 Day one and day two.
0:06:23 The way I think about this is I remember the stories of Microsoft in the early days and
0:06:29 the waterfall cascading model of development, Leslie Lamport once wrote a piece for me about
0:06:33 why software should be developed like houses because you need a blueprint.
0:06:37 And I’m not a software developer, but it felt like a very kind of old way of looking at
0:06:38 the world of code.
0:06:40 I hate that metaphor.
0:06:41 Tell me why.
0:06:44 If the thing you’re building has well understood characteristics, it makes sense.
0:06:47 So if you’re building a trust bridge, for example, there’s well-known understood models
0:06:51 of building trust bridges, you plug the parameters into the model and then you get a trust bridge
0:06:52 and it stays up.
0:06:55 Have you been to Sagrada Familia in Barcelona?
0:06:56 Oh, I love Gaudi.
0:06:57 Okay.
0:07:00 So if you go into the crypt of the Sagrada Familia, you’ll see his workshop and there’s
0:07:05 a picture, in fact, a model that he built of the Sagrada Familia, but upside down with
0:07:07 the weight simulating the stresses.
0:07:10 And so he would build all these prototypes and small prototypes because he was fundamentally
0:07:12 designing a new way of building.
0:07:17 All Gaudi’s designs were hyperbolic curves and parabolic curves and no one had used that
0:07:18 before.
0:07:19 Things that had never been pressure tested.
0:07:20 Right.
0:07:21 Literally.
0:07:22 In that case.
0:07:23 Exactly.
0:07:24 He didn’t want them to fall down.
0:07:25 So he built all these prototypes and did all this stuff.
0:07:29 He built his blueprint as he went by building and trying it out, which is a very rapid prototyping
0:07:30 kind of model.
0:07:31 Absolutely.
0:07:34 So in the situation where the thing you’re building has known characteristics and it’s
0:07:38 been done before, yeah, sure, we can take a very phased approach to it.
0:07:42 And, you know, for designing these kind of protocols that have to work in a distributed
0:07:46 context and you can actually do formal proofs of them, again, that makes sense.
0:07:51 But when we’re building products and services where particularly we don’t know what customers
0:07:55 actually want and what users actually want, it doesn’t make sense to do that because you’ll
0:07:57 build something that no one wants.
0:07:58 You can’t predict.
0:08:00 And we’re particularly bad at that, by the way.
0:08:05 Even companies like Microsoft, where they are very good at understanding what their
0:08:09 customer base looks like, they have a very mature product line.
0:08:15 Ronnie Cahave has done studies there and only about one-third of the well-designed features
0:08:16 deliver value.
0:08:18 That’s actually a really important point.
0:08:22 The mere question of does this work is something that people really clearly don’t pause to
0:08:26 ask, but I do have a question for you guys to push back, which is, is this a little bit
0:08:27 of the cult?
0:08:31 Oh, my God, it’s like so developer-centric, let’s be agile, let’s do it fast, our way,
0:08:35 you know, two pizzas, that’s the ideal size of a software team and, you know, I’m not
0:08:36 trying to mock it.
0:08:41 I’m just saying that isn’t there an element of actual practical realities like technical
0:08:46 debt and accruing a mess underneath all your code and a system that you may be there for
0:08:49 two or three years and you can go after the next startup, but okay, someone else has to
0:08:51 clean up your mess.
0:08:53 Tell me about how this fits into that big picture.
0:08:55 This is what enables all of that.
0:08:56 Oh, right.
0:08:57 Interesting.
0:08:59 So it’s not actually just creating a problem because that’s how I’m kind of hearing it.
0:09:00 No, absolutely.
0:09:05 So you still need development, you still need test, you still need QA, you still need operations,
0:09:08 you still need to deal with technical debt, you still need to deal with re-architecting
0:09:12 really difficult large monolithic code bases.
0:09:17 What this enables you to do is to find the problems, address them quickly, move forward.
0:09:22 I think that the problem that a lot of people have is that we’re so used to couching these
0:09:26 things as trade-offs and as dichotomies, the idea that if you’re going to move fast, you’re
0:09:27 going to break things.
0:09:32 The one thing which I always say is, if you take one thing away from DevOps is this, high-performing
0:09:34 companies don’t make those trade-offs.
0:09:36 They’re not going fast and breaking things.
0:09:40 They’re going fast and making more stable, more high-quality systems, and this is one
0:09:44 of the key results in the book, in our research, is this fact that high-performers do better
0:09:49 at everything because the capabilities that enable high-performance in one field, if done
0:09:51 right, enable it in other fields.
0:09:55 If you’re using version control for software, you should also be using version control for
0:09:56 your production infrastructure.
0:10:00 If there’s a problem in production, we can reproduce the state of the production environment
0:10:05 in a disaster recovery scenario, again in a predictable way that’s repeatable.
0:10:07 I think it’s important to point out that this is something that happened in manufacturing
0:10:08 as well.
0:10:09 Give it to me.
0:10:13 I love when people talk about software as drawn from hardware analogies as my favorite
0:10:14 type of metaphor.
0:10:20 Okay, so Toyota didn’t win by making shitty cars faster, they won by making higher-quality
0:10:22 cars faster and having shorter time to market.
0:10:25 The lean manufacturing method, which by the way also spawned lean startup thinking and
0:10:26 everything else connected to it.
0:10:30 And DevOps pulls very strongly from lean methodologies.
0:10:34 So you guys are probably the only people to have actually done a large-scale study of
0:10:36 organizations adopting DevOps.
0:10:38 What is your research and what did you find?
0:10:39 Sure.
0:10:44 My research really is the largest investigation of DevOps practices around the world.
0:10:48 We have over 23,000 data points, all industries.
0:10:49 Give me like a sampling, like what are the range of industries?
0:10:56 So I’ve got entertainment, I’ve got finance, I have healthcare and pharma, I have technology.
0:10:57 Government.
0:10:58 Government, education.
0:11:00 You basically have every vertical.
0:11:01 And then when you tell you around the world.
0:11:07 So we’re primarily in North America, we’re in Amia, we have India, we have a small sample
0:11:08 in Africa.
0:11:09 Right.
0:11:13 And we break down like the survey methodology questions that people have in the ethnographic
0:11:17 world, the way we would approach it is that you can never trust what people say they do.
0:11:19 You have to watch what they do.
0:11:23 However, it is absolutely true, and especially in a more scalable sense, that there are really
0:11:25 smart surveys that give you a shit ton of useful data.
0:11:26 Yes.
0:11:30 And part two of the book covers this in almost excruciating detail.
0:11:31 We like knowing methodologies.
0:11:32 Yes.
0:11:33 So it’s nice to share that.
0:11:37 Well, and it’s interesting because Jez talked about in his overview of Agile and how it changes
0:11:41 so quickly and we don’t have a really good definition, but that does is it makes it difficult
0:11:42 to measure.
0:11:43 Right.
0:11:49 And so what we do is we’ve defined core constructs, core capabilities, so that we can then measure
0:11:50 them.
0:11:57 We go back to core ideas around things like automation, process, measurement, lean principles.
0:12:02 And then I’ll get that pilot set of data and I’ll run preliminary statistics to test for
0:12:06 discriminant validity, convergent validity, composite reliability.
0:12:09 Make sure that it’s not testing what it’s not supposed to test.
0:12:12 It is testing what it is supposed to test.
0:12:15 Everyone is reading it consistently the same way that I think it’s testing.
0:12:20 I even run checks to make sure that I’m not inadvertently inserting bias or collecting
0:12:23 bias just because I’m getting all of my data from surveys.
0:12:25 Sounds pretty damn robust.
0:12:28 So tell me then what were the big findings?
0:12:30 That’s a huge question, but give me the hit list.
0:12:31 Well, okay.
0:12:35 So let’s start with one thing that Jess already talked about, speed and stability go together.
0:12:39 This is where he was talking about there not being necessarily a false dichotomy and that’s
0:12:41 one of your findings that you can actually accomplish both.
0:12:42 Yeah.
0:12:43 And it’s worth talking about how we measure those things as well.
0:12:48 So we measure speed or tempo as we call it in the book or sometimes people call it throughput
0:12:49 as well.
0:12:53 Which is a nice full circle manufacturing idea, like the semiconductor circuit throughput.
0:12:54 Yeah, absolutely.
0:12:56 I love hardware analogies for software, I told you.
0:12:57 A lot of it comes from lean.
0:13:01 So lead time, obviously one of the classic lean manufacturing measures we use.
0:13:02 How long does it take?
0:13:06 You look at the lead time from checking into version control to release into production.
0:13:09 So that part of the value stream because that’s more focused on the DevOps end of things.
0:13:11 And it’s highly predictable.
0:13:12 The other one is release frequency.
0:13:13 So how often do you do it?
0:13:17 And then we’ve got two stability metrics and one of them is time to restore.
0:13:21 So in the event that you have some kind of outage or some degradation in performance in
0:13:24 production, how long does it take you to restore service?
0:13:27 For a long time we focused on not letting things break.
0:13:30 And I think one of the changes, paradigm shifts we’ve seen in the industry, particularly
0:13:32 in DevOps, is moving away from that.
0:13:36 We accept that failure is inevitable because we’re building complex systems.
0:13:40 So not how do we prevent failure, but when failure inevitably occurs, how quickly can
0:13:41 we detect and fix it?
0:13:42 MTBF, right?
0:13:43 Mean time between failures.
0:13:48 If you only go down once a year, but you’re down for three days and it’s on Black Friday.
0:13:52 But if you’re down very small, low blast, very, very small blast radius and you can come
0:13:57 back almost immediately and your customers almost don’t notice.
0:13:58 That’s fine.
0:14:00 The other piece around stability is change fail, right?
0:14:03 When you push a change into production, what percentage of the time do you have to fix
0:14:04 it?
0:14:05 Because something went wrong.
0:14:07 By the way, what does that tell you if you have a change fail?
0:14:10 So in the lean kind of discipline, this is called percent complete and accurate.
0:14:12 And it’s a measure of a quality of your process.
0:14:17 So in a high quality process, when I do something for Nicole, Nicole can use it rather than
0:14:21 sending it back to me and say, “Hey, there’s a problem with this.”
0:14:24 And in this particular case, what percentage of the time when I deploy something to production
0:14:27 is there a problem because I didn’t test it adequately.
0:14:29 My testing environment wasn’t production like enough.
0:14:31 Those are the measures for finding this.
0:14:36 But the big finding is that you can have speed and stability together through DevOps.
0:14:38 Is that what I’m hearing?
0:14:39 Yes, yes.
0:14:40 High performers get it all.
0:14:42 Low performers kind of suck at all of it.
0:14:43 Medium performers hang out in the middle.
0:14:46 I’m not seeing trade-offs four years in a row.
0:14:50 So anyone who’s thinking, “Oh, I can be more stable if I slow down,” I don’t see it.
0:14:54 It actually breaks a very commonly held kind of urban legend around how people believe
0:14:55 these things operate.
0:14:58 So tell me, are there any other sort of findings like that?
0:14:59 Because that’s very counterintuitive.
0:15:01 Okay, so this one’s kind of fun.
0:15:07 One is that this ability to develop and deliver software with speed and stability drives organizational
0:15:08 performance.
0:15:09 Now, here’s the thing.
0:15:11 I was about to say, that’s a very obvious thing to say.
0:15:13 So it seems obvious, right?
0:15:17 Developing and delivering software with speed and stability drives things like profitability,
0:15:19 productivity, market share.
0:15:26 Okay, except if we go back to Harvard Business Review 2003, there’s a paper titled, “IT Doesn’t
0:15:27 Matter.”
0:15:32 We have decades of research, I want to say at least 30 or 40 years of research showing
0:15:37 the technology does not drive organizational performance.
0:15:38 It doesn’t drive ROI.
0:15:43 And we are now starting to find other studies and other research that backs this up.
0:15:48 Eric Brinniol sent out of MIT, James Best sent out of Boston University, 2017.
0:15:50 Did you say James Bessen?
0:15:51 Yeah.
0:15:52 Oh, I used to edit him, too.
0:15:54 Yeah, it’s fantastic.
0:15:56 Here’s why it’s different.
0:16:01 Because before, right in like the 80s and the 90s, we did this thing where like, you’d
0:16:03 buy the tech and you’d plug it in and you’d walk away.
0:16:07 It was on-prem sales model where you like deliver and leave as opposed to like software
0:16:09 as a service and the other ways that things happen.
0:16:11 And people would complain if you tried to upgrade it too often.
0:16:12 Oh, right.
0:16:17 The key is that everyone else can also buy the thing and plug it in and walk away.
0:16:22 How is that driving value or differentiation for a company?
0:16:27 If I just buy a laptop to help me do something faster, everyone else can buy a laptop to do
0:16:29 the same thing faster.
0:16:34 That doesn’t help me deliver value to my customers or to the market.
0:16:36 It’s a point of parity, not a point of distinction.
0:16:37 Right.
0:16:40 And you’re saying that point of distinction comes from how you tie together that technology
0:16:43 process and culture through DevOps.
0:16:44 Right.
0:16:46 And that it can provide a competitive advantage to your business.
0:16:50 If you’re buying something that everyone else also has access to, then it’s no longer a
0:16:51 differentiator.
0:16:54 But if you have an in-house capability and those people are finding ways to drive your
0:16:57 business, I mean, this is the classic Amazon model.
0:17:01 They’re running hundreds of experiments in production at any one time to improve the
0:17:02 product.
0:17:05 And that’s not something that anyone else can copy, that’s why Amazon keeps winning.
0:17:08 So what people are doing is copying the capability instead.
0:17:09 And that’s what we’re talking about.
0:17:10 How do you build that capability?
0:17:14 The most fascinating thing to me about all this is honestly not the technology per se,
0:17:17 but the organizational change part of it and the organizations themselves.
0:17:22 So of all the people you studied, is there an ideal organizational makeup that is ideal
0:17:23 for DevOps?
0:17:27 Or is it one of these magical formulas that has this ability to turn a big company into
0:17:31 a startup and a small company into, because that’s actually the real question.
0:17:34 From what I’ve seen, there might be two ideals.
0:17:39 The nice, happy answer is the ideal organization is the one that wants to change.
0:17:44 That’s, I mean, given this huge n equals 23,000 dataset, is it not tied to a particular profile
0:17:45 of a size of company?
0:17:47 They’re both shaking their head just for the listeners.
0:17:51 I see high performers among large companies.
0:17:52 I see high performers in small companies.
0:17:55 I see low performers in small companies.
0:17:57 I see low performers in highly regulated companies.
0:18:00 I see low performers in not regulated companies.
0:18:03 So tell me the answer you’re not supposed to say.
0:18:11 So that answer is it tends to be companies that are like, oh shit, and they’re two profiles.
0:18:16 Number one, they’re like way behind, and oh shit, and they have some kind of funds.
0:18:25 Or they are like this lovely, wonderful bastion of like they’re these really innovative, high-performing
0:18:29 companies, but they still realize they’re a handful of like two or three companies ahead
0:18:31 of them, and they don’t want to be number two.
0:18:32 They are going to be number one.
0:18:33 So those are sort of the ideal.
0:18:35 I mean, just like anthropomorphize it a little bit.
0:18:41 It’s like the 35 to 40 year old who suddenly discovers you might be pre-diabetic, so you
0:18:43 better do something about it now before it’s too late.
0:18:47 But it’s not too late because you’re not so old where you’re about to reach sort of
0:18:50 the end of a possibility to change that runway.
0:18:54 And then there’s this person who’s sort of kind of already like in the game running in
0:18:57 the race and they might be two or three, but they want to be like number one.
0:19:02 And I think to extend your metaphor, the companies that do well are the companies that never got
0:19:05 diabetic in the first place because they always just ate healthily.
0:19:07 They were already glucose monitoring.
0:19:10 They had continuous glucose monitors on, which is like DevOps actually.
0:19:11 They were always athletes.
0:19:12 Right.
0:19:15 You know, diets are terrible because at some point you have to stop the diet.
0:19:18 And it has to start and start and stop as opposed to a way of life is what you’re saying.
0:19:19 Right, exactly.
0:19:24 So if you just always eat healthily and never eat too much or very rarely eat too much and
0:19:27 do a bit of exercise every day, you never get to the stage like, oh my God, now I can
0:19:29 only eat tofu.
0:19:39 So like my loving professerness, nurture Nicole also has one more profile that like I love
0:19:42 and I worry about them like mother hen.
0:19:47 And it’s the companies that I talk to and they come to me and they’re struggling and
0:19:52 I haven’t decided if they want to change, but they’re like, so we need to do this transformation
0:19:53 and we’re going to do the transformation.
0:19:57 And it’s either because they want to or when they’ve been told that they need to.
0:20:01 And then they will insert this thing where they say, but I’m not a technology company.
0:20:08 I’m like, but we just had this 20 minute conversation about how you’re leveraging technology to drive
0:20:13 value to customers or to drive this massive process that you do.
0:20:15 And then they say, but I’m not a technology company.
0:20:19 I could almost see why they had that in their head because they were a natural resources
0:20:20 company.
0:20:23 But there was another one where they were a finance company.
0:20:27 I mean, an extension of software eats the world is really every company is a technology
0:20:28 company.
0:20:32 It’s fascinating to me that that third type exists, but it is a sign of this legacy world
0:20:38 moving into and I worry about them also, at least for me personally, you know, I lived
0:20:42 through this like mass extinction of several firms and I don’t want it to happen again.
0:20:46 And I worry about so many companies that keep insisting they’re not technology companies.
0:20:49 And I’m like, oh, honey child, you’re a tech company.
0:20:51 You know, one of the gaps in our data is actually China.
0:20:55 And I think big China is a really interesting example because they didn’t go through the
0:20:58 whole, you know, IT doesn’t matter phase.
0:21:02 They’re jumping straight from no technology to Alibaba and Tencent, right?
0:21:07 I think US companies should be scared because the moment Tencent and Alibaba already made
0:21:12 moving into other developing markets and they’re going to be incredibly competitive because
0:21:13 it’s just built into their DNA.
0:21:16 So the other fascinating thing to me is that you essentially were able to measure performance
0:21:20 of software and clearly productivity.
0:21:22 Is there any more insights on the productivity side?
0:21:23 Yes.
0:21:24 Yes.
0:21:25 I want to go.
0:21:26 This is his favorite ramp.
0:21:27 Jumping around and like waving his hand.
0:21:31 So tell us the reason the manufacturing metaphor breaks down is because in manufacturing you
0:21:32 have inventory.
0:21:33 Yes.
0:21:36 We do not have inventory in the same way in software.
0:21:39 In a factory, like the first thing your lean consultant is going to do, walking into the
0:21:42 factory is point to the piles of thing everywhere.
0:21:47 But I think if you walk into an office where there’s developers, where’s the inventory?
0:21:50 By the way, that’s what makes talking about this to executives so difficult.
0:21:51 They can’t see the process.
0:21:56 Well, it’s a hard question to answer because is the inventory the code that’s being written?
0:22:00 And people actually have done that and said, “Well, listen, lines of code are an accounting
0:22:04 measure and we’re going to capture that as, you know, capital.”
0:22:05 That’s insane.
0:22:08 It’s like an invitation to write crappy, unnecessarily long code.
0:22:09 That’s exactly what happens.
0:22:11 It’s like the olden days are getting paid for a book by how long it is and it’s like
0:22:14 actually really boring when you can actually write it in like one third of the length.
0:22:15 Let’s write it in German.
0:22:16 Right, you know.
0:22:17 I’m thinking of Charles Dickens.
0:22:19 In general, you know, you prefer people to write short programs because they’re easier
0:22:21 to maintain and so forth.
0:22:23 But lines of code have all these drawbacks.
0:22:25 We can’t use them as a measure of productivity.
0:22:27 So if you can’t measure lines of code, what can you measure?
0:22:30 Because I really want an answer like, how do you measure productivity?
0:22:31 So velocity is the other classic example.
0:22:38 Agile, there’s this concept of velocity, which is the number of story points a team manages
0:22:41 to complete in an iteration.
0:22:47 So before the start of an iteration in many agile, particularly scrum-based processes,
0:22:48 you’ve got all this work to do.
0:22:50 You’re like, “We need to build these five features.
0:22:51 How long will this feature take?”
0:22:54 And the developers fight over it and they’re like, “Oh, it’s five points.”
0:22:57 And then this one’s going to take three points, this one’s going to take two points.
0:23:00 And so you have a list of all these features and you don’t get through all of them.
0:23:03 At the end of the iteration, the customer signs off, “Well, I’m accepting this one.
0:23:04 This one’s fine.
0:23:05 This one’s fine.
0:23:06 This one’s a hot mess.
0:23:07 Go back and do it again.”
0:23:08 Whatever.
0:23:09 The number of points you complete in the iteration is the velocity.
0:23:12 So it’s like the speed at which you’re able to deliver those features.
0:23:16 So a lot of people treat it like that, but actually, that’s not really what it’s about.
0:23:20 It’s a relative measure of effort and it’s for capacity planning purposes.
0:23:23 So basically, for the next iteration, we’ll only commit to completing the same velocity
0:23:24 that we finished last time.
0:23:27 So it’s relative and it’s team dependent.
0:23:30 And so what a lot of people do is say they start comparing velocities across teams.
0:23:34 Then what happens is, a lot of work, you need to collaborate between teams.
0:23:38 But hey, if I’m going to help you with your story, that means I’m not going to get my
0:23:42 story points and you’re going to get your story points, right, people can game it as
0:23:43 well.
0:23:45 You should never use story points as a productivity measure.
0:23:48 So lines of code doesn’t work, velocity doesn’t work, what works?
0:23:53 So this is why we like, two things in particular, one thing that it’s a global measure.
0:23:57 And secondly, that it’s not just one thing, it mixes two things together, which might
0:23:59 normally be intention.
0:24:03 And so this is why we went for our measure of performance.
0:24:11 So measuring lead time, release frequency, and then time to restore and change fail rate.
0:24:15 Lead time is really interesting because lead time is on the way to production, right?
0:24:17 So all the teams have to collaborate.
0:24:21 It’s not something where I can go really fast in my velocity, but nothing ever gets delivered
0:24:22 to the customer.
0:24:23 It doesn’t count in lead time.
0:24:24 So it’s a global measure.
0:24:27 It takes care of that problem of the incentive alignment around the competitive dynamic.
0:24:30 Also, it’s an outcome.
0:24:31 It’s not an output.
0:24:33 There’s a guy called Jeff Patton.
0:24:36 He’s a really smart thinker in the kind of lead and agile space.
0:24:43 He says, minimize output, maximize outcomes, which I think is simple but brilliant.
0:24:45 It’s so simple because it just shifts the words to impact.
0:24:49 And even we don’t get all the way there because we’re not yet measuring, did the features
0:24:51 deliver the expected value to the organization or the customers?
0:24:58 Well, we do get there because we focus on speed and stability, which then deliver the
0:25:02 outcome to the organization, profitability, productivity, market share.
0:25:07 But the second half of this, which I am also hearing is, did it meet your expectations?
0:25:11 Did it perform to the level that you wanted it to?
0:25:13 Did it match what you asked for?
0:25:18 Or even if it wasn’t something you specified that you desired or needed, that seems like
0:25:19 a slightly open question.
0:25:20 So we did actually measure that.
0:25:24 We looked at non-profit organizations and these were exactly the questions we measured.
0:25:29 We asked people, did the software meet, I can’t remember what the exact questions were.
0:25:33 Effectiveness, efficiency, customer satisfaction, delivery, mission goals.
0:25:35 How fascinating that you do it non-profits because that is a larger move in the non-profit
0:25:38 measurement space to try to measure impact.
0:25:43 But we captured it everywhere because even profit seeking firms still have these goals.
0:25:47 In fact, as we know from research, companies that don’t have a mission other than making
0:25:49 money do less well than the ones that do.
0:25:54 I think, again, what the data shows is that companies that do well on the performance measures
0:25:58 we talked about outperform their low performing peers by a factor of two.
0:26:02 A hypothesis is what we’re doing when we create these high performing organizations in terms
0:26:06 of speed and stability is we’re creating feedback loops.
0:26:11 What it allows us to do is build a thin slice, a prototype of a feature, get feedback through
0:26:16 some UX mechanism, whether that’s showing people the prototype and getting their feedback,
0:26:19 whether it’s running A/B tests or multivariate tests in production.
0:26:23 It’s what creates these feedback loops that allow you to shift direction very fast.
0:26:25 I mean, that is the heart of Lean Startup.
0:26:29 It’s the heart of anything you’re putting out into the world is you have to kind of
0:26:30 bring it full circle.
0:26:33 It is a secret of success to Amazon, as you cited earlier.
0:26:35 I would distill it to just that.
0:26:37 I think I heard Jeff Bezos say the best line.
0:26:40 It was at the Internet Association dinner in DC last year where he came and asked me
0:26:41 about an innovation.
0:26:44 He’s like, to him, an innovation is something that people actually use.
0:26:47 And that’s what I love about the feedback loop thing, is it actually reinforces that
0:26:49 mindset of that’s what innovation is.
0:26:50 Right.
0:26:54 So to sum up, the way you can frame this is DevOps is that technological capability
0:26:59 that underpins your ability to practice Lean Startup and all these very rapid iterative
0:27:00 processes.
0:27:02 So I have a couple of questions then.
0:27:07 So one is going back to this original taxonomy question, and you guys described that there
0:27:09 isn’t necessarily an ideal organizational type.
0:27:11 Which by the way, should be encouraging.
0:27:12 I agree.
0:27:17 It’s super encouraging and more importantly democratizing that anybody can become a hit
0:27:18 player.
0:27:19 We were doing this in the federal government.
0:27:20 I love that.
0:27:24 But one of my questions is, when we had Adrian Cockroft on this podcast a couple of years
0:27:27 ago talking about microservices, and the thing that I thought was so liberating about what
0:27:34 he was describing the Netflix story was that it was a way for teams to essentially become
0:27:40 little mini product management units and essentially self-organize because the infrastructure
0:27:48 by being broken down into these micro pieces versus say a monolithic kind of uniform architecture,
0:27:53 I would think that being a organization that’s containerized its code in that way that has
0:27:58 this microservices architecture would be more suited to DevOps.
0:28:00 Or is that a wrong belief?
0:28:04 I’m just trying to understand again that taxonomy thing of how these pieces all fit together.
0:28:07 So we actually studied this as a whole section of architecture in the book where we looked
0:28:09 at exactly this question.
0:28:12 Architecture has been studied for a long time and people talk about architectural characteristics.
0:28:16 There’s the ATAM, the architectural trade-off model that kind of email and developed.
0:28:21 There’s some additional things we have to care about, testability and deployability.
0:28:27 Can my team test its stuff without having to rely on this very complex integrated environment?
0:28:31 Can my team deploy its code to production without these very complex orchestrated deployments?
0:28:34 Basically, can we do things without dependencies?
0:28:38 That is one of the biggest predictors in our cohort of IT performance is the ability of
0:28:43 teams to get stuff done on their own without dependencies on other teams, whether that’s
0:28:46 testing or whether it’s deploying or whether it’s planning.
0:28:47 Even just communicating.
0:28:53 Can you get things done without having to do mass communication and checking in permissions?
0:28:57 Question I love, love, love asking on this podcast is we always revisit the 1937 Coast
0:29:02 paper about the theory of the firm and its idea that transaction costs are more efficient.
0:29:07 This is like the ultimate model for reducing friction and those transaction costs, communication,
0:29:08 coordination costs, all of it.
0:29:11 That’s what all the technical and process stuff is about that.
0:29:13 I mean, Don Robinson once came to one of my talks on continuous delivery.
0:29:18 At the end, he said, “So, continuous delivery, that’s just about reducing transaction costs,
0:29:19 right?”
0:29:20 And I’m like…
0:29:21 An economist view of DevOps.
0:29:22 I love it.
0:29:23 You’re right.
0:29:25 You’ve reduced my entire body of work to one sentence.
0:29:27 It’s so much Conway’s Law, right?
0:29:28 This would remind me what Conway’s Law is.
0:29:33 Organizations which design systems are constrained to produce designs which are copies of the
0:29:35 communication structures of these organizations.
0:29:36 Oh, right.
0:29:39 It’s that idea basically that your software code looks like the shape of the organization
0:29:40 itself.
0:29:41 Right.
0:29:42 And how we communicate, right?
0:29:46 So, which, you know, Jez just summarized, if you have to be communicating and coordinating
0:29:48 with all of these other different groups…
0:29:52 Command and control looks like waterfall, a more decentralized model looks like independent
0:29:53 teams.
0:29:54 Right.
0:29:55 So, the data shows that.
0:29:58 A lot of people jump on the microservices, containerization, bandwagon.
0:30:03 There’s one thing that is very important to bear in mind, implementing those technologies
0:30:05 does not give you those outcomes we talked about.
0:30:07 We actually looked at people doing mainframe stuff.
0:30:10 You can achieve these results with mainframes.
0:30:16 Equally, you can use the, you know, Kubernetes and, you know, Docker and microservices and
0:30:17 not achieve these outcomes.
0:30:22 We see no statistical correlation with performance, whether you’re on a mainframe or a greenfield
0:30:24 or a brownfield system.
0:30:28 If you’re building something brand new or if you’re working on existing build.
0:30:31 And one thing I wanted to bring up that we didn’t before is I said, you know, day one
0:30:32 is short, day two is long.
0:30:36 And I talked about things that live on the internet and live on the web.
0:30:40 This is still a really, really smart approach for package software.
0:30:47 And I know people who are working in and running package software companies that use this methodology
0:30:51 because it allows them to still work in small, fast approaches.
0:30:56 And all they do is they push to a small package pre-production database.
0:31:01 And then when it’s time to push that code onto some media, they do that.
0:31:02 Okay.
0:31:05 So what I love hearing about this is that it’s actually not necessarily tied again to the
0:31:07 architecture or the type of company you are.
0:31:11 There’s this opportunity for everybody, but there is this mindset of like an organization
0:31:12 that is ready.
0:31:14 It’s like a readiness level for a company.
0:31:15 Oh, I hear that all the time.
0:31:19 I don’t know if I’d say there’s any such thing as readiness, right?
0:31:21 Like there’s always an opportunity to get better.
0:31:24 There’s always an opportunity to transform.
0:31:29 The other thing that really drives me crazy and makes my head explode is this whole maturity
0:31:30 model thing.
0:31:31 Okay.
0:31:33 Are you ready to start transforming?
0:31:38 Well, like you can just not transform and then maybe fail, right?
0:31:42 Maturity models, they’re really popular in industry right now, but I really can’t stress
0:31:47 enough that they’re not really an appropriate way to think about a technology transformation.
0:31:50 I was thinking of readiness in the context of like NASA technology readiness levels or
0:31:54 TRLs, which is something we use to think about a lot for very early stage things, but you’re
0:31:58 describing maturity of an organization and it sounds like there’s some kind of a framework
0:32:02 for assessing the maturity of an organization and you’re saying that doesn’t work, but first
0:32:05 of all, what is that framework and why doesn’t it work?
0:32:10 Well, so so many people think that they want a snapshot of their like DevOps or their technology
0:32:14 transformation and spit back a number, right?
0:32:18 And then you will have one number to compare yourself against everything.
0:32:24 The challenge though is that a maturity model usually is leveraged to help you think about
0:32:27 arriving somewhere and then here’s the problem.
0:32:29 Once you’ve arrived, what happens?
0:32:30 Oh, we’re done.
0:32:31 You’re done.
0:32:33 And then the resources are gone.
0:32:38 And by resources, I don’t just mean money, I mean time, I mean attention.
0:32:44 We see year over year over year, the best, most innovative companies continue to push.
0:32:46 So what happens when you’ve arrived, I’m using my finger quotes.
0:32:47 You stop pushing.
0:32:48 You stop pushing.
0:32:54 What happens when executives or leaders or whomever decide that you no longer need resources
0:32:55 of any type?
0:33:00 I have to push back again though, doesn’t this help because it is helpful to give executives
0:33:04 in particular, particularly those that are not tech native, coming from the seeds of
0:33:09 the engineering organization, some kind of metric to put your head around where are we,
0:33:10 where are we at?
0:33:12 So you can use a capability model.
0:33:17 You can think about the capabilities that are necessary to drive your ability to develop
0:33:20 and deliver software with speed and stability.
0:33:24 Another limitation is that they’re often kind of a lockstep or a linear formula, right?
0:33:25 No, right.
0:33:28 It’s like a stepwise A, B, C, D, E, one, two, three, four.
0:33:32 And in fact, the very nature of anything iterative is it’s very nonlinear and circular.
0:33:33 Feedback loops are circled.
0:33:34 Right.
0:33:37 And maturity models just don’t allow that.
0:33:42 No, another thing that’s really, really nice is that capability models allow us to think
0:33:46 about capabilities in terms of these outcomes.
0:33:48 Capabilities drive impact.
0:33:53 Maturity models are just this thing where you have this level one, level two, level
0:33:54 three, level four.
0:33:55 It’s a bit performative.
0:34:02 And then finally, maturity models just sort of take this snapshot of the world and describe
0:34:03 it.
0:34:05 How fast is technology and business changing?
0:34:11 If we create a maturity model now, let’s wait, let’s say four years, that maturity model
0:34:14 is old and dead and dusty and gone.
0:34:16 Do new technologies change the way you think about this?
0:34:20 Because I’ve been thinking a lot about how product management for certain types of technologies
0:34:24 changes with the technology itself and that machine learning and deep learning might be
0:34:25 a different beast.
0:34:26 And I’m just wondering if you guys have any thoughts on that.
0:34:27 Yeah.
0:34:30 I mean, me and Dave Farley wrote the continuous delivery book back in 2010.
0:34:34 And since then, you know, there’s Docker and Kubernetes and large-scale adoption of the
0:34:37 cloud and all these things that you had no idea would happen.
0:34:40 People sometimes ask me, you know, isn’t it time you wrote a new edition of the book?
0:34:43 I mean, yeah, we would probably rewrite it.
0:34:45 Does it change any of the fundamental principles?
0:34:46 No.
0:34:50 Do these new tools allow you to achieve those principles in new ways?
0:34:51 Yes.
0:34:54 So, I think, you know, this is how I always come back to any problem is go back to first
0:34:55 principles.
0:34:56 Yeah.
0:34:59 And the first principles, I mean, they will change over the course of centuries.
0:35:04 I mean, we’ve got modern management versus kind of scientific management, but they don’t
0:35:06 change over the course of like a couple of years.
0:35:08 The principles are still the same.
0:35:11 These give you new ways to do them, and that’s what’s interesting about them.
0:35:13 Equally, things can go backwards.
0:35:17 A great example of this is one of the capabilities we talk about in the book is working off a
0:35:22 shared trunk or master inversion control, not going on these long-lived feature branches.
0:35:26 And the reason for that is actually because of feedback loops.
0:35:29 You know, developers love going off into a corner, putting headphones on their head and
0:35:34 just coding something for like days, and then they try and integrate it into trunk, you
0:35:35 know, and that’s a total nightmare.
0:35:38 And not just for them, more critically for everyone else who then has to merge their
0:35:41 coding so whatever they’re working on.
0:35:42 So that’s hugely painful.
0:35:45 Git is one of these examples of a tool that makes it very easy for people like, “Oh, I
0:35:46 can use feature branches.”
0:35:49 So I think, again, it’s non-linear in the way that you describe.
0:35:50 Right.
0:35:51 Gives you new ways to do things, are they good and bad?
0:35:52 It depends.
0:35:55 But the thing that strikes me about what you guys have been talking about as a theme in
0:35:59 this podcast that seems to lend itself, well, to the world of machine learning and deep
0:36:03 learning where that technology might be different, is it sort of lends itself to a probabilistic
0:36:09 way of thinking and that things are not necessarily always complete, and that there is not a beginning
0:36:13 and an end, and that you can actually live very comfortably in an environment where things
0:36:18 are by nature complex, and that complexity is not necessarily something to avoid.
0:36:22 So in that sense, I do think there might be something kind of neat about ML and deep learning
0:36:26 and AI for that matter, because it is very much lending itself to that sort of mindset.
0:36:27 Yeah.
0:36:30 And in our research, we talk about working in small batches.
0:36:35 There’s a great video by Brett Victor called Inventing on Principle, where he talks about
0:36:39 how important it is to the creative process to be able to see what you’re doing, and
0:36:43 he has this great demo of this game he’s building where he can change the code and the game
0:36:46 changes its behavior instantly when you’re doing things like that.
0:36:48 You don’t get to see that.
0:36:52 No, and the whole thing with machine learning is how can we get the shortest possible feedback
0:36:56 from changing the input parameters to seeing the effect so that the machine can learn,
0:37:01 and that the moment you have very long feedback loops, the ML becomes much, much harder because
0:37:04 you don’t know which of the input changes caused the change in output that the machine
0:37:06 is supposed to be learning from.
0:37:10 So the same thing is true of organizational change and process, and product development
0:37:14 as well, by the way, which is working in small batches so that you can actually reason about
0:37:15 causing effects.
0:37:16 I changed this thing.
0:37:17 It had this effect.
0:37:20 Again, that requires short feedback loops.
0:37:21 That requires small batches.
0:37:24 That’s one of the key capabilities we talk about in the book, and that’s what DevOps enables.
0:37:28 So we’ve been this hallway style conversation around all these themes of DevOps, measuring
0:37:31 it, why it matters, and what it means for organizations.
0:37:36 But practically speaking, if a company, and you guys are basically arguing it, any company,
0:37:40 not necessarily a “company” that thinks it’s a tech company, and necessarily a company
0:37:44 that has this amazing modern infrastructure stack, it could be a company that’s still
0:37:45 working off mainframes.
0:37:48 What should people actually do to get started, and how do they know where they are?
0:37:52 So what you need to do is take a look at your capabilities, understand what’s holding you
0:37:56 back, try to figure out what your constraints are.
0:38:02 But the thing that I love about much of this is you can start somewhere, and culture is
0:38:04 such a core, important piece.
0:38:09 We’ve seen across so many industries, culture is truly transformative.
0:38:13 In fact, we measure it in our work, and we can show that culture has a predictive effect
0:38:17 on organizational outcomes and on technology capabilities.
0:38:23 We use a model from a guy called Ron Westrom, who was a social scientist studying safety
0:38:27 outcomes, in fact, in safety-critical industries like healthcare and aviation.
0:38:33 He created a typology where he organizes organizations based on whether they’re pathological, bureaucratic
0:38:34 or generative.
0:38:35 That’s actually a great topology.
0:38:37 I wanted to apply that to people I date.
0:38:38 I know, right?
0:38:39 Too real.
0:38:40 I wanted to apply it to people.
0:38:41 Too real.
0:38:42 There’s a book in there, definitely.
0:38:46 I like how I’m trying to anthropomorphize all these organizational things into people.
0:38:47 But anyway, go on.
0:38:52 Instead of the five love languages, we can have the three relationship types.
0:38:55 Pathological organizations are characterized by a low cooperation between different departments
0:38:58 and up and down the organizational hierarchy.
0:39:00 How do we deal with people who bring us bad news?
0:39:03 Do we ignore them, or do we shoot people who bring us bad news?
0:39:04 How do we deal with responsibilities?
0:39:08 Are they defined tightly so that when something goes wrong, we know whose fault it is, so
0:39:09 we can punish them?
0:39:12 Or do we share risks, because we know we’re all in it together, and it’s the team?
0:39:13 You all have to get in the game.
0:39:14 You’re all accountable, right?
0:39:15 Exactly.
0:39:16 We’re all in different departments.
0:39:18 And crucially, how do we deal with failure?
0:39:23 As we discussed earlier, in any complex system, including organizational systems, failure
0:39:24 is inevitable.
0:39:28 So failure should be treated as a learning opportunity, not whose fault was it, but why
0:39:32 did that person not have the information they needed, the tools they needed?
0:39:35 How can we make sure that when someone does something, it doesn’t lead to catastrophic
0:39:39 outcomes, but instead it leads to contained small blast radiuses?
0:39:40 Right.
0:39:41 Not an outage on Black Friday.
0:39:42 Right.
0:39:43 Exactly.
0:39:45 So how do we deal with novelty?
0:39:48 Is novelty crushed, or is it implemented, or does it lead to problems?
0:39:52 One of the pieces of research that kind of confirms what we were talking about was some
0:39:56 research that was done by Google, they were trying to find what makes the greatest Google
0:39:57 team.
0:40:01 You know, is it four Stanford graduates and no developer and fire all the managers?
0:40:04 Is it a data scientist and a Node.js programmer and a manager?
0:40:05 Right.
0:40:08 One product manager paired with one system engineer, with one.
0:40:14 And what they found was that the number one ingredient was psychological safety.
0:40:17 Does the team feel safe to take risks?
0:40:19 And this ties together failure and novelty.
0:40:25 If people don’t feel that when things go wrong, they’re going to be supported, they’re not
0:40:26 going to take risks.
0:40:29 And then you’re not going to get any novelty, because novelty by definition involves taking
0:40:30 risks.
0:40:34 So we see that one of the biggest things you can do is create teams where it’s safe to
0:40:39 go wrong and make mistakes, and where people will treat that as a learning experience.
0:40:42 This is a principle that applies, again, not just in product development, you know, the
0:40:46 lean start up, fail early, fail often, but also in the way we deal with problems at an
0:40:48 operational level as well.
0:40:50 And how we interact with our team when these things happen.
0:40:54 So just to kind of summarize that, you have pathological, this is a power oriented thing
0:40:58 where you know the people are scared, the messenger is going to be shot.
0:41:02 Then you have this bureaucratic kind of rule oriented world where the messengers aren’t
0:41:03 heard.
0:41:07 And then you have the sort of generative, and again, I really wish I could apply this
0:41:11 to people, but we’re talking about organizations here for culture, which is more performance
0:41:12 oriented.
0:41:15 And I just want to add one thing about this, you know, working in the federal government,
0:41:17 you would imagine that to be a very bureaucratic organization.
0:41:18 I would actually.
0:41:22 And actually, what was surprising to me was that yes, there’s lots of rules.
0:41:23 The rules aren’t necessarily bad.
0:41:26 That’s how we can operate at scale is by having rules.
0:41:28 But what I found was there was a lot of people who are mission oriented.
0:41:32 And I think that’s a nice alternative way to think about generative organizations.
0:41:34 You need to think about mission orientation.
0:41:38 The rules are there, but if it’s important to the mission, we’ll break the rules.
0:41:40 And we measure this at the team level, right?
0:41:45 Because you can be in the government and there were pockets that were very generative.
0:41:53 You can be in a startup and you can see startups that act very bureaucratic or very pathological.
0:41:54 Right.
0:41:55 The culture of the CEO.
0:41:59 Where it’s not charismatic, inspirational vision, but to the expense of actually being
0:42:01 heard and the messenger is shot, et cetera.
0:42:05 And we have several companies around the world now that are measuring their culture on a
0:42:09 quarterly cadence and basis because we show in the book how to measure it.
0:42:12 Western’s typology was the table itself.
0:42:16 And so we turn that into a scientific psychometric way to measure it.
0:42:19 Now this makes sense why I’m putting these anthropomorphic analogies because in this
0:42:22 sense organizations are like people.
0:42:23 They’re made of people.
0:42:24 Teams are organic entities.
0:42:28 And I love that you said that the unit of analysis is a team because it means you can
0:42:29 actually do something.
0:42:31 You can start there and then you can like see if it actually spreads or doesn’t spread
0:42:34 bridges, doesn’t bridge, et cetera.
0:42:38 And what I also love about this framework is it also moves away from this cult of failure
0:42:42 mindset that I think people tend to have where it’s like failing for the sake of failing.
0:42:44 And you actually want to avoid failure.
0:42:45 Right.
0:42:48 And the whole point of failing is to actually learn something and then be better and take
0:42:49 risks.
0:42:50 So you can implement these new things.
0:42:52 And very smart risks.
0:42:53 So what’s your final?
0:42:58 I mean, there’s a lot of really great things here, but like what’s your final sort of parting
0:43:02 take away for listeners or people who might want to get started or think about how they
0:43:03 are doing.
0:43:06 So I think, you know, we’re in a world where technology matters.
0:43:10 Anyone can do this stuff, but you have to get the technology part of it right.
0:43:15 That means investing in your engineering capabilities, in your process, in your culture, in your
0:43:17 architecture.
0:43:20 We dealt with a lot of things here that people think are intangible and we’re here to tell
0:43:21 you they’re not intangible.
0:43:22 You can measure them.
0:43:24 They will impact the performance of your organization.
0:43:29 So take a scientific approach to improving your organization and you will read the dividends.
0:43:32 When you guys talk about, you know, anyone can do this, the teams can do this, but what
0:43:37 role in the organization is usually most empowered to be the owner of where to get started?
0:43:39 Is it like the VP of engineering?
0:43:41 Is it the CTO, the CIO?
0:43:46 I was going to say, don’t minimize the role of and the importance of leadership.
0:43:53 DevOps sort of started as a grassroots movement, but right now we’re seeing roles like VP and
0:43:58 CTO being really impactful in part because they can set the vision for an organization,
0:44:01 but also in part because they have resources that they can dedicate to this.
0:44:04 We see a lot of CEOs and CTOs and CIOs in our business.
0:44:05 We have like a whole briefing center.
0:44:08 We hear what’s top of mind for them all the time.
0:44:09 Everyone thinks they’re transformational.
0:44:13 So like what actually makes a visionary type of leader who has that, not just the purse
0:44:18 strings and the decision-making power, but the actual characteristics that are right
0:44:19 for this.
0:44:20 Right.
0:44:21 And that’s such a great question.
0:44:24 We dug into that in our research and we find that there are five characteristics that end
0:44:31 up being predictive of driving change and really amplifying all of the other capabilities
0:44:32 that we found.
0:44:38 And these five characteristics are vision, intellectual stimulation, inspirational communication,
0:44:40 supportive leadership, and personal recognition.
0:44:46 And so what we end up recommending to organizations is absolutely invest in the technology.
0:44:51 So invest in leadership in your people because that can really help drive your transformation
0:44:52 home.
0:44:56 Well, Nicole, Jez, thank you for joining the A6 and Z podcast.
0:45:02 The book Just Out is Accelerate, Building and Scaling High-Performing Technology Organizations.
0:45:03 Thank you so much, you guys.
0:45:04 Thanks for having us.
0:45:04 Thank you.

0:00:07 Hi everyone, welcome to the A6NZ podcast, I’m Sonal.
0:00:12 We’ve been covering the novel coronavirus and COVID-19 disease on our other show, 16
0:00:16 Minutes, which you can find in a separate feed if you haven’t subscribed already, but
0:00:22 given that the topic of health system preparedness is top of mind right now and that the latest
0:00:28 CDC briefing touched on issues with test kits, the patient perspective of how the U.S. health
0:00:32 care system works, with clinicians calling the health department and so on, we’re running
0:00:38 last week’s episode of 16 Minutes in this feed because in it we covered how these tests
0:00:44 work, how the U.S. health care system works today when it comes to epidemics and preparedness
0:00:46 and where we might go in the future.
0:00:54 As a reminder, you can visit cdc.gov and who.int for more information, but as of now, the World
0:00:58 Health Organization reported on February 25th that for the first time since the onset of
0:01:04 symptoms of the first identified case of COVID-19, there have been more new cases reported from
0:01:09 countries outside of China than from China, and the CDC reported on February 26th that
0:01:15 there’s news about community spread, which means that cases of COVID-19 are appearing
0:01:18 without a known source of exposure.
0:01:23 And those communities currently include Hong Kong, Italy, Iran, Singapore, South Korea,
0:01:25 Taiwan, and Thailand.
0:01:29 So that’s the latest updates, now onto last week’s episode.
0:01:34 Hi everyone, welcome to the 23rd episode of 16 Minutes, our show where we cover the headlines
0:01:38 and what’s in the news, what’s happening, and tease apart what’s hype, what’s real
0:01:39 from our vantage point in tech.
0:01:41 I’m Sonal.
0:01:44 Today we’re doing another update on the topic of coronavirus.
0:01:48 We did a deeper dive in episode 21, which you can find in this feed or on our website
0:01:51 at a6nz.com/16minutes.
0:01:55 Much of that background is still relevant today, but in this episode we’re going to
0:01:56 cover two segments.
0:02:00 First, we’ll do a high-level overview of some of the practical implications for the U.S.
0:02:05 healthcare system with a6nz.bio general partner Julie Yu, and then the second segment is a
0:02:09 quick situation update from our previous episode with Judy Svitskaya.
0:02:12 As a reminder, all the sources and reports cited in this episode will be included in
0:02:16 the show notes, and we are not covering the clinical infectious disease specifics as we
0:02:20 will bring on our other experts for that in an upcoming episode.
0:02:21 So that’s a context.
0:02:25 Now before we chat, let me quickly share the latest updates, which are that the day after
0:02:30 we dropped our last episode, the World Health Organization declared on January 30th that
0:02:34 the coronavirus outbreak is a, quote, “public health emergency of international concern.”
0:02:37 And then the day after that, the Health and Human Services Secretary of the United States
0:02:41 declared a public health emergency to aid the nation’s healthcare community in responding
0:02:45 to the novel coronavirus, which, by the way, was officially named last week.
0:02:48 It is now known as COVID-19.
0:02:52 And to be clear, that’s the name of the disease, not the virus, which, as mentioned in our
0:02:58 last episode, was known as 2019 NCOV, but is now known as SARS-CoV-2.
0:03:03 And then also, as of last week, a lot happened in a week, the CMS, the U.S. Centers for
0:03:08 Medicare and Medicaid Services, developed a new billing code for providers and laboratories
0:03:12 to test patients for this virus that causes COVID-19.
0:03:16 And we’ll share details about how that test works in the second half of this episode, as
0:03:18 well as the latest global numbers.
0:03:22 But first, now let’s cover the U.S. care delivery aspects.
0:03:28 According to the CDC, as of February 17th, there are a total of 467 persons under investigation
0:03:34 for this in the United States, identified across 42 states, with 15 confirmed positive
0:03:36 for it and 60 pending.
0:03:40 So Julie, practically speaking, what’s actually happening here in our health care system as
0:03:41 it works today?
0:03:43 What happens when someone walks into a hospital?
0:03:47 So it depends a lot on where you’re walking into.
0:03:53 Most of the time, because our health care system is characterized by such access constraints,
0:03:56 you may see a lot of these patients actually showing up in the emergency room.
0:04:01 What happens is that they will check in with a registrar, essentially, and be asked, “What
0:04:02 is your reason for being here?”
0:04:05 Typically, they’ll also be collecting your insurance information.
0:04:07 You will do a physical visual assessment.
0:04:09 They might ask some very generic questions.
0:04:14 One of the training motions that’s happening in hospitals is that people are trying to
0:04:18 train those frontline staff to ask questions like, “Have you traveled to China in the last
0:04:19 two weeks?”
0:04:20 Et cetera.
0:04:24 And so you have to deploy field resource to make sure that even those frontline questions
0:04:25 are being asked.
0:04:26 Right.
0:04:27 I get it.
0:04:30 Sort of the difference between a generalist triage model and a more specialist triage
0:04:31 model.
0:04:32 Exactly.
0:04:33 Because that’s the biggest blind spot right now.
0:04:36 One of the big risks of this particular virus is that the pre-syntomatic period while you
0:04:39 are still contagious is fairly lengthy.
0:04:42 And so that’s one of the big sort of gaps right now is, you know, how do we just identify
0:04:43 those people?
0:04:46 So after they go through the ER and then what happens?
0:04:51 So assuming that people are being appropriately assessed and there is a determination that
0:04:56 there’s a potential risk that you are a coronavirus patient, you administer the test, again, assuming
0:05:00 that the test kits are in supply and based on those results.
0:05:05 If the patient is quarantined and assuming, again, that they are in an acute care facility
0:05:10 that has infrastructure to actually perform an appropriate quarantine, typically those
0:05:14 quarantine rooms are what are called negative pressure rooms, which basically means the
0:05:18 air in that room is sort of minimal seepage externally.
0:05:21 You’re essentially isolating the potential germs and contagion.
0:05:26 But again, that sort of begs again the point of, “Are you showing up at an ED of a facility
0:05:28 that actually has all this infrastructure?”
0:05:32 Many of these patients might just be walking into an urgent care clinic or a primary care
0:05:33 clinic.
0:05:37 And so oftentimes it might be the case that the patient could get sent home or referred
0:05:41 into one of these facilities with further delay, further exposure points, et cetera.
0:05:46 So basically there’s a potential for a lot of chaos on the front lines because we don’t
0:05:49 clearly understand where the risk points are.
0:05:53 And we’re sort of waiting for the patients to essentially show up versus being able to
0:05:54 be proactive.
0:05:55 Right.
0:05:56 And how about on the treatment side?
0:05:57 So there currently is no treatment for this.
0:06:02 My partner Jorge and I, and Jorge being an expert in genomics, oftentimes talk about the
0:06:07 areas of medicine and healthcare where clearly there’s an application for technology that
0:06:11 makes complete sense, but oftentimes it’s the business model component of it that holds
0:06:12 things back.
0:06:13 What do you guys mean by that?
0:06:19 So we have the capabilities to rapidly sequence bugs and other viral forms.
0:06:25 And there’s in theory a capability that says if you are able to quickly identify and rapidly
0:06:30 identify in the field, what type of bug you’re dealing with, that you could also synthesize
0:06:36 a vaccine on demand based on the fact that we can increasingly print genomic tools and
0:06:37 genomic content.
0:06:38 That’s technically possible right now?
0:06:39 The technology exists.
0:06:41 It’s not yet been deployed into practice.
0:06:45 There’s still a great degree of validation and testing.
0:06:49 Obviously we have a very rigorous system through which these types of technology are brought
0:06:53 to market with regards to clinical trials and regulation and whatnot.
0:06:58 The other piece is historically vaccines and other types of treatments like this have not
0:07:03 been a lucrative area for businesses to go into for a number of reasons.
0:07:08 And that’s another area that’s TBD is can you actually find the reimbursement path for
0:07:09 getting these products to market?
0:07:10 Okay.
0:07:13 So where we are right now, it seems like the focus is on what I’m calling the 3Gs, gowns,
0:07:14 goggles, and gloves.
0:07:19 I’m also very interested at a broader level because the World Health Organization did their
0:07:22 first annual report on global preparedness for health emergencies.
0:07:25 They basically wrote in their report, just came out last year, and they have targets
0:07:30 to September 2020 for progress towards that, that countries, donors, and multilateral institutions
0:07:35 must be prepared for the worst, quote, “A rapidly spreading pandemic due to a lethal
0:07:40 respiratory pathogen, whether naturally emergent or accidentally or deliberately released,
0:07:44 poses additional preparedness requirements, and that we must ensure adequate investment
0:07:48 in developing innovative vaccines and therapeutics as you talked about, surge manufacturing capacity,
0:07:51 appropriate non-pharmaceutical interventions, et cetera, et cetera.”
0:07:53 I guess I have two questions for you here.
0:07:58 One, where are we as a country from a systemic point on that readiness?
0:08:01 And then two, what does it say about where we should be?
0:08:05 So this to me is one of the biggest cases to be made for this concept of the unbundling
0:08:06 of the hospital.
0:08:12 When you look back at the history of how the facility side of healthcare has evolved, hospitals
0:08:16 were something that were born in the last century or so on the premise that if you were
0:08:21 to centralize the scarce resources, the doctors, the clinicians in a central location that
0:08:22 you could get efficiencies.
0:08:27 And by the way, also have the infrastructure, like the op-ex and cap-ex required to do big
0:08:30 labs and centralize the facilities and high-end procedures.
0:08:31 Right.
0:08:32 Exactly.
0:08:33 Not just the people.
0:08:34 Exactly.
0:08:37 And so the unfortunate consequence of that is that, yes, you can have these now very
0:08:41 high-end facilities that perform very advanced procedures, but where, again, we are forcing
0:08:45 the patients to travel outside of their homes, but also get exposed to others who have other
0:08:46 illnesses.
0:08:51 In fact, hospital-acquired infections are one of the major contributors to comorbidities
0:08:53 for patients who come to these acute care facilities.
0:08:56 We are in the middle of flu season, remember?
0:08:57 Yes.
0:09:01 So you’re already having rooms full of patients who suspect that they have some kind of illness.
0:09:05 That’s kind of like an iconic motion within our healthcare system is that we force patients
0:09:11 to come to these central monolithic facilities to get any kind of care versus going to them,
0:09:12 making it convenient to them.
0:09:16 And actually, it’s interesting that when you look back in the early 1900s, nearly half
0:09:19 of healthcare was actually delivered in the home.
0:09:22 Less than 1% of healthcare is now delivered in the home, even for the most senior and
0:09:24 frail patients in America.
0:09:25 Right.
0:09:28 And it’s also an access issue, because it means that people who can’t afford or live
0:09:33 in big hubs where you can afford these types of high, varying quality.
0:09:38 What that’s predicated on, though, it requires productization of the types of technologies
0:09:42 that you see in these hospital settings in such a way that can be decentralized.
0:09:47 A great basic but kind of elegant example of this is you see companies, one in particular
0:09:50 comes to mind that is doing a connected thermometer.
0:09:53 It’s marketed towards parents as something that they can use for their kids.
0:09:58 And when you look at the back end of their business, it’s basically a data company that
0:10:03 is acting as a sentinel to collect information about temperatures in communities and essentially
0:10:07 predict when there will be a flu outbreak or a cold outbreak, et cetera.
0:10:12 And they actually notify not only the end users, but they have connectivity into schools,
0:10:14 churches, other institutions.
0:10:19 And you can imagine that a system like that at scale for various types of diseases could
0:10:22 actually enable this sort of truly decentralized model.
0:10:27 But the only way that this can happen is if you have interoperable data systems that can
0:10:32 not only collect data from the clinical setting and make it readily available on an ad hoc
0:10:37 basis and across facilities across the country, but also take into account non-traditional
0:10:41 data sources like these smart devices that are connected in the communities to augment
0:10:44 your visibility across patient populations.
0:10:45 Okay.
0:10:47 So that’s sort of the unbundling of the hospital.
0:10:51 In that context, it all comes together like connects the dots.
0:10:56 This is how interoperability and data liquidity and data from unconventional sources and all
0:10:58 this stuff comes together.
0:11:01 That’s on the future of where we could go and what the ideal state could be.
0:11:04 What are some of the things that can happen now inside the hospital?
0:11:07 There’s, I would say, the human elements, the operational elements, and then the technology
0:11:08 elements.
0:11:14 So on the human side, this is what these organizations are sort of designed to do is deploy large
0:11:19 swaths of human labor in such a way that can sort of react to healthcare needs.
0:11:24 Operationally speaking, we mentioned earlier the logistics of how a patient flows through
0:11:25 the hospital.
0:11:28 You need to anticipate all the potential entry points that patients are coming in.
0:11:32 The hospital these days are health systems, really, and they need to have connectivity
0:11:37 into their primary care clinics, their urgent care clinics, et cetera, to really understand
0:11:40 systematically what’s going on across the network.
0:11:41 And now the tech part.
0:11:42 That’s what I’m most interested in given your vantage point.
0:11:43 Yeah.
0:11:44 Here’s what I’m going to say.
0:11:48 One nice thing about EHRs now, they are literally the primary tool that frontline clinicians
0:11:50 are using.
0:11:56 You’ve seen this now, hospitals literally interjecting very basic questions into the medical record
0:12:01 to prompt them to ask the things that could qualify whether or not a patient might potentially
0:12:03 be at risk for coronavirus.
0:12:08 So that’s where the fact that we now have this broad infrastructure layer laid down can
0:12:12 actually provide tremendous value in that you can make one change that does get propagated
0:12:15 to all of the endpoints in the care delivery system.
0:12:17 So doing things at scale through technology, basically.
0:12:22 So bottom line for me, Julie, how should we think about this in terms of the tech and
0:12:26 the delivery side and preparedness for the epidemic at that level?
0:12:31 I think this is shedding light on the fact that we as a healthcare system have many nodes
0:12:37 of potential failure when it comes to widespread epidemics and pandemics, but the direction
0:12:42 and everything that we’ve talked about around the notion of decentralization, of unbundling
0:12:48 of hospital, of using technology and distributed data streams to be able to be more responsive
0:12:51 and nimble is coming to light.
0:12:54 And so we will take learnings from this and apply it towards what the future of healthcare
0:12:55 needs to look like.
0:12:56 Thank you for joining this segment.
0:12:57 All right.
0:12:58 Thank you so much.
0:13:01 Now let me introduce Judy Savitskaya on the A6NZ Bio team.
0:13:02 Welcome, Judy.
0:13:03 Thanks, Donald.
0:13:04 Okay.
0:13:07 So let me just give a quick update on the stats of the disease.
0:13:11 This is Situation Report Number 25 from the World Health Organization.
0:13:13 We just came out February 14th.
0:13:16 Here’s the high-level summary of the numbers.
0:13:21 So globally, there are now 49,053 laboratory-confirmed cases.
0:13:26 In China, there are 48,548 laboratory-confirmed ones.
0:13:31 And then outside of China, there are 505 across 24 countries with two deaths outside of China.
0:13:35 The other thing, though, is there was a huge spike in the numbers.
0:13:39 And that was because they will include the number of clinically diagnosed cases into
0:13:43 the number of confirmed cases so that patients could receive timely treatment.
0:13:46 And previously, patients could only be diagnosed by test kits.
0:13:47 What does this mean scientifically?
0:13:48 Yeah.
0:13:53 So these cases have in the past been basically labeled as coronavirus cases, whether they
0:13:56 have the right nucleic acid sequence that belongs to that virus.
0:14:01 What they’re saying now is that they’re also going to count anybody who is symptomatic
0:14:05 in all the same ways that the virus has been presenting itself in other patients and has
0:14:06 the CT scan evidence.
0:14:11 So the former FDA commissioner, Scott Gottlieb, noted that this is happening because it’s
0:14:14 in the absence of a PCR test, which we briefly talked about last time.
0:14:18 There’s an open question about why these are not yet available at scale.
0:14:21 But can you give us a little bit more detail about what is the PCR test scientifically?
0:14:22 Yeah.
0:14:27 So PCR test stands for polymerase chain reaction, basically with amplifying a piece of DNA or
0:14:31 RNA nucleic acid by copying it over and over and over again.
0:14:35 What you’re doing in this test is basically you’re taking a sample from the patient.
0:14:37 There’s some nucleic acids in there.
0:14:41 The sequence is very long, but you take a small sequence of DNA, RNA, whatever you’re
0:14:43 trying to amplify.
0:14:44 This is an RNA virus.
0:14:49 So you’re trying to bind it with a sequence that you know belongs to that virus.
0:14:52 You attach it, it’s about 20 bases in length.
0:14:56 You use the polymerase chain reaction to extend out that 20 base primer to cover the entire
0:15:00 sequence or whatever like piece of the sequence that you’re trying to amplify.
0:15:02 And then you get many, many copies this way.
0:15:04 What does that give you having the many, many copies?
0:15:05 Surveillance or absence, right?
0:15:08 So, and amounts, it’s called real time PCR.
0:15:09 I actually don’t love the name.
0:15:14 I think QPCR is a better name for this quantitative PCR because it’s telling you how many pieces
0:15:18 of essentially like what is the viral load in the bloodstream or the load of whatever
0:15:20 pathogen you’re looking for.
0:15:24 So Keith Robison, who’s currently principal scientist at Ginkgo wrote about, you know,
0:15:25 how some of these tests work.
0:15:30 And he basically agrees with you that it should be called QPCR because as you note, what you’re
0:15:31 basically describing as it’s quantitative.
0:15:32 Yeah.
0:15:33 And real time doesn’t really mean much.
0:15:38 But there’s another critical reason why people don’t like RT-PCR is because there’s a completely
0:15:41 different concept that is called reverse transcriptase PCR.
0:15:44 That’s why it’s kind of hard to talk about this with this virus because it’s an RNA virus.
0:15:45 Right.
0:15:48 In fact, he also talks about the fact that PCR works with DNA.
0:15:52 But yet you’re telling me coronavirus is RNA, so can you help explain that distinction?
0:15:53 Absolutely.
0:15:58 So in this reaction, what you’re doing is using a polymerase that is used to binding either
0:16:00 DNA or RNA and then extending it.
0:16:04 So in the case of the RNA viruses, you need the reverse transcriptase.
0:16:10 So this is a weirdo polymerase that binds RNA templates and then extends and produces
0:16:11 DNA.
0:16:13 And the reason that you want DNA is that it’s really stable.
0:16:15 We have a ton of ways to measure it.
0:16:17 RNA is a little bit more fickle.
0:16:23 So if you can turn this RNA signature, this RNA message into a DNA output that actually
0:16:26 substantially simplifies downstream processing.
0:16:30 So this reverse transcriptase piece is what is doing the RNA to DNA translation.
0:16:31 Okay.
0:16:33 So we’ve talked about what’s going on in the test.
0:16:36 Let’s quickly talk about some of the differences from what we last talked about.
0:16:41 We talked about R0 last time, which is really practically how many people, a newly infected
0:16:46 person is likely to pass a virus on to and you explained what variables go into it.
0:16:48 What is your take on where we are with the R0?
0:16:49 Yeah.
0:16:52 We just talked about the spike, the definition of what this disease is.
0:16:55 Is it the viral load or is it like these symptoms?
0:16:56 That’s changing as well.
0:16:59 So I still think it’s too early to calculate an R0.
0:17:04 There’s still a ton of cases out there that are not showing symptoms, so we can’t really
0:17:06 calculate the number of people who have gotten infected.
0:17:11 I think we have technically approached the point where it is a pandemic, although the
0:17:13 definition for pandemic is quite loose.
0:17:17 The World Health Organization defines it as a worldwide spread of a new disease.
0:17:21 The Centers for Disease Control and Prevention, the CDC and the U.S. have a bit looser of
0:17:25 a definition describing as a disease that spreads across regions.
0:17:29 And quote from the CDC website is the fact that this virus has caused illness, including
0:17:35 illness resulting in death and sustained person-to-person spread in China is concerning.
0:17:39 These factors meet two of the criteria of a pandemic.
0:17:41 And by the way, people want to read an excellent piece.
0:17:45 Helen Branswell, and I mentioned her in our last episode, has a great piece in stat news
0:17:49 with the headline quote, “Undershining Pandemics, What They Mean, Don’t Mean, and What Comes
0:17:51 Next with the Coronavirus.”
0:17:53 From your take, why is it so freaking confusing?
0:17:58 The term pandemic is not particularly useful in this case because it only tells you about
0:17:59 the geographical spread.
0:18:02 It’s not actually telling you about the danger of the disease.
0:18:07 Like flu is a global pandemic annually, but the term doesn’t necessarily mean, you know,
0:18:09 very fatal or spreads very fast.
0:18:13 It just means it’s been into more than two geographies outside of its original origin.
0:18:20 So if a flu is a pandemic, that’s also endemic in that it is in our population and circulates.
0:18:22 Can you actually explain endemic?
0:18:25 Because my understanding of the word comes from like understanding evolution and Darwin
0:18:28 and knowing about endemic species and the Galapagos.
0:18:29 Yeah.
0:18:30 What does that mean?
0:18:32 So endemic is a more useful term than pandemic.
0:18:37 It’s something that is going to live in a latent way in the population or in the environment.
0:18:41 We should see a returning flu as the quintessential example of this.
0:18:45 It’s still an open question as to whether this coronavirus is going to become endemic.
0:18:46 Okay.
0:18:49 So that’s the difference between pandemics, endemic, and add one more name to the list,
0:18:50 which is misinfodemic.
0:18:56 I’ve read a lot of people describing this potentially as an infodemic because of the
0:19:00 spread of some fantastic rapid science, which you talked about last time, but there’s also
0:19:03 a spread of misinformation as well.
0:19:05 And so the two of these things are going hand in hand.
0:19:08 There’s a group that has already published an epidemiological model of what they expect
0:19:09 the spread to be.
0:19:15 Again, if any of the data that’s going into there is either intentionally falsified or
0:19:17 it is just too early.
0:19:18 Incomplete.
0:19:19 We don’t have good enough data.
0:19:21 Or like the measurements have changed, right?
0:19:26 So in the middle of last week, the way that Chinese hospitals were measuring cases changed.
0:19:29 So that’s going to mess up the data pretty substantially.
0:19:33 So I think that these models are going to suffer if garbage in, garbage out.
0:19:35 If this infodemic issue continues.
0:19:36 Okay.
0:19:40 So beyond the numbers and the definition, let’s quickly talk about some of the weightings.
0:19:44 According to the World Health Organization, some of the data from China last before this
0:19:50 big spike suggested that 82% of confirmed cases have only mild infection.
0:19:56 About 15% are severe enough to require hospital care and about 3% need intensive care.
0:19:59 And then preliminary data suggested that roughly 2% of the people who tested positive for the
0:20:00 virus have died.
0:20:05 And that’s important because last time we reported the CFR, the case fatality rates,
0:20:07 which for SARS was at 10%.
0:20:14 And for MERS, it was actually 37% in Saudi Arabia, but 34% outside of that region.
0:20:18 So last time you talked about the paradox between deadliness and the R0.
0:20:20 What’s your updates, if any, on that?
0:20:25 So the reason for that is that you can’t really have a high fatality rate and a fast
0:20:26 spreading virus.
0:20:30 Basically, dead people can’t spread the disease and people who are, you know, confined to
0:20:32 their beds also can’t spread the disease as fast.
0:20:35 But there’s another variable, which is incubation time.
0:20:41 So this is the length of time that it takes for the infection to demonstrate some symptoms.
0:20:44 And there’s a different period of time that’s called the latent period, which is the time
0:20:49 between getting infected and becoming infectious.
0:20:50 So these are two different variables.
0:20:52 And these interplay in a really interesting way.
0:20:56 If the latent time is really short, so you are infectious almost as soon as you’ve been
0:21:00 infected, but the incubation time is long, you have no idea that you’re infected.
0:21:02 You have no symptoms.
0:21:05 You feel completely normal, but it turns out that you’re actually spreading the virus.
0:21:10 So in that case, this sort of paradox between the case fatality and the spread rate is going
0:21:14 to break because you can start spreading without actually having symptoms.
0:21:18 It’s also probably too early to tell what the exact incubation period is going to be.
0:21:22 Most estimates I’ve seen have topped out at about 14 days, but that’s still pretty long.
0:21:25 So it’s something to definitely take into consideration.
0:21:26 Okay.
0:21:30 So bottom line it for me, where are we now in the situation update from the news and
0:21:31 your perspective?
0:21:35 So the bottom line is it’s still too early to put hard numbers on any of these facts.
0:21:40 It’s important to keep track of where the cases are coming up, where they’re being reported,
0:21:44 and don’t jump to any conclusions about case fatality rates, about R knots, because it’s
0:21:46 just too early.
0:21:49 Other than that, the same precautions apply.
0:21:50 Thank you for joining the segment.

0:00:05 The content here is for informational purposes only, should not be taken as legal business
0:00:10 tax or investment advice or be used to evaluate any investment or security and is not directed
0:00:15 at any investors or potential investors in any A16Z fund. For more details, please see
0:00:21 A16Z.com/disclosures. Hi, everyone. It’s Marketplaces Week over here. Thanks
0:00:26 to our consumer team releasing a new index of the next industry-defining marketplaces.
0:00:32 You can check that out at A16Z.com/marketplace100. But what happens as such marketplaces and other
0:00:38 platforms and products evolve over time as do their users? The conversation that follows
0:00:41 is one of our more popular episodes from a couple years ago, featuring general partners
0:00:47 Andrew Chen and Jeff Jordan in conversation with me on how after you acquire users, which
0:00:52 we covered in another episode, then how do you keep them engaged, retain them, and even
0:00:58 resurrect or re-engage them and what are the key metrics? But first, we began with what
0:01:02 happens after the initial acquisition as different kinds of users join a product or
0:01:07 platform over time. What does that mean for engagement and where do cohort analyses come
0:01:08 in?
0:01:12 One of the things that you see is that people end up using these products very differently
0:01:17 because the kinds of users that you’re getting are changing over time. When you look at something
0:01:24 like rideshare, all the early cohorts are basically people in urban areas. These days,
0:01:29 all of rideshare is more like suburban or rural folks because you’ve saturated all of
0:01:36 the center. What you tend to see is as you acquire your core demographic out, that actually
0:01:42 ends up showing up in the engagement. Going back to natural gravity to the whole thing,
0:01:46 because gravity also hits the engagement side of things as well, and then ultimately the
0:01:52 LTV because your users are typically getting less valuable. It may take years to see this
0:01:56 kind of play out, but that’s kind of the natural law of things.
0:01:59 There is a progression in these and particularly the ones that are really successful. Early
0:02:04 on, it’s all about getting users. Just like users, users, users. If you’re wildly successful
0:02:10 at doing that, you run out of users or you start running low on users and you have to
0:02:15 go to engagement. Pinterest has a very high quality problem right now. Most women in
0:02:18 America have downloaded the Pinterest app.
0:02:20 Oh yeah, I’ve had it for years.
0:02:24 Some growth can come through, okay, there are some number of women who’ve never heard of
0:02:28 Pinterest somewhere in the country, but much more so, they need to engage and re-engage
0:02:33 the existing audience. I mean, we love engagement from an investor standpoint because it’s just
0:02:34 …
0:02:36 It shows stickiness.
0:02:40 You can often hack your way into new users. It’s really hard to hack your way into true
0:02:41 engagement.
0:02:42 Keeping them.
0:02:47 If someone’s spending 20 minutes a day on your site, Pinterest and OfferUp, the major
0:02:53 investment thesis was, “Oh my God, look at that engagement kind of thing.” If they can
0:02:55 scale the user base, it’s a beautiful deal.
0:02:59 What we mean by engagement is actually interacting with them and seeing their activity because
0:03:05 to Andrew’s three points of acquisition engagement retention, the third piece is keeping them.
0:03:10 The way that we’ll often analyze this is looking at cohort analyses where we’ll look at each
0:03:15 batch of users that’s joining in each week and really start to dissect, “Well, how active
0:03:16 are they really?”
0:03:22 To compare all these cohorts over time, you’re basically putting the users that come in from
0:03:27 a particular time frame, let’s say it’s a week, and you’re putting them into a bucket.
0:03:31 What you’re doing is you want to compare all of these different buckets against each other.
0:03:36 What you typically do is you look at a bucket of a cohort of users and you say, “Okay, well,
0:03:40 once they’ve signed up, the week after, how active are they? What about the week after
0:03:44 that and the week after that and you can build out a curve?” It just turns out that these
0:03:49 curves, once you’ve looked at enough of them, surprisingly, human nature, they all look
0:03:50 kind of the same.
0:03:51 I bet.
0:03:55 They kind of all kind of curve down and for the good ones, they start to flatten out and
0:03:59 they plateau and then for the really good ones, they’ll actually swing back up and people
0:04:01 will come back to the surface.
0:04:06 What you want to do is you want to compare the various cohorts against each other in
0:04:13 time to see if you can spot any trends on how the usage patterns are increasing or decreasing.
0:04:18 When you add a new layer to layer cake, you might unlock a bunch of new behavior.
0:04:22 You might unlock a bunch of new frequency that didn’t exist before or alternatively,
0:04:27 over long thresholds of time, people tend to become less active as you move out of your
0:04:28 course.
0:04:29 The cohort graduates.
0:04:33 If not, a specific cohort of users flattens out is really important because if you’re
0:04:37 in a world where they slowly degrades and then all of a sudden it’ll actually go to
0:04:42 zero, that means that you’re naturally always filling up the bucket.
0:04:43 You kind of have a leaky bucket.
0:04:44 You’re constantly filling it out.
0:04:45 You’re always filling it up.
0:04:46 Right?
0:04:51 What happens is that gets progressively harder if you want to keep your overall growth rate
0:04:57 because that means you have to double, triple, quadruple your acquisition in order to counteract
0:04:58 for that.
0:05:03 That growth accounting equation that’s often thrown around is that your net MAUs, so your
0:05:09 net monthly active users, equals all the new people that you’re acquiring, right, minus
0:05:14 all the people that are churned, right, and then plus all the people that you’re resurrecting,
0:05:15 you know.
0:05:16 Reengaging.
0:05:17 Reengaging, exactly.
0:05:18 Right.
0:05:19 That are coming back after they’ve churned.
0:05:23 And so what happens is for a new startup, you are completely focused on new users because
0:05:26 you don’t really have that many users to churn.
0:05:30 And over time, as you get bigger and bigger and bigger, what you find is that your churn
0:05:33 rate starts to, you know, it’s a percentage of your actives.
0:05:37 And so, you know, the evolution of most of these companies as they’re getting bigger
0:05:43 tends to start with acquisition, then focus much more on, you know, churn and retention,
0:05:47 and then ultimately also to layer in resurrection as well.
0:05:50 And the cohort curves have a couple other features that I love.
0:05:56 And usually in marketplace businesses, the best models are built off of the cohort curves.
0:05:57 Oh, interesting.
0:06:00 Because you have to understand that degradation and where it goes.
0:06:03 Using cohorts really give you a sense of, are there network effects?
0:06:07 And network effect is the business gets more valuable, the more users that use it.
0:06:12 If it gets more valuable, your newer cohorts should behave better than your early cohorts.
0:06:13 Why is that?
0:06:16 Because the service is more valuable given how many there are.
0:06:17 Oh, interesting.
0:06:18 So that’s kind of a tip.
0:06:19 I get that.
0:06:22 If you’re going to get more restaurants, you’re going to get a whole lot more reservations
0:06:25 per diner because you are serving more than you need.
0:06:30 So the open table cohorts would climb up and get more attractive over time.
0:06:34 Versus, you know, we talked about typically they tended to grade over time.
0:06:38 If you’ve reversed the polarity and they’re growing over time, it means you’ve made the
0:06:41 business more valuable and then you start projecting forward.
0:06:44 Okay, what a better way to know the business is actually more valuable than thinking is
0:06:46 valuable and bleeding your own mind.
0:06:50 And in network effects business, we always ask, show us the cohorts.
0:06:52 Everyone asserts, I’m a network effect.
0:06:53 I’m a network effect.
0:07:00 But when you say show me the data, cohort curves, or they don’t lie, they don’t lie.
0:07:02 The other really interesting part is segmenting it.
0:07:04 I was about to actually ask you what are the buckets of cohorts?
0:07:06 Are they all demographic data?
0:07:10 For a bunch of hyper-local type businesses, the reason why segmenting it based on market
0:07:15 geography, why that’s so valuable is because then you can compare markets against each
0:07:16 other.
0:07:19 You can say, well, you know, this market which has much more density in terms of the number
0:07:23 of scooters behaves like this, and you can start to draw conclusions, you know, sort
0:07:26 of a natural A/B test in order to do that.
0:07:30 And I think the similar kind of analysis you can do for B2B companies is for products that
0:07:33 have different size teams using it.
0:07:37 If you have a really large team that are all using a product, well, are they all using
0:07:39 the product more as a result?
0:07:42 And let’s compare that to something that maybe only has a couple, right?
0:07:45 And so this way, you can start to kind of disassemble, you know, the structure of these
0:07:48 networks and do they actually lead to higher engagement?
0:07:50 Slack would be a perfect example of that.
0:07:54 You know, just if you have five people in the organization using Slack, you get one use
0:07:55 curve.
0:07:58 If you have, you know, if the organization, it’s the operating system for the organization,
0:07:59 you have a very different curve.
0:08:03 Though it’s not just an accident, you have to sort of architect it, not just expect like
0:08:05 serendipity to fall into place.
0:08:10 So after you get the new users, the way that you have to think about it is around quality,
0:08:11 right?
0:08:14 And you have to make sure that the new users turn into engaged users.
0:08:17 One of the things people often talk about is just sort of this idea of like an aha
0:08:22 moment or a magic moment where the user really understands the true value of the product.
0:08:23 But often that involves a bunch of setup.
0:08:27 So for example, for all the different social products, whether that’s Twitter or Facebook
0:08:31 or Pinterest, et cetera, you have to make sure that when you first bring a new user
0:08:36 and they have to follow all the right people, they have to get the onboarding experience.
0:08:39 Which by the way, isn’t just signing up, but it’s actually, you know, doing all the
0:08:45 things to get to this like, aha, where you’re like, oh, I get this product, it’s for me.
0:08:49 And once you get that, then they’re kind of, then you have the opportunity to keep them
0:08:51 in this engaged state over time.
0:08:55 Is that really such a thing that there is like an aha moment or is it sort of like
0:08:56 accumulative?
0:08:59 A lot of the later users on Facebook came because everyone else was already there.
0:09:01 Is this only tied to new users?
0:09:05 In the case of Facebook, actually the fact that everyone was already there makes the
0:09:07 aha moment that much more powerful, right?
0:09:11 Because all your friends and family, they’re already there, your feed’s already full of
0:09:12 content.
0:09:15 And the first time that you see photos that maybe, you know, of someone that you went
0:09:16 to high school with, right?
0:09:17 Yeah.
0:09:18 That’s actually what happened to me.
0:09:20 I was so excited when I saw an old friend, right?
0:09:21 Right.
0:09:22 Yeah, exactly.
0:09:25 And so what that means is like, you get the product and then afterwards, you know, when
0:09:28 you actually are getting these push notifications or emails that are like, hey, it’s someone’s
0:09:32 birthday or you know, whatever, like you’ve internalized what that product is.
0:09:36 And you know, this moment is different for all sorts of different companies.
0:09:39 I’ve always heard this referred to as the magic number.
0:09:43 When you show up and it’s a blank slate, it’s like, what is this about?
0:09:49 But they would drive you maniacally to, you know, follow people because that when you
0:09:54 got to their magic number where they had statistically correlated the number of followers with long
0:09:59 term engagement and retention, they would kill you to get you there doing what felt
0:10:04 like a natural acts of like, yeah, you log on with follow and you say no and say, yes.
0:10:10 But when they got you there, it kicked in and the service then quote unquote worked for
0:10:11 you.
0:10:13 A lot of the entrepreneurs I work with are trying to figure out what is my magic moment
0:10:16 that then creates the awareness of the value problem.
0:10:18 So take the example of Pinterest.
0:10:21 Pinterest when it goes to a new market, first of all, they figure out they need a lot of
0:10:23 local content to make it compelling to local users.
0:10:29 The U.S. corpus of images doesn’t necessarily is helpful in international markets but isn’t
0:10:30 sufficient.
0:10:31 You’re right.
0:10:35 Like, sorry, I don’t only want like skirts, you may not be able to wear in certain regions.
0:10:36 Exactly.
0:10:39 I haven’t worn a sorry in North America at a long time.
0:10:44 But then once you have the content set, then you have to get compelling information to
0:10:47 that individual in front of them, which, you know, you don’t know the individual when
0:10:48 they walk in the door.
0:10:52 The faster they do that, the more quickly, the better the business performs, engagement
0:10:54 goes up, retention goes up and it works.
0:10:58 So different entrepreneurs had to figure out what is that?
0:11:01 What experience do they want to deliver where people get it?
0:11:03 And then how do you engineer your way into delivering it?
0:11:04 Okay.
0:11:08 So they’ve kind of come up through acquisition and you’ve gotten new users.
0:11:09 They get the product.
0:11:13 You even have hopefully a way to measure that and see and track it over time.
0:11:17 Do you want to then go into trying to get different users?
0:11:18 Do you take your existing users?
0:11:22 One of the things that we covered very early on is that with SaaS, you always want to try
0:11:28 to take existing users and upsell them because it’s way more expensive to acquire a new customer
0:11:29 in that context.
0:11:30 I mean, of course, you want to grow your customers.
0:11:32 How does this play out in this context?
0:11:33 What happens next?
0:11:34 A lot of companies is a progression.
0:11:38 So almost all the early activity in the company is, okay, how do I get the users?
0:11:44 As you get users, you get more and more leverage from efforts at activation and retention and
0:11:45 engagement.
0:11:46 So, use Pinterest as an example again.
0:11:50 A very high percentage of women in America have downloaded Pinterest.
0:11:56 Then the leverage quickly goes into, okay, how do I keep them engaged, reactivates the
0:12:02 one who disappears and their acquisition efforts in the U.S. get de-emphasized and all the
0:12:06 leverage is there, except as they’re going international, they’re still in that acquisition
0:12:07 part of the curve.
0:12:11 So, I think the leverage changes over time based on the situation in the company.
0:12:16 Facebook hasn’t had any users in the U.S. forever because they have them all.
0:12:19 This kind of goes back to this portfolio approach to thinking about your users.
0:12:25 Once you have an active base of users and customers, what starts to get really interesting
0:12:29 is to really analyze what are the things that actually set that group up to be successful,
0:12:33 really long-term sticky users versus what are the behaviors and profiles where users
0:12:34 aren’t successful, right?
0:12:38 You actually throw your data science team on it to figure out all the different weights
0:12:44 for both behavioral as well as the demographic and profile-based stuff on there.
0:12:48 So, one of the first things that you figure out is that each one of these products actually
0:12:55 has this ladder of engagement where oftentimes new users show up to do something that’s
0:13:00 valuable but potentially infrequent, and you need to actually level them up to something
0:13:02 that happens all the time.
0:13:06 For example, when you first install Dropbox, the easiest thing that you can do is you can
0:13:10 use it to just sync your home and your work computers, right?
0:13:15 And that’s great, but really the way to get those users to become really valuable is for
0:13:18 them to start sharing folders at work with their colleagues.
0:13:22 Because once they have that and people are dragging files in, they’re really starting
0:13:25 to collaborate on things, that’s like the next level of value that you can actually
0:13:30 have on a daily basis versus this thing that is in the background that’s just syncing
0:13:31 your storage.
0:13:34 So, what are some of the things that people can then do to move those users up that ladder
0:13:35 of engagement?
0:13:40 Step one is really segmenting your users into this kind of engagement map.
0:13:45 Oftentimes, you’ll see this visualized as a kind of state machine where you have folks
0:13:48 that are new, you have folks that are casual, and you want to track how much they’re moving
0:13:51 up or down in each one of these steps.
0:13:53 And then once you have that, then the question is, okay, well, great, how do you actually
0:13:55 get them to move from one place to the other?
0:13:57 First, there’s content and education.
0:14:01 They need to know in context that they can actually do something.
0:14:06 So, for example, if you can get your users to set their home and work for a transportation
0:14:11 product, then you can maybe figure out, okay, should I prompt them in the morning to try
0:14:13 their ride based on what the ETAs are, right?
0:14:15 Like, in the app, there would be some kind of notification.
0:14:17 Like, life cycle messaging factor in there.
0:14:21 The second is, of course, if your product has some kind of monetary component, then you
0:14:23 can use incentives 10 bucks off.
0:14:27 Your next subscription, if you do this behavior that we know for sure, gets you kind of to
0:14:28 the next step.
0:14:32 And then the third thing is really just refining the product for that particular use case.
0:14:36 But maybe there are certain kinds of products that are transacted all the time, and so you
0:14:41 maybe want to waive fees or give some credits or you do something in order to get people
0:14:43 to kind of get addicted to that as a thing.
0:14:47 And a really interesting thing is the frequency with which something’s consumed.
0:14:53 I mean, eBay had enormous levels of engagement early on for a commerce app in particular.
0:14:57 People would spend hours just browsing because early on it was about collectibles and it
0:14:58 was about people’s passion.
0:15:04 So if someone’s passionate about depression or glass, they will spend hours if you give
0:15:08 them that depth of content to say, “Oh my God, I just found the perfect item.”
0:15:11 Open table and Airbnb are both typically much more episodic.
0:15:14 Most people don’t dine at fine dining restaurants with high frequency.
0:15:18 Our median user dined twice a year on open table.
0:15:22 And so that has completely different marketing implications and user implications.
0:15:26 Measurement’s probably even more important in the engagement retention thing because
0:15:32 we got our data scientists to understand the different consumer journeys through our product.
0:15:38 And then we tried to develop tactics to accelerate the journeys we wanted and limit the journeys
0:15:39 we don’t want.
0:15:44 But in order to develop your strategy, you really need to understand how your users are
0:15:47 behaving at a really refined level.
0:15:49 So what are some of the engagement metrics?
0:15:51 One really important area is frequency.
0:15:55 Just how often are you using the product regardless of the intensity and the length of the sessions
0:15:58 and all that other stuff, you know, just literally just frequency of sessions.
0:16:04 We might often ask for a daily active user divided by monthly active user ratio.
0:16:07 And that gives you a sense for like how many days is a user active.
0:16:08 Doubt them out.
0:16:13 You recently put a post out on the DAU/MAU metric.
0:16:17 And when it works and when it doesn’t, there’s a lot of nuances around when to apply it and
0:16:18 when not to.
0:16:22 DAU/MAU was very much popularized by the fact that Facebook used it, including in their
0:16:24 public financial statements.
0:16:27 And it really makes sense for them because they’re an advertising business and it matters
0:16:30 a lot that people use them a lot all the time, right?
0:16:31 Right.
0:16:34 It’s like counting impressions and being able to sell that to advertisers, right?
0:16:35 Exactly.
0:16:39 And they have historically been 60% plus daily actives over monthly actives.
0:16:40 And that’s very high.
0:16:44 You know, you’re using it more than half the days in a month.
0:16:48 On the flip side, what I was talking about in my essay about this is that DAU/MAU can
0:16:51 tell you if something’s really high frequency and if it’s working.
0:16:55 But a lot of times products are actually lower DAU/MAU for a very good reason because they’re
0:16:58 sort of just a natural cadence, you know, to the product.
0:17:02 Like you’re not going to get somebody who is using a travel product to use it more than
0:17:04 a couple of times per year.
0:17:07 And yet there are many valuable travel companies, obviously.
0:17:09 So you’re saying don’t live and die by that alone.
0:17:10 Exactly, right.
0:17:13 Because it really depends on the product you have, the type of nature of use that it has,
0:17:14 et cetera.
0:17:18 You just want to make sure that the metric reflects whatever strategy that you’re putting
0:17:19 in place.
0:17:22 If you think that your product is a daily use product and you’re going to monetize using,
0:17:25 you know, a little bit of money that you’re making over a long period of time, but your
0:17:30 DAU/MAU is low, is like sub 15%, then like it’s probably not going to work.
0:17:35 And then a metric called L28, which is something else that was pioneered certainly early at
0:17:36 Facebook.
0:17:37 And it’s a histogram.
0:17:38 And what you want to do is…
0:17:40 And a histogram is a frequency diagram.
0:17:41 Right.
0:17:45 A frequency diagram that basically says, okay, you know, show a bar showing how many users
0:17:50 have visited once in that month, then twice in the month, and then three times in the
0:17:52 month, and then four times in the month, and you kind of build that all the way out to
0:17:53 28 days.
0:17:55 Because there’s 28 days in the month on average, right?
0:17:57 And the 28 days is to remove seasonality.
0:18:00 And then related one obviously is like L7, right?
0:18:01 So just like last seven days.
0:18:02 And so what you want to see…
0:18:04 You don’t want to see wows, weekly, active users.
0:18:05 Is that really a thing, by the way?
0:18:06 Am I just making that up?
0:18:07 Right.
0:18:08 Yeah, wow.
0:18:09 That was over a while ago.
0:18:10 You just coined it.
0:18:11 No, great.
0:18:12 I’m not reclaiming retainment.
0:18:13 Why not?
0:18:14 Right.
0:18:17 And so the idea with, you know, an L28 or an L7 is the idea that you should actually
0:18:23 start to see when you graph this out that there’s a group of people who just use it 28
0:18:25 days out of 28 days, right?
0:18:29 And that there’s a big group of people who use it 27 days out of 28 days, right?
0:18:30 And that there’s a big cluster.
0:18:33 And so that’s how you know that you actually have a hardcore segment.
0:18:38 And that’s really important because in all these products you typically have a core part
0:18:41 of the network that’s driving the rest of it, whether that’s power sellers or power
0:18:45 buyers or, you know, in a social network, the creators versus the consumers.
0:18:49 Actually, I’ve heard this referred to as the smile because the one use is always pretty
0:18:50 big.
0:18:51 A lot of people show up once.
0:18:53 I don’t understand what this is and disappear.
0:18:54 So that’s the one.
0:18:57 And then they typically slide down more people use it.
0:19:03 Fewer people use it two days in one, three days in two, done right, it starts to increase
0:19:04 at the end.
0:19:06 So you basically get a smile, you just go down.
0:19:08 And I mean, that’s really powerful.
0:19:10 Facebook had a smile.
0:19:12 WhatsApp had a smile.
0:19:13 Instagram had a smile.
0:19:18 If you take a step back, it’s a precondition for investing in a venture business essentially
0:19:19 that there’s growth.
0:19:24 If it didn’t market and then you want to see growth, but growth by itself is not sufficient.
0:19:26 Investors love engagement.
0:19:31 So Pinterest, the key driver of Pinterest, it was growing, but the users were using it
0:19:32 maniacally.
0:19:33 Oh my God.
0:19:36 I think I spent an entire Thanksgiving using Pinterest.
0:19:41 Was the engagement that blew my mind much more than the growth offer up has engagement
0:19:45 that’s similar to social sites like Instagram and Snap.
0:19:50 I mean, a commerce site, mobile classifieds, people just sit there in troll looking for
0:19:55 marketers.
0:19:59 So Datamow, Smile, all these metrics are so core to us because a big engaged audience
0:20:01 is so rare.
0:20:04 And as a result, it’s almost always incredibly valuable.
0:20:07 And the engagement ends up being very related to acquisition.
0:20:11 Because when you look at all the different acquisition loops, whether it’s paid marketing
0:20:15 or a viral loop or whatever, all of those things are actually powered by engagement
0:20:16 ultimately.
0:20:21 We need people to get excited about a product in order to share content off of that platform
0:20:24 to other platforms in order to get a viral loop going.
0:20:30 And so one of the things I was going to also add on Daumau and L28 is that they’re actually
0:20:31 really hard to game, right?
0:20:32 Which is fascinating.
0:20:36 Whereas growth can be very easy to game.
0:20:37 Right.
0:20:38 Exactly.
0:20:39 Yeah.
0:20:40 Why is that?
0:20:41 What’s the difference?
0:20:42 The typical approach is to say, well, I’m going to add an email notifications.
0:20:43 I’m going to do more push notifications.
0:20:44 I’m going to do more of this and that.
0:20:48 And then automatically, you know, these metrics ought to go up, right?
0:20:53 The challenging thing is actually usually sending out more notifications will actually
0:20:56 cause more of your casual users to show up because your hardcore users were already kind
0:20:58 of showing up, you know, already.
0:21:03 And what that does is that’ll increase your monthly actives number, but actually not
0:21:05 increase your daily actives as much.
0:21:10 So I’ve actually seen cases where sending out more email decreases your Daumau as opposed
0:21:11 to increasing it.
0:21:12 That’s really interesting.
0:21:16 When you think about this portfolio of metrics, it really tells you a story about people are
0:21:18 kind of coming but not really staying.
0:21:22 If you get an email or push notification every day, eventually you turn them off and then
0:21:23 you just stop.
0:21:27 So then you get counted as a male for that period of time and then, you know, you lose
0:21:28 them as it out.
0:21:31 Acquisition is super easy to game because you can just spend money.
0:21:35 Or you got a distribution hack that works early on in the Facebook platform.
0:21:40 Companies literally got to a million users and it felt like minutes just because there
0:21:46 were so many people on Facebook and the ones who were early just got exploding user bases.
0:21:50 There were a number of concepts whose mean number of visits was one.
0:21:51 They never came back.
0:21:56 So you get to see these incredibly seductive growth curves, but our job is essentially
0:22:01 to be cynical and just say, okay, we need to go below that because there are a lot of
0:22:03 talented growth hackers who can drive growth.
0:22:08 I looked at a number of businesses that tens of millions of users and no one ever came
0:22:09 back.
0:22:11 Engagement is so, so key.
0:22:14 So we’ve talked especially about the fact that growth and network effects are not the
0:22:18 exact same thing because network effects by definition are that a network becomes more
0:22:21 valuable the more users that use it.
0:22:23 What happens on the engagement side with network effects?
0:22:26 What are the things we should be thinking about in that context?
0:22:30 Typically network effects, if they’re real, manifest in data.
0:22:33 Things like the cohort curves improve over time.
0:22:35 Usually there’s a decay with network effects.
0:22:39 There often is a reversal where they’re growing because it’s more valuable.
0:22:40 Their smile essentially.
0:22:44 My diligence at OpenTable was I looked at San Francisco, which was their first market
0:22:50 and sales rep productivity grew over time, restaurant churn decreased over time.
0:22:52 The number of diners per restaurant increased over time.
0:22:56 The percentage that booked through OpenTable versus the restaurant’s own website moved
0:23:00 towards OpenTable dramatically, every metric improved.
0:23:04 And so, you know, that’s where it both drives good engagement, but also it just improves
0:23:05 the investment basis.
0:23:06 The value overall, right?
0:23:11 The interesting points about network effects is that we often talk about it as if it’s
0:23:12 a binary thing.
0:23:13 Right.
0:23:14 Or homogenous.
0:23:15 Like all network effects are equal when they’re not.
0:23:16 Exactly, right.
0:23:18 When you look at the data, what you really figure out is network effect is actually like
0:23:22 a curve and it’s not like a binary yes/no kind of thing.
0:23:27 So, you know, for example, I would guess that if you plotted the number, if you took a bunch
0:23:32 of cities, every city was a data point and you graphed on one side the number of restaurants
0:23:36 in the city versus the conversion rate for that city.
0:23:39 You would quickly find that when cities have more restaurants, the conversion rate is higher,
0:23:40 right?
0:23:41 I’m just guessing.
0:23:43 It’s almost perfect with one refinement.
0:23:47 The number of restaurants you have is a percent of that market’s restaurant universe.
0:23:48 Okay, right.
0:23:50 Because having a hundred restaurants in Des Moines is different than having a hundred
0:23:51 restaurants in Manhattan.
0:23:52 Makes total sense.
0:23:57 So, not only that, what you then quickly figure out is that there’s some kind of a diminishing
0:24:00 effect to these things often in many cases.
0:24:05 So, for example, in rideshare, if you are going to get a car called 15 minutes versus
0:24:07 10 minutes, that’s very meaningful.
0:24:11 But if it’s, you know, five minutes versus two minutes, your conversion rate doesn’t
0:24:12 actually go up.
0:24:16 If you can express your network effect in this kind of a manner, then what you can start
0:24:21 to show is, okay, yeah, we have a couple new investment markets that maybe, you know,
0:24:24 don’t have as many restaurants or don’t have as many cars.
0:24:29 But if we put money into them and invest in them and build the right products, et cetera,
0:24:30 then you can grow.
0:24:34 You can do this kind of same analysis, whether you’re talking about, you know, YouTube channels
0:24:37 and the number of subscribers you might have.
0:24:38 Having more videos is better.
0:24:39 I’m sure you can show that.
0:24:43 If you go into the workplace and you start thinking about collaboration tools, then what
0:24:50 you ought to see is that as more people use your chat platform or your collaborative document
0:24:53 editing platform, then the engagement on that is going to be higher.
0:24:56 And you should be able to show that in the data by comparing a whole bunch of different
0:24:57 teams.
0:25:01 Okay, so we’ve talked about engagement and also how it applies to network effects.
0:25:04 Are engagement and retention the same thing?
0:25:06 I mean, in all honesty, they sound like they would be the same thing.
0:25:07 There’s overlap.
0:25:08 Yeah, there’s overlap.
0:25:09 Yeah.
0:25:10 Just to give a couple examples.
0:25:14 So weather is low frequency, but high retention because like, you’re actually going to need
0:25:15 to know what the weather is.
0:25:16 Only once a day.
0:25:17 You know, yeah.
0:25:19 Unless you live in San Francisco, you got to check it like 20 times a day with all the
0:25:20 network.
0:25:21 Right, exactly.
0:25:22 Or if you live down here, you have to check it twice a year.
0:25:23 That’s true.
0:25:24 It’s actually the same year around.
0:25:28 That’s actually what it showed was actually more that generally people didn’t really check
0:25:29 it that often.
0:25:33 However, we were highly likely to have it installed and running after 90 days because
0:25:34 that’s, it’s a reference thing.
0:25:35 It’s so important.
0:25:36 Yeah.
0:25:41 Versus if you look at something like games or ebooks or, you know, those kinds of products,
0:25:45 like really high engagement because you’re like, all right, I’m going to get to, I’m
0:25:48 going to finish this like trashy science fiction novel that I’ve been reading.
0:25:50 I’m just going to like get to it.
0:25:53 But then as soon as you’re done, you’re like, okay, there’s no reason why I would go back
0:25:54 and read it again.
0:25:58 So the real difference is that engagement obviously varies depending on the product.
0:26:02 The type of thing it is, whether it’s a weather or ebook and retention is, are you still using
0:26:04 it after X amount of time?
0:26:06 And different companies have different cadences.
0:26:12 If the average users twice a year retention is, did they book annually other businesses
0:26:13 or did they come daily?
0:26:17 But the model behind retention is completely different and the model behind engagement
0:26:18 is completely different.
0:26:19 Right.
0:26:22 The chart that I’d love to really see is one that was a bunch of different categories
0:26:26 that’s retention versus frequency versus monetization.
0:26:30 And I think you got to be like really good at least on one of those axes.
0:26:32 So we’ve done sort of this taxonomy of metrics.
0:26:34 We’ve talked about the acquisition metrics.
0:26:38 We’ve talked about some engagement metrics, primarily frequency and engagement.
0:26:42 It’s also time, not just how frequent someone is, but just how much time do they spend,
0:26:48 time spent on site, on the piece, writing comments, not just because I mean, the number
0:26:52 of businesses that have great engagement is not high because they’re only so many minutes
0:26:53 in the day.
0:26:58 And so you’re just looking for where, okay, they’re just passing time and enjoying it.
0:27:01 And they both have obvious monetization associated with that behavior.
0:27:04 This is why Netflix is so freaking genius because when they literally invented the format
0:27:08 of binge watching, which you could not do, which is, I love it because it’s a very internet
0:27:09 native concept.
0:27:11 I mean, they’ve literally fucked up everyone else’s engagement numbers.
0:27:16 I think that’s one of the narratives on why building consumer products is much harder
0:27:17 these days.
0:27:18 And do you think it’s true or not?
0:27:21 Well, because it used to be that you were, you know, what kind of time were you competing
0:27:23 for in the first couple of years of the smartphone?
0:27:26 You were competing against literally, I’m going to stare at the back of this person’s
0:27:30 head, or I can like use some cool app that I’ve downloaded, right?
0:27:34 Versus these days, you actually have to take minutes away from other products.
0:27:35 Yes.
0:27:40 And it’s typically other behemoths because the top apps are almost all done by Facebook,
0:27:45 Amazon, Google, and, you know, breaking through just, Mark calls it the first page.
0:27:50 The people who are on the first screen are just such the incumbents.
0:27:54 And sure, most people have Facebook on the screen and YouTube on the screen and Amazon
0:27:55 on the screen.
0:28:01 You know, that competition is a big overhang right now in consumer investing because you
0:28:03 have to displace someone’s minutes.
0:28:06 Yeah, I would add one more layer to that, at least on the content side, which is I think
0:28:09 a lot of people make a lot of category errors because they think they’re competing with
0:28:11 like-minded players.
0:28:14 And in fact, when it comes to things like content and attention, you’re competing with just
0:28:15 about anything that grabs your attention.
0:28:17 It’s not just other media outlets.
0:28:18 It’s Tinder.
0:28:19 It’s Tinder.
0:28:20 It’s a dating app.
0:28:21 It’s something else.
0:28:26 I’m riding on the train for an hour. I could, you know, see one of my friends on our Facebook,
0:28:27 watch videos on YouTube.
0:28:28 Yeah, that actually changes the time blocks.
0:28:31 Xerox Park did a really interesting study on micro-weighting moments, and they’re literally
0:28:35 the little snatches of time, like two seconds here and there, that you might be weighting
0:28:39 in line or doing something so you can do a lot of snack size things in that period, which
0:28:41 is also another interesting thing to think about for how people engage with your business.
0:28:45 So it’s actually funny because there’s some businesses that have good engagement where
0:28:49 it’s one session that goes on for a while, YouTube or Netflix or something like that.
0:28:55 There are others that are multiple small sessions at an aggregate because it’s the micro opportunities
0:28:56 to.
0:28:58 And Google is the best example of this, right?
0:29:03 In fact, if you spend a lot of time on Google.com, you know, refining your searches and clicking
0:29:05 around, that means actually the service is doing poorly.
0:29:06 They failed.
0:29:07 Interesting.
0:29:10 Their goal is to get you to their advertisers as fast as they can.
0:29:14 That’s a frequency play and a monetization play ultimately as opposed to an engagement
0:29:15 one.
0:29:16 Yes.
0:29:17 That’s fascinating.
0:29:18 And then some products are more on the engagement side.
0:29:20 You have to optimize it based on how you’re monetizing it.
0:29:22 What are some of the metrics for retention?
0:29:24 I mean, is it just, should I stay or should I go?
0:29:25 Is that the retention metric?
0:29:29 And I think the big thing is the concept of churn is a tricky one.
0:29:34 In some cases, like subscription, Hulu, Netflix, and then also in the SaaS world, whether or
0:29:36 not you’re still continuing to pay or not, right?
0:29:38 And that’s really obvious.
0:29:41 The thing that’s tricky for a lot of these consumer products, especially episodic ones
0:29:44 and it’s actually less whether they’ve quote unquote churned or not.
0:29:47 It’s actually just whether or not they’re active or inactive and whether or not that’s
0:29:52 happening at a rate that you in your business strategy have decided is acceptable or not.
0:29:56 If every Halloween, you know how there’s those costume stores that like open all over the
0:29:57 place.
0:30:01 If every Halloween, you go back and you buy a costume, but you’re inactive the rest
0:30:02 of the time.
0:30:03 Have you churned or not?
0:30:04 Like it’s not clear.
0:30:07 And I would argue you’ve not churned because you’re doing exactly what they want, which
0:30:09 is to buy a costume every Halloween.
0:30:12 It seems like it makes assessing the retention of a consumer business very difficult.
0:30:14 You adjust the time period that you’re relevant on it.
0:30:20 If the average diner dines twice a year, so you can apply that metric travels a similar
0:30:24 thing Airbnb is, you know, for the average user relatively infrequent, you have to tailor
0:30:26 your look to what are they trying to do.
0:30:30 So if you’re trying to stay up with your friends and you’re doing it twice a year, yeah, that
0:30:31 doesn’t work.
0:30:32 So Facebook has got a whole different center.
0:30:36 One of the things that companies can often do is to measure upstream signal.
0:30:40 So for example, Zillow, you’re probably not going to buy a house very often, right?
0:30:43 Maybe, you know, a couple of times in your life.
0:30:47 However, what’s really interesting is they can say, well, you know, maybe folks aren’t
0:30:49 buying houses, but at least are we top of mind?
0:30:52 Are they checking the houses that are going on sale in their neighborhood?
0:30:53 Are they opening up the emails?
0:30:55 Are they, you know, doing searches, right?
0:30:57 Why do you call that upstream?
0:30:58 In the funnel.
0:31:01 You’re kind of going up in the funnel and you’re tracking those metrics as opposed to, you
0:31:02 know, purchases.
0:31:06 So even, you know, for open table, it’s like, okay, great, well, maybe if you’re not actually
0:31:10 completing the reservations, are you at least checking the app for availability?
0:31:12 What’s new restaurants where I want to dine there?
0:31:16 There’s some level of content consumption throughout this entire episode.
0:31:21 There seems to be this interesting dance between architecting and discovering like you might
0:31:25 know some things upfront because you’re trying to be intentional and build these things.
0:31:29 And then there are things that you discover along the way as your product and your views
0:31:31 and your data evolves.
0:31:34 How do you advise people to sort of navigate that dance?
0:31:38 You iterate, you develop hypotheses, you put it out there and you test the hypothesis.
0:31:41 You know, I think my product is going to behave this way and then did it.
0:31:45 Probably the most important thing is for me, marketing can be art, marketing can be science.
0:31:48 In the consumer internet, it’s more science.
0:31:53 Some companies can effectively do TV campaigns, large media budgets, things like that.
0:31:58 For me, the better companies typically just rip apart their metrics, understand the dynamics
0:32:03 of their business and then figure out ways to improve the business through that knowledge.
0:32:06 And that knowledge could feed back into new product executions or new marketing strategies
0:32:08 or new something.
0:32:13 This consideration, but it’s informed by the data at a level that, you know, on the best
0:32:16 companies is really, really deep.
0:32:23 Ultimately, you have a set of strategies that you’re trying to pursue and you pick the metrics
0:32:26 to validate that you’re on the right track, right?
0:32:29 And a lot of what we’ve talked about today has really been the idea that actually there’s
0:32:32 a lot of nature versus nurture kind of parts about this.
0:32:35 You know, your product could just be located in but high monetization.
0:32:38 And so you shouldn’t look at, you know, DA/UMAU.
0:32:41 And so you have to find really the right set of metrics that show that you’re providing
0:32:46 value to your customers first and foremost, and then really, you know, build your team
0:32:50 and your product roadmap and everything in order to reinforce that.
0:32:54 Find the loops and the networks that exist within your product, because those are the
0:32:58 things that are going to keep your engagement improving over time, even in the face of competition.
0:32:59 Growth is good.
0:33:02 Growth and engagement is really, really, really good.
0:33:03 Okay.
0:33:04 So that’s fabulous.
0:33:05 Well, thank you guys for joining the A6NC podcast.