AI transcript
0:00:05 The content here is for informational purposes only, should not be taken as legal business
0:00:10 tax or investment advice or be used to evaluate any investment or security and is not directed
0:00:15 at any investors or potential investors in any A16Z fund. For more details, please see
0:00:21 A16Z.com/disclosures. Hi, everyone. It’s Marketplaces Week over here. Thanks
0:00:26 to our consumer team releasing a new index of the next industry-defining marketplaces.
0:00:32 You can check that out at A16Z.com/marketplace100. But what happens as such marketplaces and other
0:00:38 platforms and products evolve over time as do their users? The conversation that follows
0:00:41 is one of our more popular episodes from a couple years ago, featuring general partners
0:00:47 Andrew Chen and Jeff Jordan in conversation with me on how after you acquire users, which
0:00:52 we covered in another episode, then how do you keep them engaged, retain them, and even
0:00:58 resurrect or re-engage them and what are the key metrics? But first, we began with what
0:01:02 happens after the initial acquisition as different kinds of users join a product or
0:01:07 platform over time. What does that mean for engagement and where do cohort analyses come
0:01:08 in?
0:01:12 One of the things that you see is that people end up using these products very differently
0:01:17 because the kinds of users that you’re getting are changing over time. When you look at something
0:01:24 like rideshare, all the early cohorts are basically people in urban areas. These days,
0:01:29 all of rideshare is more like suburban or rural folks because you’ve saturated all of
0:01:36 the center. What you tend to see is as you acquire your core demographic out, that actually
0:01:42 ends up showing up in the engagement. Going back to natural gravity to the whole thing,
0:01:46 because gravity also hits the engagement side of things as well, and then ultimately the
0:01:52 LTV because your users are typically getting less valuable. It may take years to see this
0:01:56 kind of play out, but that’s kind of the natural law of things.
0:01:59 There is a progression in these and particularly the ones that are really successful. Early
0:02:04 on, it’s all about getting users. Just like users, users, users. If you’re wildly successful
0:02:10 at doing that, you run out of users or you start running low on users and you have to
0:02:15 go to engagement. Pinterest has a very high quality problem right now. Most women in
0:02:18 America have downloaded the Pinterest app.
0:02:20 Oh yeah, I’ve had it for years.
0:02:24 Some growth can come through, okay, there are some number of women who’ve never heard of
0:02:28 Pinterest somewhere in the country, but much more so, they need to engage and re-engage
0:02:33 the existing audience. I mean, we love engagement from an investor standpoint because it’s just
0:02:34 …
0:02:36 It shows stickiness.
0:02:40 You can often hack your way into new users. It’s really hard to hack your way into true
0:02:41 engagement.
0:02:42 Keeping them.
0:02:47 If someone’s spending 20 minutes a day on your site, Pinterest and OfferUp, the major
0:02:53 investment thesis was, “Oh my God, look at that engagement kind of thing.” If they can
0:02:55 scale the user base, it’s a beautiful deal.
0:02:59 What we mean by engagement is actually interacting with them and seeing their activity because
0:03:05 to Andrew’s three points of acquisition engagement retention, the third piece is keeping them.
0:03:10 The way that we’ll often analyze this is looking at cohort analyses where we’ll look at each
0:03:15 batch of users that’s joining in each week and really start to dissect, “Well, how active
0:03:16 are they really?”
0:03:22 To compare all these cohorts over time, you’re basically putting the users that come in from
0:03:27 a particular time frame, let’s say it’s a week, and you’re putting them into a bucket.
0:03:31 What you’re doing is you want to compare all of these different buckets against each other.
0:03:36 What you typically do is you look at a bucket of a cohort of users and you say, “Okay, well,
0:03:40 once they’ve signed up, the week after, how active are they? What about the week after
0:03:44 that and the week after that and you can build out a curve?” It just turns out that these
0:03:49 curves, once you’ve looked at enough of them, surprisingly, human nature, they all look
0:03:50 kind of the same.
0:03:51 I bet.
0:03:55 They kind of all kind of curve down and for the good ones, they start to flatten out and
0:03:59 they plateau and then for the really good ones, they’ll actually swing back up and people
0:04:01 will come back to the surface.
0:04:06 What you want to do is you want to compare the various cohorts against each other in
0:04:13 time to see if you can spot any trends on how the usage patterns are increasing or decreasing.
0:04:18 When you add a new layer to layer cake, you might unlock a bunch of new behavior.
0:04:22 You might unlock a bunch of new frequency that didn’t exist before or alternatively,
0:04:27 over long thresholds of time, people tend to become less active as you move out of your
0:04:28 course.
0:04:29 The cohort graduates.
0:04:33 If not, a specific cohort of users flattens out is really important because if you’re
0:04:37 in a world where they slowly degrades and then all of a sudden it’ll actually go to
0:04:42 zero, that means that you’re naturally always filling up the bucket.
0:04:43 You kind of have a leaky bucket.
0:04:44 You’re constantly filling it out.
0:04:45 You’re always filling it up.
0:04:46 Right?
0:04:51 What happens is that gets progressively harder if you want to keep your overall growth rate
0:04:57 because that means you have to double, triple, quadruple your acquisition in order to counteract
0:04:58 for that.
0:05:03 That growth accounting equation that’s often thrown around is that your net MAUs, so your
0:05:09 net monthly active users, equals all the new people that you’re acquiring, right, minus
0:05:14 all the people that are churned, right, and then plus all the people that you’re resurrecting,
0:05:15 you know.
0:05:16 Reengaging.
0:05:17 Reengaging, exactly.
0:05:18 Right.
0:05:19 That are coming back after they’ve churned.
0:05:23 And so what happens is for a new startup, you are completely focused on new users because
0:05:26 you don’t really have that many users to churn.
0:05:30 And over time, as you get bigger and bigger and bigger, what you find is that your churn
0:05:33 rate starts to, you know, it’s a percentage of your actives.
0:05:37 And so, you know, the evolution of most of these companies as they’re getting bigger
0:05:43 tends to start with acquisition, then focus much more on, you know, churn and retention,
0:05:47 and then ultimately also to layer in resurrection as well.
0:05:50 And the cohort curves have a couple other features that I love.
0:05:56 And usually in marketplace businesses, the best models are built off of the cohort curves.
0:05:57 Oh, interesting.
0:06:00 Because you have to understand that degradation and where it goes.
0:06:03 Using cohorts really give you a sense of, are there network effects?
0:06:07 And network effect is the business gets more valuable, the more users that use it.
0:06:12 If it gets more valuable, your newer cohorts should behave better than your early cohorts.
0:06:13 Why is that?
0:06:16 Because the service is more valuable given how many there are.
0:06:17 Oh, interesting.
0:06:18 So that’s kind of a tip.
0:06:19 I get that.
0:06:22 If you’re going to get more restaurants, you’re going to get a whole lot more reservations
0:06:25 per diner because you are serving more than you need.
0:06:30 So the open table cohorts would climb up and get more attractive over time.
0:06:34 Versus, you know, we talked about typically they tended to grade over time.
0:06:38 If you’ve reversed the polarity and they’re growing over time, it means you’ve made the
0:06:41 business more valuable and then you start projecting forward.
0:06:44 Okay, what a better way to know the business is actually more valuable than thinking is
0:06:46 valuable and bleeding your own mind.
0:06:50 And in network effects business, we always ask, show us the cohorts.
0:06:52 Everyone asserts, I’m a network effect.
0:06:53 I’m a network effect.
0:07:00 But when you say show me the data, cohort curves, or they don’t lie, they don’t lie.
0:07:02 The other really interesting part is segmenting it.
0:07:04 I was about to actually ask you what are the buckets of cohorts?
0:07:06 Are they all demographic data?
0:07:10 For a bunch of hyper-local type businesses, the reason why segmenting it based on market
0:07:15 geography, why that’s so valuable is because then you can compare markets against each
0:07:16 other.
0:07:19 You can say, well, you know, this market which has much more density in terms of the number
0:07:23 of scooters behaves like this, and you can start to draw conclusions, you know, sort
0:07:26 of a natural A/B test in order to do that.
0:07:30 And I think the similar kind of analysis you can do for B2B companies is for products that
0:07:33 have different size teams using it.
0:07:37 If you have a really large team that are all using a product, well, are they all using
0:07:39 the product more as a result?
0:07:42 And let’s compare that to something that maybe only has a couple, right?
0:07:45 And so this way, you can start to kind of disassemble, you know, the structure of these
0:07:48 networks and do they actually lead to higher engagement?
0:07:50 Slack would be a perfect example of that.
0:07:54 You know, just if you have five people in the organization using Slack, you get one use
0:07:55 curve.
0:07:58 If you have, you know, if the organization, it’s the operating system for the organization,
0:07:59 you have a very different curve.
0:08:03 Though it’s not just an accident, you have to sort of architect it, not just expect like
0:08:05 serendipity to fall into place.
0:08:10 So after you get the new users, the way that you have to think about it is around quality,
0:08:11 right?
0:08:14 And you have to make sure that the new users turn into engaged users.
0:08:17 One of the things people often talk about is just sort of this idea of like an aha
0:08:22 moment or a magic moment where the user really understands the true value of the product.
0:08:23 But often that involves a bunch of setup.
0:08:27 So for example, for all the different social products, whether that’s Twitter or Facebook
0:08:31 or Pinterest, et cetera, you have to make sure that when you first bring a new user
0:08:36 and they have to follow all the right people, they have to get the onboarding experience.
0:08:39 Which by the way, isn’t just signing up, but it’s actually, you know, doing all the
0:08:45 things to get to this like, aha, where you’re like, oh, I get this product, it’s for me.
0:08:49 And once you get that, then they’re kind of, then you have the opportunity to keep them
0:08:51 in this engaged state over time.
0:08:55 Is that really such a thing that there is like an aha moment or is it sort of like
0:08:56 accumulative?
0:08:59 A lot of the later users on Facebook came because everyone else was already there.
0:09:01 Is this only tied to new users?
0:09:05 In the case of Facebook, actually the fact that everyone was already there makes the
0:09:07 aha moment that much more powerful, right?
0:09:11 Because all your friends and family, they’re already there, your feed’s already full of
0:09:12 content.
0:09:15 And the first time that you see photos that maybe, you know, of someone that you went
0:09:16 to high school with, right?
0:09:17 Yeah.
0:09:18 That’s actually what happened to me.
0:09:20 I was so excited when I saw an old friend, right?
0:09:21 Right.
0:09:22 Yeah, exactly.
0:09:25 And so what that means is like, you get the product and then afterwards, you know, when
0:09:28 you actually are getting these push notifications or emails that are like, hey, it’s someone’s
0:09:32 birthday or you know, whatever, like you’ve internalized what that product is.
0:09:36 And you know, this moment is different for all sorts of different companies.
0:09:39 I’ve always heard this referred to as the magic number.
0:09:43 When you show up and it’s a blank slate, it’s like, what is this about?
0:09:49 But they would drive you maniacally to, you know, follow people because that when you
0:09:54 got to their magic number where they had statistically correlated the number of followers with long
0:09:59 term engagement and retention, they would kill you to get you there doing what felt
0:10:04 like a natural acts of like, yeah, you log on with follow and you say no and say, yes.
0:10:10 But when they got you there, it kicked in and the service then quote unquote worked for
0:10:11 you.
0:10:13 A lot of the entrepreneurs I work with are trying to figure out what is my magic moment
0:10:16 that then creates the awareness of the value problem.
0:10:18 So take the example of Pinterest.
0:10:21 Pinterest when it goes to a new market, first of all, they figure out they need a lot of
0:10:23 local content to make it compelling to local users.
0:10:29 The U.S. corpus of images doesn’t necessarily is helpful in international markets but isn’t
0:10:30 sufficient.
0:10:31 You’re right.
0:10:35 Like, sorry, I don’t only want like skirts, you may not be able to wear in certain regions.
0:10:36 Exactly.
0:10:39 I haven’t worn a sorry in North America at a long time.
0:10:44 But then once you have the content set, then you have to get compelling information to
0:10:47 that individual in front of them, which, you know, you don’t know the individual when
0:10:48 they walk in the door.
0:10:52 The faster they do that, the more quickly, the better the business performs, engagement
0:10:54 goes up, retention goes up and it works.
0:10:58 So different entrepreneurs had to figure out what is that?
0:11:01 What experience do they want to deliver where people get it?
0:11:03 And then how do you engineer your way into delivering it?
0:11:04 Okay.
0:11:08 So they’ve kind of come up through acquisition and you’ve gotten new users.
0:11:09 They get the product.
0:11:13 You even have hopefully a way to measure that and see and track it over time.
0:11:17 Do you want to then go into trying to get different users?
0:11:18 Do you take your existing users?
0:11:22 One of the things that we covered very early on is that with SaaS, you always want to try
0:11:28 to take existing users and upsell them because it’s way more expensive to acquire a new customer
0:11:29 in that context.
0:11:30 I mean, of course, you want to grow your customers.
0:11:32 How does this play out in this context?
0:11:33 What happens next?
0:11:34 A lot of companies is a progression.
0:11:38 So almost all the early activity in the company is, okay, how do I get the users?
0:11:44 As you get users, you get more and more leverage from efforts at activation and retention and
0:11:45 engagement.
0:11:46 So, use Pinterest as an example again.
0:11:50 A very high percentage of women in America have downloaded Pinterest.
0:11:56 Then the leverage quickly goes into, okay, how do I keep them engaged, reactivates the
0:12:02 one who disappears and their acquisition efforts in the U.S. get de-emphasized and all the
0:12:06 leverage is there, except as they’re going international, they’re still in that acquisition
0:12:07 part of the curve.
0:12:11 So, I think the leverage changes over time based on the situation in the company.
0:12:16 Facebook hasn’t had any users in the U.S. forever because they have them all.
0:12:19 This kind of goes back to this portfolio approach to thinking about your users.
0:12:25 Once you have an active base of users and customers, what starts to get really interesting
0:12:29 is to really analyze what are the things that actually set that group up to be successful,
0:12:33 really long-term sticky users versus what are the behaviors and profiles where users
0:12:34 aren’t successful, right?
0:12:38 You actually throw your data science team on it to figure out all the different weights
0:12:44 for both behavioral as well as the demographic and profile-based stuff on there.
0:12:48 So, one of the first things that you figure out is that each one of these products actually
0:12:55 has this ladder of engagement where oftentimes new users show up to do something that’s
0:13:00 valuable but potentially infrequent, and you need to actually level them up to something
0:13:02 that happens all the time.
0:13:06 For example, when you first install Dropbox, the easiest thing that you can do is you can
0:13:10 use it to just sync your home and your work computers, right?
0:13:15 And that’s great, but really the way to get those users to become really valuable is for
0:13:18 them to start sharing folders at work with their colleagues.
0:13:22 Because once they have that and people are dragging files in, they’re really starting
0:13:25 to collaborate on things, that’s like the next level of value that you can actually
0:13:30 have on a daily basis versus this thing that is in the background that’s just syncing
0:13:31 your storage.
0:13:34 So, what are some of the things that people can then do to move those users up that ladder
0:13:35 of engagement?
0:13:40 Step one is really segmenting your users into this kind of engagement map.
0:13:45 Oftentimes, you’ll see this visualized as a kind of state machine where you have folks
0:13:48 that are new, you have folks that are casual, and you want to track how much they’re moving
0:13:51 up or down in each one of these steps.
0:13:53 And then once you have that, then the question is, okay, well, great, how do you actually
0:13:55 get them to move from one place to the other?
0:13:57 First, there’s content and education.
0:14:01 They need to know in context that they can actually do something.
0:14:06 So, for example, if you can get your users to set their home and work for a transportation
0:14:11 product, then you can maybe figure out, okay, should I prompt them in the morning to try
0:14:13 their ride based on what the ETAs are, right?
0:14:15 Like, in the app, there would be some kind of notification.
0:14:17 Like, life cycle messaging factor in there.
0:14:21 The second is, of course, if your product has some kind of monetary component, then you
0:14:23 can use incentives 10 bucks off.
0:14:27 Your next subscription, if you do this behavior that we know for sure, gets you kind of to
0:14:28 the next step.
0:14:32 And then the third thing is really just refining the product for that particular use case.
0:14:36 But maybe there are certain kinds of products that are transacted all the time, and so you
0:14:41 maybe want to waive fees or give some credits or you do something in order to get people
0:14:43 to kind of get addicted to that as a thing.
0:14:47 And a really interesting thing is the frequency with which something’s consumed.
0:14:53 I mean, eBay had enormous levels of engagement early on for a commerce app in particular.
0:14:57 People would spend hours just browsing because early on it was about collectibles and it
0:14:58 was about people’s passion.
0:15:04 So if someone’s passionate about depression or glass, they will spend hours if you give
0:15:08 them that depth of content to say, “Oh my God, I just found the perfect item.”
0:15:11 Open table and Airbnb are both typically much more episodic.
0:15:14 Most people don’t dine at fine dining restaurants with high frequency.
0:15:18 Our median user dined twice a year on open table.
0:15:22 And so that has completely different marketing implications and user implications.
0:15:26 Measurement’s probably even more important in the engagement retention thing because
0:15:32 we got our data scientists to understand the different consumer journeys through our product.
0:15:38 And then we tried to develop tactics to accelerate the journeys we wanted and limit the journeys
0:15:39 we don’t want.
0:15:44 But in order to develop your strategy, you really need to understand how your users are
0:15:47 behaving at a really refined level.
0:15:49 So what are some of the engagement metrics?
0:15:51 One really important area is frequency.
0:15:55 Just how often are you using the product regardless of the intensity and the length of the sessions
0:15:58 and all that other stuff, you know, just literally just frequency of sessions.
0:16:04 We might often ask for a daily active user divided by monthly active user ratio.
0:16:07 And that gives you a sense for like how many days is a user active.
0:16:08 Doubt them out.
0:16:13 You recently put a post out on the DAU/MAU metric.
0:16:17 And when it works and when it doesn’t, there’s a lot of nuances around when to apply it and
0:16:18 when not to.
0:16:22 DAU/MAU was very much popularized by the fact that Facebook used it, including in their
0:16:24 public financial statements.
0:16:27 And it really makes sense for them because they’re an advertising business and it matters
0:16:30 a lot that people use them a lot all the time, right?
0:16:31 Right.
0:16:34 It’s like counting impressions and being able to sell that to advertisers, right?
0:16:35 Exactly.
0:16:39 And they have historically been 60% plus daily actives over monthly actives.
0:16:40 And that’s very high.
0:16:44 You know, you’re using it more than half the days in a month.
0:16:48 On the flip side, what I was talking about in my essay about this is that DAU/MAU can
0:16:51 tell you if something’s really high frequency and if it’s working.
0:16:55 But a lot of times products are actually lower DAU/MAU for a very good reason because they’re
0:16:58 sort of just a natural cadence, you know, to the product.
0:17:02 Like you’re not going to get somebody who is using a travel product to use it more than
0:17:04 a couple of times per year.
0:17:07 And yet there are many valuable travel companies, obviously.
0:17:09 So you’re saying don’t live and die by that alone.
0:17:10 Exactly, right.
0:17:13 Because it really depends on the product you have, the type of nature of use that it has,
0:17:14 et cetera.
0:17:18 You just want to make sure that the metric reflects whatever strategy that you’re putting
0:17:19 in place.
0:17:22 If you think that your product is a daily use product and you’re going to monetize using,
0:17:25 you know, a little bit of money that you’re making over a long period of time, but your
0:17:30 DAU/MAU is low, is like sub 15%, then like it’s probably not going to work.
0:17:35 And then a metric called L28, which is something else that was pioneered certainly early at
0:17:36 Facebook.
0:17:37 And it’s a histogram.
0:17:38 And what you want to do is…
0:17:40 And a histogram is a frequency diagram.
0:17:41 Right.
0:17:45 A frequency diagram that basically says, okay, you know, show a bar showing how many users
0:17:50 have visited once in that month, then twice in the month, and then three times in the
0:17:52 month, and then four times in the month, and you kind of build that all the way out to
0:17:53 28 days.
0:17:55 Because there’s 28 days in the month on average, right?
0:17:57 And the 28 days is to remove seasonality.
0:18:00 And then related one obviously is like L7, right?
0:18:01 So just like last seven days.
0:18:02 And so what you want to see…
0:18:04 You don’t want to see wows, weekly, active users.
0:18:05 Is that really a thing, by the way?
0:18:06 Am I just making that up?
0:18:07 Right.
0:18:08 Yeah, wow.
0:18:09 That was over a while ago.
0:18:10 You just coined it.
0:18:11 No, great.
0:18:12 I’m not reclaiming retainment.
0:18:13 Why not?
0:18:14 Right.
0:18:17 And so the idea with, you know, an L28 or an L7 is the idea that you should actually
0:18:23 start to see when you graph this out that there’s a group of people who just use it 28
0:18:25 days out of 28 days, right?
0:18:29 And that there’s a big group of people who use it 27 days out of 28 days, right?
0:18:30 And that there’s a big cluster.
0:18:33 And so that’s how you know that you actually have a hardcore segment.
0:18:38 And that’s really important because in all these products you typically have a core part
0:18:41 of the network that’s driving the rest of it, whether that’s power sellers or power
0:18:45 buyers or, you know, in a social network, the creators versus the consumers.
0:18:49 Actually, I’ve heard this referred to as the smile because the one use is always pretty
0:18:50 big.
0:18:51 A lot of people show up once.
0:18:53 I don’t understand what this is and disappear.
0:18:54 So that’s the one.
0:18:57 And then they typically slide down more people use it.
0:19:03 Fewer people use it two days in one, three days in two, done right, it starts to increase
0:19:04 at the end.
0:19:06 So you basically get a smile, you just go down.
0:19:08 And I mean, that’s really powerful.
0:19:10 Facebook had a smile.
0:19:12 WhatsApp had a smile.
0:19:13 Instagram had a smile.
0:19:18 If you take a step back, it’s a precondition for investing in a venture business essentially
0:19:19 that there’s growth.
0:19:24 If it didn’t market and then you want to see growth, but growth by itself is not sufficient.
0:19:26 Investors love engagement.
0:19:31 So Pinterest, the key driver of Pinterest, it was growing, but the users were using it
0:19:32 maniacally.
0:19:33 Oh my God.
0:19:36 I think I spent an entire Thanksgiving using Pinterest.
0:19:41 Was the engagement that blew my mind much more than the growth offer up has engagement
0:19:45 that’s similar to social sites like Instagram and Snap.
0:19:50 I mean, a commerce site, mobile classifieds, people just sit there in troll looking for
0:19:55 marketers.
0:19:59 So Datamow, Smile, all these metrics are so core to us because a big engaged audience
0:20:01 is so rare.
0:20:04 And as a result, it’s almost always incredibly valuable.
0:20:07 And the engagement ends up being very related to acquisition.
0:20:11 Because when you look at all the different acquisition loops, whether it’s paid marketing
0:20:15 or a viral loop or whatever, all of those things are actually powered by engagement
0:20:16 ultimately.
0:20:21 We need people to get excited about a product in order to share content off of that platform
0:20:24 to other platforms in order to get a viral loop going.
0:20:30 And so one of the things I was going to also add on Daumau and L28 is that they’re actually
0:20:31 really hard to game, right?
0:20:32 Which is fascinating.
0:20:36 Whereas growth can be very easy to game.
0:20:37 Right.
0:20:38 Exactly.
0:20:39 Yeah.
0:20:40 Why is that?
0:20:41 What’s the difference?
0:20:42 The typical approach is to say, well, I’m going to add an email notifications.
0:20:43 I’m going to do more push notifications.
0:20:44 I’m going to do more of this and that.
0:20:48 And then automatically, you know, these metrics ought to go up, right?
0:20:53 The challenging thing is actually usually sending out more notifications will actually
0:20:56 cause more of your casual users to show up because your hardcore users were already kind
0:20:58 of showing up, you know, already.
0:21:03 And what that does is that’ll increase your monthly actives number, but actually not
0:21:05 increase your daily actives as much.
0:21:10 So I’ve actually seen cases where sending out more email decreases your Daumau as opposed
0:21:11 to increasing it.
0:21:12 That’s really interesting.
0:21:16 When you think about this portfolio of metrics, it really tells you a story about people are
0:21:18 kind of coming but not really staying.
0:21:22 If you get an email or push notification every day, eventually you turn them off and then
0:21:23 you just stop.
0:21:27 So then you get counted as a male for that period of time and then, you know, you lose
0:21:28 them as it out.
0:21:31 Acquisition is super easy to game because you can just spend money.
0:21:35 Or you got a distribution hack that works early on in the Facebook platform.
0:21:40 Companies literally got to a million users and it felt like minutes just because there
0:21:46 were so many people on Facebook and the ones who were early just got exploding user bases.
0:21:50 There were a number of concepts whose mean number of visits was one.
0:21:51 They never came back.
0:21:56 So you get to see these incredibly seductive growth curves, but our job is essentially
0:22:01 to be cynical and just say, okay, we need to go below that because there are a lot of
0:22:03 talented growth hackers who can drive growth.
0:22:08 I looked at a number of businesses that tens of millions of users and no one ever came
0:22:09 back.
0:22:11 Engagement is so, so key.
0:22:14 So we’ve talked especially about the fact that growth and network effects are not the
0:22:18 exact same thing because network effects by definition are that a network becomes more
0:22:21 valuable the more users that use it.
0:22:23 What happens on the engagement side with network effects?
0:22:26 What are the things we should be thinking about in that context?
0:22:30 Typically network effects, if they’re real, manifest in data.
0:22:33 Things like the cohort curves improve over time.
0:22:35 Usually there’s a decay with network effects.
0:22:39 There often is a reversal where they’re growing because it’s more valuable.
0:22:40 Their smile essentially.
0:22:44 My diligence at OpenTable was I looked at San Francisco, which was their first market
0:22:50 and sales rep productivity grew over time, restaurant churn decreased over time.
0:22:52 The number of diners per restaurant increased over time.
0:22:56 The percentage that booked through OpenTable versus the restaurant’s own website moved
0:23:00 towards OpenTable dramatically, every metric improved.
0:23:04 And so, you know, that’s where it both drives good engagement, but also it just improves
0:23:05 the investment basis.
0:23:06 The value overall, right?
0:23:11 The interesting points about network effects is that we often talk about it as if it’s
0:23:12 a binary thing.
0:23:13 Right.
0:23:14 Or homogenous.
0:23:15 Like all network effects are equal when they’re not.
0:23:16 Exactly, right.
0:23:18 When you look at the data, what you really figure out is network effect is actually like
0:23:22 a curve and it’s not like a binary yes/no kind of thing.
0:23:27 So, you know, for example, I would guess that if you plotted the number, if you took a bunch
0:23:32 of cities, every city was a data point and you graphed on one side the number of restaurants
0:23:36 in the city versus the conversion rate for that city.
0:23:39 You would quickly find that when cities have more restaurants, the conversion rate is higher,
0:23:40 right?
0:23:41 I’m just guessing.
0:23:43 It’s almost perfect with one refinement.
0:23:47 The number of restaurants you have is a percent of that market’s restaurant universe.
0:23:48 Okay, right.
0:23:50 Because having a hundred restaurants in Des Moines is different than having a hundred
0:23:51 restaurants in Manhattan.
0:23:52 Makes total sense.
0:23:57 So, not only that, what you then quickly figure out is that there’s some kind of a diminishing
0:24:00 effect to these things often in many cases.
0:24:05 So, for example, in rideshare, if you are going to get a car called 15 minutes versus
0:24:07 10 minutes, that’s very meaningful.
0:24:11 But if it’s, you know, five minutes versus two minutes, your conversion rate doesn’t
0:24:12 actually go up.
0:24:16 If you can express your network effect in this kind of a manner, then what you can start
0:24:21 to show is, okay, yeah, we have a couple new investment markets that maybe, you know,
0:24:24 don’t have as many restaurants or don’t have as many cars.
0:24:29 But if we put money into them and invest in them and build the right products, et cetera,
0:24:30 then you can grow.
0:24:34 You can do this kind of same analysis, whether you’re talking about, you know, YouTube channels
0:24:37 and the number of subscribers you might have.
0:24:38 Having more videos is better.
0:24:39 I’m sure you can show that.
0:24:43 If you go into the workplace and you start thinking about collaboration tools, then what
0:24:50 you ought to see is that as more people use your chat platform or your collaborative document
0:24:53 editing platform, then the engagement on that is going to be higher.
0:24:56 And you should be able to show that in the data by comparing a whole bunch of different
0:24:57 teams.
0:25:01 Okay, so we’ve talked about engagement and also how it applies to network effects.
0:25:04 Are engagement and retention the same thing?
0:25:06 I mean, in all honesty, they sound like they would be the same thing.
0:25:07 There’s overlap.
0:25:08 Yeah, there’s overlap.
0:25:09 Yeah.
0:25:10 Just to give a couple examples.
0:25:14 So weather is low frequency, but high retention because like, you’re actually going to need
0:25:15 to know what the weather is.
0:25:16 Only once a day.
0:25:17 You know, yeah.
0:25:19 Unless you live in San Francisco, you got to check it like 20 times a day with all the
0:25:20 network.
0:25:21 Right, exactly.
0:25:22 Or if you live down here, you have to check it twice a year.
0:25:23 That’s true.
0:25:24 It’s actually the same year around.
0:25:28 That’s actually what it showed was actually more that generally people didn’t really check
0:25:29 it that often.
0:25:33 However, we were highly likely to have it installed and running after 90 days because
0:25:34 that’s, it’s a reference thing.
0:25:35 It’s so important.
0:25:36 Yeah.
0:25:41 Versus if you look at something like games or ebooks or, you know, those kinds of products,
0:25:45 like really high engagement because you’re like, all right, I’m going to get to, I’m
0:25:48 going to finish this like trashy science fiction novel that I’ve been reading.
0:25:50 I’m just going to like get to it.
0:25:53 But then as soon as you’re done, you’re like, okay, there’s no reason why I would go back
0:25:54 and read it again.
0:25:58 So the real difference is that engagement obviously varies depending on the product.
0:26:02 The type of thing it is, whether it’s a weather or ebook and retention is, are you still using
0:26:04 it after X amount of time?
0:26:06 And different companies have different cadences.
0:26:12 If the average users twice a year retention is, did they book annually other businesses
0:26:13 or did they come daily?
0:26:17 But the model behind retention is completely different and the model behind engagement
0:26:18 is completely different.
0:26:19 Right.
0:26:22 The chart that I’d love to really see is one that was a bunch of different categories
0:26:26 that’s retention versus frequency versus monetization.
0:26:30 And I think you got to be like really good at least on one of those axes.
0:26:32 So we’ve done sort of this taxonomy of metrics.
0:26:34 We’ve talked about the acquisition metrics.
0:26:38 We’ve talked about some engagement metrics, primarily frequency and engagement.
0:26:42 It’s also time, not just how frequent someone is, but just how much time do they spend,
0:26:48 time spent on site, on the piece, writing comments, not just because I mean, the number
0:26:52 of businesses that have great engagement is not high because they’re only so many minutes
0:26:53 in the day.
0:26:58 And so you’re just looking for where, okay, they’re just passing time and enjoying it.
0:27:01 And they both have obvious monetization associated with that behavior.
0:27:04 This is why Netflix is so freaking genius because when they literally invented the format
0:27:08 of binge watching, which you could not do, which is, I love it because it’s a very internet
0:27:09 native concept.
0:27:11 I mean, they’ve literally fucked up everyone else’s engagement numbers.
0:27:16 I think that’s one of the narratives on why building consumer products is much harder
0:27:17 these days.
0:27:18 And do you think it’s true or not?
0:27:21 Well, because it used to be that you were, you know, what kind of time were you competing
0:27:23 for in the first couple of years of the smartphone?
0:27:26 You were competing against literally, I’m going to stare at the back of this person’s
0:27:30 head, or I can like use some cool app that I’ve downloaded, right?
0:27:34 Versus these days, you actually have to take minutes away from other products.
0:27:35 Yes.
0:27:40 And it’s typically other behemoths because the top apps are almost all done by Facebook,
0:27:45 Amazon, Google, and, you know, breaking through just, Mark calls it the first page.
0:27:50 The people who are on the first screen are just such the incumbents.
0:27:54 And sure, most people have Facebook on the screen and YouTube on the screen and Amazon
0:27:55 on the screen.
0:28:01 You know, that competition is a big overhang right now in consumer investing because you
0:28:03 have to displace someone’s minutes.
0:28:06 Yeah, I would add one more layer to that, at least on the content side, which is I think
0:28:09 a lot of people make a lot of category errors because they think they’re competing with
0:28:11 like-minded players.
0:28:14 And in fact, when it comes to things like content and attention, you’re competing with just
0:28:15 about anything that grabs your attention.
0:28:17 It’s not just other media outlets.
0:28:18 It’s Tinder.
0:28:19 It’s Tinder.
0:28:20 It’s a dating app.
0:28:21 It’s something else.
0:28:26 I’m riding on the train for an hour. I could, you know, see one of my friends on our Facebook,
0:28:27 watch videos on YouTube.
0:28:28 Yeah, that actually changes the time blocks.
0:28:31 Xerox Park did a really interesting study on micro-weighting moments, and they’re literally
0:28:35 the little snatches of time, like two seconds here and there, that you might be weighting
0:28:39 in line or doing something so you can do a lot of snack size things in that period, which
0:28:41 is also another interesting thing to think about for how people engage with your business.
0:28:45 So it’s actually funny because there’s some businesses that have good engagement where
0:28:49 it’s one session that goes on for a while, YouTube or Netflix or something like that.
0:28:55 There are others that are multiple small sessions at an aggregate because it’s the micro opportunities
0:28:56 to.
0:28:58 And Google is the best example of this, right?
0:29:03 In fact, if you spend a lot of time on Google.com, you know, refining your searches and clicking
0:29:05 around, that means actually the service is doing poorly.
0:29:06 They failed.
0:29:07 Interesting.
0:29:10 Their goal is to get you to their advertisers as fast as they can.
0:29:14 That’s a frequency play and a monetization play ultimately as opposed to an engagement
0:29:15 one.
0:29:16 Yes.
0:29:17 That’s fascinating.
0:29:18 And then some products are more on the engagement side.
0:29:20 You have to optimize it based on how you’re monetizing it.
0:29:22 What are some of the metrics for retention?
0:29:24 I mean, is it just, should I stay or should I go?
0:29:25 Is that the retention metric?
0:29:29 And I think the big thing is the concept of churn is a tricky one.
0:29:34 In some cases, like subscription, Hulu, Netflix, and then also in the SaaS world, whether or
0:29:36 not you’re still continuing to pay or not, right?
0:29:38 And that’s really obvious.
0:29:41 The thing that’s tricky for a lot of these consumer products, especially episodic ones
0:29:44 and it’s actually less whether they’ve quote unquote churned or not.
0:29:47 It’s actually just whether or not they’re active or inactive and whether or not that’s
0:29:52 happening at a rate that you in your business strategy have decided is acceptable or not.
0:29:56 If every Halloween, you know how there’s those costume stores that like open all over the
0:29:57 place.
0:30:01 If every Halloween, you go back and you buy a costume, but you’re inactive the rest
0:30:02 of the time.
0:30:03 Have you churned or not?
0:30:04 Like it’s not clear.
0:30:07 And I would argue you’ve not churned because you’re doing exactly what they want, which
0:30:09 is to buy a costume every Halloween.
0:30:12 It seems like it makes assessing the retention of a consumer business very difficult.
0:30:14 You adjust the time period that you’re relevant on it.
0:30:20 If the average diner dines twice a year, so you can apply that metric travels a similar
0:30:24 thing Airbnb is, you know, for the average user relatively infrequent, you have to tailor
0:30:26 your look to what are they trying to do.
0:30:30 So if you’re trying to stay up with your friends and you’re doing it twice a year, yeah, that
0:30:31 doesn’t work.
0:30:32 So Facebook has got a whole different center.
0:30:36 One of the things that companies can often do is to measure upstream signal.
0:30:40 So for example, Zillow, you’re probably not going to buy a house very often, right?
0:30:43 Maybe, you know, a couple of times in your life.
0:30:47 However, what’s really interesting is they can say, well, you know, maybe folks aren’t
0:30:49 buying houses, but at least are we top of mind?
0:30:52 Are they checking the houses that are going on sale in their neighborhood?
0:30:53 Are they opening up the emails?
0:30:55 Are they, you know, doing searches, right?
0:30:57 Why do you call that upstream?
0:30:58 In the funnel.
0:31:01 You’re kind of going up in the funnel and you’re tracking those metrics as opposed to, you
0:31:02 know, purchases.
0:31:06 So even, you know, for open table, it’s like, okay, great, well, maybe if you’re not actually
0:31:10 completing the reservations, are you at least checking the app for availability?
0:31:12 What’s new restaurants where I want to dine there?
0:31:16 There’s some level of content consumption throughout this entire episode.
0:31:21 There seems to be this interesting dance between architecting and discovering like you might
0:31:25 know some things upfront because you’re trying to be intentional and build these things.
0:31:29 And then there are things that you discover along the way as your product and your views
0:31:31 and your data evolves.
0:31:34 How do you advise people to sort of navigate that dance?
0:31:38 You iterate, you develop hypotheses, you put it out there and you test the hypothesis.
0:31:41 You know, I think my product is going to behave this way and then did it.
0:31:45 Probably the most important thing is for me, marketing can be art, marketing can be science.
0:31:48 In the consumer internet, it’s more science.
0:31:53 Some companies can effectively do TV campaigns, large media budgets, things like that.
0:31:58 For me, the better companies typically just rip apart their metrics, understand the dynamics
0:32:03 of their business and then figure out ways to improve the business through that knowledge.
0:32:06 And that knowledge could feed back into new product executions or new marketing strategies
0:32:08 or new something.
0:32:13 This consideration, but it’s informed by the data at a level that, you know, on the best
0:32:16 companies is really, really deep.
0:32:23 Ultimately, you have a set of strategies that you’re trying to pursue and you pick the metrics
0:32:26 to validate that you’re on the right track, right?
0:32:29 And a lot of what we’ve talked about today has really been the idea that actually there’s
0:32:32 a lot of nature versus nurture kind of parts about this.
0:32:35 You know, your product could just be located in but high monetization.
0:32:38 And so you shouldn’t look at, you know, DA/UMAU.
0:32:41 And so you have to find really the right set of metrics that show that you’re providing
0:32:46 value to your customers first and foremost, and then really, you know, build your team
0:32:50 and your product roadmap and everything in order to reinforce that.
0:32:54 Find the loops and the networks that exist within your product, because those are the
0:32:58 things that are going to keep your engagement improving over time, even in the face of competition.
0:32:59 Growth is good.
0:33:02 Growth and engagement is really, really, really good.
0:33:03 Okay.
0:33:04 So that’s fabulous.
0:33:05 Well, thank you guys for joining the A6NC podcast.
0:00:10 tax or investment advice or be used to evaluate any investment or security and is not directed
0:00:15 at any investors or potential investors in any A16Z fund. For more details, please see
0:00:21 A16Z.com/disclosures. Hi, everyone. It’s Marketplaces Week over here. Thanks
0:00:26 to our consumer team releasing a new index of the next industry-defining marketplaces.
0:00:32 You can check that out at A16Z.com/marketplace100. But what happens as such marketplaces and other
0:00:38 platforms and products evolve over time as do their users? The conversation that follows
0:00:41 is one of our more popular episodes from a couple years ago, featuring general partners
0:00:47 Andrew Chen and Jeff Jordan in conversation with me on how after you acquire users, which
0:00:52 we covered in another episode, then how do you keep them engaged, retain them, and even
0:00:58 resurrect or re-engage them and what are the key metrics? But first, we began with what
0:01:02 happens after the initial acquisition as different kinds of users join a product or
0:01:07 platform over time. What does that mean for engagement and where do cohort analyses come
0:01:08 in?
0:01:12 One of the things that you see is that people end up using these products very differently
0:01:17 because the kinds of users that you’re getting are changing over time. When you look at something
0:01:24 like rideshare, all the early cohorts are basically people in urban areas. These days,
0:01:29 all of rideshare is more like suburban or rural folks because you’ve saturated all of
0:01:36 the center. What you tend to see is as you acquire your core demographic out, that actually
0:01:42 ends up showing up in the engagement. Going back to natural gravity to the whole thing,
0:01:46 because gravity also hits the engagement side of things as well, and then ultimately the
0:01:52 LTV because your users are typically getting less valuable. It may take years to see this
0:01:56 kind of play out, but that’s kind of the natural law of things.
0:01:59 There is a progression in these and particularly the ones that are really successful. Early
0:02:04 on, it’s all about getting users. Just like users, users, users. If you’re wildly successful
0:02:10 at doing that, you run out of users or you start running low on users and you have to
0:02:15 go to engagement. Pinterest has a very high quality problem right now. Most women in
0:02:18 America have downloaded the Pinterest app.
0:02:20 Oh yeah, I’ve had it for years.
0:02:24 Some growth can come through, okay, there are some number of women who’ve never heard of
0:02:28 Pinterest somewhere in the country, but much more so, they need to engage and re-engage
0:02:33 the existing audience. I mean, we love engagement from an investor standpoint because it’s just
0:02:34 …
0:02:36 It shows stickiness.
0:02:40 You can often hack your way into new users. It’s really hard to hack your way into true
0:02:41 engagement.
0:02:42 Keeping them.
0:02:47 If someone’s spending 20 minutes a day on your site, Pinterest and OfferUp, the major
0:02:53 investment thesis was, “Oh my God, look at that engagement kind of thing.” If they can
0:02:55 scale the user base, it’s a beautiful deal.
0:02:59 What we mean by engagement is actually interacting with them and seeing their activity because
0:03:05 to Andrew’s three points of acquisition engagement retention, the third piece is keeping them.
0:03:10 The way that we’ll often analyze this is looking at cohort analyses where we’ll look at each
0:03:15 batch of users that’s joining in each week and really start to dissect, “Well, how active
0:03:16 are they really?”
0:03:22 To compare all these cohorts over time, you’re basically putting the users that come in from
0:03:27 a particular time frame, let’s say it’s a week, and you’re putting them into a bucket.
0:03:31 What you’re doing is you want to compare all of these different buckets against each other.
0:03:36 What you typically do is you look at a bucket of a cohort of users and you say, “Okay, well,
0:03:40 once they’ve signed up, the week after, how active are they? What about the week after
0:03:44 that and the week after that and you can build out a curve?” It just turns out that these
0:03:49 curves, once you’ve looked at enough of them, surprisingly, human nature, they all look
0:03:50 kind of the same.
0:03:51 I bet.
0:03:55 They kind of all kind of curve down and for the good ones, they start to flatten out and
0:03:59 they plateau and then for the really good ones, they’ll actually swing back up and people
0:04:01 will come back to the surface.
0:04:06 What you want to do is you want to compare the various cohorts against each other in
0:04:13 time to see if you can spot any trends on how the usage patterns are increasing or decreasing.
0:04:18 When you add a new layer to layer cake, you might unlock a bunch of new behavior.
0:04:22 You might unlock a bunch of new frequency that didn’t exist before or alternatively,
0:04:27 over long thresholds of time, people tend to become less active as you move out of your
0:04:28 course.
0:04:29 The cohort graduates.
0:04:33 If not, a specific cohort of users flattens out is really important because if you’re
0:04:37 in a world where they slowly degrades and then all of a sudden it’ll actually go to
0:04:42 zero, that means that you’re naturally always filling up the bucket.
0:04:43 You kind of have a leaky bucket.
0:04:44 You’re constantly filling it out.
0:04:45 You’re always filling it up.
0:04:46 Right?
0:04:51 What happens is that gets progressively harder if you want to keep your overall growth rate
0:04:57 because that means you have to double, triple, quadruple your acquisition in order to counteract
0:04:58 for that.
0:05:03 That growth accounting equation that’s often thrown around is that your net MAUs, so your
0:05:09 net monthly active users, equals all the new people that you’re acquiring, right, minus
0:05:14 all the people that are churned, right, and then plus all the people that you’re resurrecting,
0:05:15 you know.
0:05:16 Reengaging.
0:05:17 Reengaging, exactly.
0:05:18 Right.
0:05:19 That are coming back after they’ve churned.
0:05:23 And so what happens is for a new startup, you are completely focused on new users because
0:05:26 you don’t really have that many users to churn.
0:05:30 And over time, as you get bigger and bigger and bigger, what you find is that your churn
0:05:33 rate starts to, you know, it’s a percentage of your actives.
0:05:37 And so, you know, the evolution of most of these companies as they’re getting bigger
0:05:43 tends to start with acquisition, then focus much more on, you know, churn and retention,
0:05:47 and then ultimately also to layer in resurrection as well.
0:05:50 And the cohort curves have a couple other features that I love.
0:05:56 And usually in marketplace businesses, the best models are built off of the cohort curves.
0:05:57 Oh, interesting.
0:06:00 Because you have to understand that degradation and where it goes.
0:06:03 Using cohorts really give you a sense of, are there network effects?
0:06:07 And network effect is the business gets more valuable, the more users that use it.
0:06:12 If it gets more valuable, your newer cohorts should behave better than your early cohorts.
0:06:13 Why is that?
0:06:16 Because the service is more valuable given how many there are.
0:06:17 Oh, interesting.
0:06:18 So that’s kind of a tip.
0:06:19 I get that.
0:06:22 If you’re going to get more restaurants, you’re going to get a whole lot more reservations
0:06:25 per diner because you are serving more than you need.
0:06:30 So the open table cohorts would climb up and get more attractive over time.
0:06:34 Versus, you know, we talked about typically they tended to grade over time.
0:06:38 If you’ve reversed the polarity and they’re growing over time, it means you’ve made the
0:06:41 business more valuable and then you start projecting forward.
0:06:44 Okay, what a better way to know the business is actually more valuable than thinking is
0:06:46 valuable and bleeding your own mind.
0:06:50 And in network effects business, we always ask, show us the cohorts.
0:06:52 Everyone asserts, I’m a network effect.
0:06:53 I’m a network effect.
0:07:00 But when you say show me the data, cohort curves, or they don’t lie, they don’t lie.
0:07:02 The other really interesting part is segmenting it.
0:07:04 I was about to actually ask you what are the buckets of cohorts?
0:07:06 Are they all demographic data?
0:07:10 For a bunch of hyper-local type businesses, the reason why segmenting it based on market
0:07:15 geography, why that’s so valuable is because then you can compare markets against each
0:07:16 other.
0:07:19 You can say, well, you know, this market which has much more density in terms of the number
0:07:23 of scooters behaves like this, and you can start to draw conclusions, you know, sort
0:07:26 of a natural A/B test in order to do that.
0:07:30 And I think the similar kind of analysis you can do for B2B companies is for products that
0:07:33 have different size teams using it.
0:07:37 If you have a really large team that are all using a product, well, are they all using
0:07:39 the product more as a result?
0:07:42 And let’s compare that to something that maybe only has a couple, right?
0:07:45 And so this way, you can start to kind of disassemble, you know, the structure of these
0:07:48 networks and do they actually lead to higher engagement?
0:07:50 Slack would be a perfect example of that.
0:07:54 You know, just if you have five people in the organization using Slack, you get one use
0:07:55 curve.
0:07:58 If you have, you know, if the organization, it’s the operating system for the organization,
0:07:59 you have a very different curve.
0:08:03 Though it’s not just an accident, you have to sort of architect it, not just expect like
0:08:05 serendipity to fall into place.
0:08:10 So after you get the new users, the way that you have to think about it is around quality,
0:08:11 right?
0:08:14 And you have to make sure that the new users turn into engaged users.
0:08:17 One of the things people often talk about is just sort of this idea of like an aha
0:08:22 moment or a magic moment where the user really understands the true value of the product.
0:08:23 But often that involves a bunch of setup.
0:08:27 So for example, for all the different social products, whether that’s Twitter or Facebook
0:08:31 or Pinterest, et cetera, you have to make sure that when you first bring a new user
0:08:36 and they have to follow all the right people, they have to get the onboarding experience.
0:08:39 Which by the way, isn’t just signing up, but it’s actually, you know, doing all the
0:08:45 things to get to this like, aha, where you’re like, oh, I get this product, it’s for me.
0:08:49 And once you get that, then they’re kind of, then you have the opportunity to keep them
0:08:51 in this engaged state over time.
0:08:55 Is that really such a thing that there is like an aha moment or is it sort of like
0:08:56 accumulative?
0:08:59 A lot of the later users on Facebook came because everyone else was already there.
0:09:01 Is this only tied to new users?
0:09:05 In the case of Facebook, actually the fact that everyone was already there makes the
0:09:07 aha moment that much more powerful, right?
0:09:11 Because all your friends and family, they’re already there, your feed’s already full of
0:09:12 content.
0:09:15 And the first time that you see photos that maybe, you know, of someone that you went
0:09:16 to high school with, right?
0:09:17 Yeah.
0:09:18 That’s actually what happened to me.
0:09:20 I was so excited when I saw an old friend, right?
0:09:21 Right.
0:09:22 Yeah, exactly.
0:09:25 And so what that means is like, you get the product and then afterwards, you know, when
0:09:28 you actually are getting these push notifications or emails that are like, hey, it’s someone’s
0:09:32 birthday or you know, whatever, like you’ve internalized what that product is.
0:09:36 And you know, this moment is different for all sorts of different companies.
0:09:39 I’ve always heard this referred to as the magic number.
0:09:43 When you show up and it’s a blank slate, it’s like, what is this about?
0:09:49 But they would drive you maniacally to, you know, follow people because that when you
0:09:54 got to their magic number where they had statistically correlated the number of followers with long
0:09:59 term engagement and retention, they would kill you to get you there doing what felt
0:10:04 like a natural acts of like, yeah, you log on with follow and you say no and say, yes.
0:10:10 But when they got you there, it kicked in and the service then quote unquote worked for
0:10:11 you.
0:10:13 A lot of the entrepreneurs I work with are trying to figure out what is my magic moment
0:10:16 that then creates the awareness of the value problem.
0:10:18 So take the example of Pinterest.
0:10:21 Pinterest when it goes to a new market, first of all, they figure out they need a lot of
0:10:23 local content to make it compelling to local users.
0:10:29 The U.S. corpus of images doesn’t necessarily is helpful in international markets but isn’t
0:10:30 sufficient.
0:10:31 You’re right.
0:10:35 Like, sorry, I don’t only want like skirts, you may not be able to wear in certain regions.
0:10:36 Exactly.
0:10:39 I haven’t worn a sorry in North America at a long time.
0:10:44 But then once you have the content set, then you have to get compelling information to
0:10:47 that individual in front of them, which, you know, you don’t know the individual when
0:10:48 they walk in the door.
0:10:52 The faster they do that, the more quickly, the better the business performs, engagement
0:10:54 goes up, retention goes up and it works.
0:10:58 So different entrepreneurs had to figure out what is that?
0:11:01 What experience do they want to deliver where people get it?
0:11:03 And then how do you engineer your way into delivering it?
0:11:04 Okay.
0:11:08 So they’ve kind of come up through acquisition and you’ve gotten new users.
0:11:09 They get the product.
0:11:13 You even have hopefully a way to measure that and see and track it over time.
0:11:17 Do you want to then go into trying to get different users?
0:11:18 Do you take your existing users?
0:11:22 One of the things that we covered very early on is that with SaaS, you always want to try
0:11:28 to take existing users and upsell them because it’s way more expensive to acquire a new customer
0:11:29 in that context.
0:11:30 I mean, of course, you want to grow your customers.
0:11:32 How does this play out in this context?
0:11:33 What happens next?
0:11:34 A lot of companies is a progression.
0:11:38 So almost all the early activity in the company is, okay, how do I get the users?
0:11:44 As you get users, you get more and more leverage from efforts at activation and retention and
0:11:45 engagement.
0:11:46 So, use Pinterest as an example again.
0:11:50 A very high percentage of women in America have downloaded Pinterest.
0:11:56 Then the leverage quickly goes into, okay, how do I keep them engaged, reactivates the
0:12:02 one who disappears and their acquisition efforts in the U.S. get de-emphasized and all the
0:12:06 leverage is there, except as they’re going international, they’re still in that acquisition
0:12:07 part of the curve.
0:12:11 So, I think the leverage changes over time based on the situation in the company.
0:12:16 Facebook hasn’t had any users in the U.S. forever because they have them all.
0:12:19 This kind of goes back to this portfolio approach to thinking about your users.
0:12:25 Once you have an active base of users and customers, what starts to get really interesting
0:12:29 is to really analyze what are the things that actually set that group up to be successful,
0:12:33 really long-term sticky users versus what are the behaviors and profiles where users
0:12:34 aren’t successful, right?
0:12:38 You actually throw your data science team on it to figure out all the different weights
0:12:44 for both behavioral as well as the demographic and profile-based stuff on there.
0:12:48 So, one of the first things that you figure out is that each one of these products actually
0:12:55 has this ladder of engagement where oftentimes new users show up to do something that’s
0:13:00 valuable but potentially infrequent, and you need to actually level them up to something
0:13:02 that happens all the time.
0:13:06 For example, when you first install Dropbox, the easiest thing that you can do is you can
0:13:10 use it to just sync your home and your work computers, right?
0:13:15 And that’s great, but really the way to get those users to become really valuable is for
0:13:18 them to start sharing folders at work with their colleagues.
0:13:22 Because once they have that and people are dragging files in, they’re really starting
0:13:25 to collaborate on things, that’s like the next level of value that you can actually
0:13:30 have on a daily basis versus this thing that is in the background that’s just syncing
0:13:31 your storage.
0:13:34 So, what are some of the things that people can then do to move those users up that ladder
0:13:35 of engagement?
0:13:40 Step one is really segmenting your users into this kind of engagement map.
0:13:45 Oftentimes, you’ll see this visualized as a kind of state machine where you have folks
0:13:48 that are new, you have folks that are casual, and you want to track how much they’re moving
0:13:51 up or down in each one of these steps.
0:13:53 And then once you have that, then the question is, okay, well, great, how do you actually
0:13:55 get them to move from one place to the other?
0:13:57 First, there’s content and education.
0:14:01 They need to know in context that they can actually do something.
0:14:06 So, for example, if you can get your users to set their home and work for a transportation
0:14:11 product, then you can maybe figure out, okay, should I prompt them in the morning to try
0:14:13 their ride based on what the ETAs are, right?
0:14:15 Like, in the app, there would be some kind of notification.
0:14:17 Like, life cycle messaging factor in there.
0:14:21 The second is, of course, if your product has some kind of monetary component, then you
0:14:23 can use incentives 10 bucks off.
0:14:27 Your next subscription, if you do this behavior that we know for sure, gets you kind of to
0:14:28 the next step.
0:14:32 And then the third thing is really just refining the product for that particular use case.
0:14:36 But maybe there are certain kinds of products that are transacted all the time, and so you
0:14:41 maybe want to waive fees or give some credits or you do something in order to get people
0:14:43 to kind of get addicted to that as a thing.
0:14:47 And a really interesting thing is the frequency with which something’s consumed.
0:14:53 I mean, eBay had enormous levels of engagement early on for a commerce app in particular.
0:14:57 People would spend hours just browsing because early on it was about collectibles and it
0:14:58 was about people’s passion.
0:15:04 So if someone’s passionate about depression or glass, they will spend hours if you give
0:15:08 them that depth of content to say, “Oh my God, I just found the perfect item.”
0:15:11 Open table and Airbnb are both typically much more episodic.
0:15:14 Most people don’t dine at fine dining restaurants with high frequency.
0:15:18 Our median user dined twice a year on open table.
0:15:22 And so that has completely different marketing implications and user implications.
0:15:26 Measurement’s probably even more important in the engagement retention thing because
0:15:32 we got our data scientists to understand the different consumer journeys through our product.
0:15:38 And then we tried to develop tactics to accelerate the journeys we wanted and limit the journeys
0:15:39 we don’t want.
0:15:44 But in order to develop your strategy, you really need to understand how your users are
0:15:47 behaving at a really refined level.
0:15:49 So what are some of the engagement metrics?
0:15:51 One really important area is frequency.
0:15:55 Just how often are you using the product regardless of the intensity and the length of the sessions
0:15:58 and all that other stuff, you know, just literally just frequency of sessions.
0:16:04 We might often ask for a daily active user divided by monthly active user ratio.
0:16:07 And that gives you a sense for like how many days is a user active.
0:16:08 Doubt them out.
0:16:13 You recently put a post out on the DAU/MAU metric.
0:16:17 And when it works and when it doesn’t, there’s a lot of nuances around when to apply it and
0:16:18 when not to.
0:16:22 DAU/MAU was very much popularized by the fact that Facebook used it, including in their
0:16:24 public financial statements.
0:16:27 And it really makes sense for them because they’re an advertising business and it matters
0:16:30 a lot that people use them a lot all the time, right?
0:16:31 Right.
0:16:34 It’s like counting impressions and being able to sell that to advertisers, right?
0:16:35 Exactly.
0:16:39 And they have historically been 60% plus daily actives over monthly actives.
0:16:40 And that’s very high.
0:16:44 You know, you’re using it more than half the days in a month.
0:16:48 On the flip side, what I was talking about in my essay about this is that DAU/MAU can
0:16:51 tell you if something’s really high frequency and if it’s working.
0:16:55 But a lot of times products are actually lower DAU/MAU for a very good reason because they’re
0:16:58 sort of just a natural cadence, you know, to the product.
0:17:02 Like you’re not going to get somebody who is using a travel product to use it more than
0:17:04 a couple of times per year.
0:17:07 And yet there are many valuable travel companies, obviously.
0:17:09 So you’re saying don’t live and die by that alone.
0:17:10 Exactly, right.
0:17:13 Because it really depends on the product you have, the type of nature of use that it has,
0:17:14 et cetera.
0:17:18 You just want to make sure that the metric reflects whatever strategy that you’re putting
0:17:19 in place.
0:17:22 If you think that your product is a daily use product and you’re going to monetize using,
0:17:25 you know, a little bit of money that you’re making over a long period of time, but your
0:17:30 DAU/MAU is low, is like sub 15%, then like it’s probably not going to work.
0:17:35 And then a metric called L28, which is something else that was pioneered certainly early at
0:17:36 Facebook.
0:17:37 And it’s a histogram.
0:17:38 And what you want to do is…
0:17:40 And a histogram is a frequency diagram.
0:17:41 Right.
0:17:45 A frequency diagram that basically says, okay, you know, show a bar showing how many users
0:17:50 have visited once in that month, then twice in the month, and then three times in the
0:17:52 month, and then four times in the month, and you kind of build that all the way out to
0:17:53 28 days.
0:17:55 Because there’s 28 days in the month on average, right?
0:17:57 And the 28 days is to remove seasonality.
0:18:00 And then related one obviously is like L7, right?
0:18:01 So just like last seven days.
0:18:02 And so what you want to see…
0:18:04 You don’t want to see wows, weekly, active users.
0:18:05 Is that really a thing, by the way?
0:18:06 Am I just making that up?
0:18:07 Right.
0:18:08 Yeah, wow.
0:18:09 That was over a while ago.
0:18:10 You just coined it.
0:18:11 No, great.
0:18:12 I’m not reclaiming retainment.
0:18:13 Why not?
0:18:14 Right.
0:18:17 And so the idea with, you know, an L28 or an L7 is the idea that you should actually
0:18:23 start to see when you graph this out that there’s a group of people who just use it 28
0:18:25 days out of 28 days, right?
0:18:29 And that there’s a big group of people who use it 27 days out of 28 days, right?
0:18:30 And that there’s a big cluster.
0:18:33 And so that’s how you know that you actually have a hardcore segment.
0:18:38 And that’s really important because in all these products you typically have a core part
0:18:41 of the network that’s driving the rest of it, whether that’s power sellers or power
0:18:45 buyers or, you know, in a social network, the creators versus the consumers.
0:18:49 Actually, I’ve heard this referred to as the smile because the one use is always pretty
0:18:50 big.
0:18:51 A lot of people show up once.
0:18:53 I don’t understand what this is and disappear.
0:18:54 So that’s the one.
0:18:57 And then they typically slide down more people use it.
0:19:03 Fewer people use it two days in one, three days in two, done right, it starts to increase
0:19:04 at the end.
0:19:06 So you basically get a smile, you just go down.
0:19:08 And I mean, that’s really powerful.
0:19:10 Facebook had a smile.
0:19:12 WhatsApp had a smile.
0:19:13 Instagram had a smile.
0:19:18 If you take a step back, it’s a precondition for investing in a venture business essentially
0:19:19 that there’s growth.
0:19:24 If it didn’t market and then you want to see growth, but growth by itself is not sufficient.
0:19:26 Investors love engagement.
0:19:31 So Pinterest, the key driver of Pinterest, it was growing, but the users were using it
0:19:32 maniacally.
0:19:33 Oh my God.
0:19:36 I think I spent an entire Thanksgiving using Pinterest.
0:19:41 Was the engagement that blew my mind much more than the growth offer up has engagement
0:19:45 that’s similar to social sites like Instagram and Snap.
0:19:50 I mean, a commerce site, mobile classifieds, people just sit there in troll looking for
0:19:55 marketers.
0:19:59 So Datamow, Smile, all these metrics are so core to us because a big engaged audience
0:20:01 is so rare.
0:20:04 And as a result, it’s almost always incredibly valuable.
0:20:07 And the engagement ends up being very related to acquisition.
0:20:11 Because when you look at all the different acquisition loops, whether it’s paid marketing
0:20:15 or a viral loop or whatever, all of those things are actually powered by engagement
0:20:16 ultimately.
0:20:21 We need people to get excited about a product in order to share content off of that platform
0:20:24 to other platforms in order to get a viral loop going.
0:20:30 And so one of the things I was going to also add on Daumau and L28 is that they’re actually
0:20:31 really hard to game, right?
0:20:32 Which is fascinating.
0:20:36 Whereas growth can be very easy to game.
0:20:37 Right.
0:20:38 Exactly.
0:20:39 Yeah.
0:20:40 Why is that?
0:20:41 What’s the difference?
0:20:42 The typical approach is to say, well, I’m going to add an email notifications.
0:20:43 I’m going to do more push notifications.
0:20:44 I’m going to do more of this and that.
0:20:48 And then automatically, you know, these metrics ought to go up, right?
0:20:53 The challenging thing is actually usually sending out more notifications will actually
0:20:56 cause more of your casual users to show up because your hardcore users were already kind
0:20:58 of showing up, you know, already.
0:21:03 And what that does is that’ll increase your monthly actives number, but actually not
0:21:05 increase your daily actives as much.
0:21:10 So I’ve actually seen cases where sending out more email decreases your Daumau as opposed
0:21:11 to increasing it.
0:21:12 That’s really interesting.
0:21:16 When you think about this portfolio of metrics, it really tells you a story about people are
0:21:18 kind of coming but not really staying.
0:21:22 If you get an email or push notification every day, eventually you turn them off and then
0:21:23 you just stop.
0:21:27 So then you get counted as a male for that period of time and then, you know, you lose
0:21:28 them as it out.
0:21:31 Acquisition is super easy to game because you can just spend money.
0:21:35 Or you got a distribution hack that works early on in the Facebook platform.
0:21:40 Companies literally got to a million users and it felt like minutes just because there
0:21:46 were so many people on Facebook and the ones who were early just got exploding user bases.
0:21:50 There were a number of concepts whose mean number of visits was one.
0:21:51 They never came back.
0:21:56 So you get to see these incredibly seductive growth curves, but our job is essentially
0:22:01 to be cynical and just say, okay, we need to go below that because there are a lot of
0:22:03 talented growth hackers who can drive growth.
0:22:08 I looked at a number of businesses that tens of millions of users and no one ever came
0:22:09 back.
0:22:11 Engagement is so, so key.
0:22:14 So we’ve talked especially about the fact that growth and network effects are not the
0:22:18 exact same thing because network effects by definition are that a network becomes more
0:22:21 valuable the more users that use it.
0:22:23 What happens on the engagement side with network effects?
0:22:26 What are the things we should be thinking about in that context?
0:22:30 Typically network effects, if they’re real, manifest in data.
0:22:33 Things like the cohort curves improve over time.
0:22:35 Usually there’s a decay with network effects.
0:22:39 There often is a reversal where they’re growing because it’s more valuable.
0:22:40 Their smile essentially.
0:22:44 My diligence at OpenTable was I looked at San Francisco, which was their first market
0:22:50 and sales rep productivity grew over time, restaurant churn decreased over time.
0:22:52 The number of diners per restaurant increased over time.
0:22:56 The percentage that booked through OpenTable versus the restaurant’s own website moved
0:23:00 towards OpenTable dramatically, every metric improved.
0:23:04 And so, you know, that’s where it both drives good engagement, but also it just improves
0:23:05 the investment basis.
0:23:06 The value overall, right?
0:23:11 The interesting points about network effects is that we often talk about it as if it’s
0:23:12 a binary thing.
0:23:13 Right.
0:23:14 Or homogenous.
0:23:15 Like all network effects are equal when they’re not.
0:23:16 Exactly, right.
0:23:18 When you look at the data, what you really figure out is network effect is actually like
0:23:22 a curve and it’s not like a binary yes/no kind of thing.
0:23:27 So, you know, for example, I would guess that if you plotted the number, if you took a bunch
0:23:32 of cities, every city was a data point and you graphed on one side the number of restaurants
0:23:36 in the city versus the conversion rate for that city.
0:23:39 You would quickly find that when cities have more restaurants, the conversion rate is higher,
0:23:40 right?
0:23:41 I’m just guessing.
0:23:43 It’s almost perfect with one refinement.
0:23:47 The number of restaurants you have is a percent of that market’s restaurant universe.
0:23:48 Okay, right.
0:23:50 Because having a hundred restaurants in Des Moines is different than having a hundred
0:23:51 restaurants in Manhattan.
0:23:52 Makes total sense.
0:23:57 So, not only that, what you then quickly figure out is that there’s some kind of a diminishing
0:24:00 effect to these things often in many cases.
0:24:05 So, for example, in rideshare, if you are going to get a car called 15 minutes versus
0:24:07 10 minutes, that’s very meaningful.
0:24:11 But if it’s, you know, five minutes versus two minutes, your conversion rate doesn’t
0:24:12 actually go up.
0:24:16 If you can express your network effect in this kind of a manner, then what you can start
0:24:21 to show is, okay, yeah, we have a couple new investment markets that maybe, you know,
0:24:24 don’t have as many restaurants or don’t have as many cars.
0:24:29 But if we put money into them and invest in them and build the right products, et cetera,
0:24:30 then you can grow.
0:24:34 You can do this kind of same analysis, whether you’re talking about, you know, YouTube channels
0:24:37 and the number of subscribers you might have.
0:24:38 Having more videos is better.
0:24:39 I’m sure you can show that.
0:24:43 If you go into the workplace and you start thinking about collaboration tools, then what
0:24:50 you ought to see is that as more people use your chat platform or your collaborative document
0:24:53 editing platform, then the engagement on that is going to be higher.
0:24:56 And you should be able to show that in the data by comparing a whole bunch of different
0:24:57 teams.
0:25:01 Okay, so we’ve talked about engagement and also how it applies to network effects.
0:25:04 Are engagement and retention the same thing?
0:25:06 I mean, in all honesty, they sound like they would be the same thing.
0:25:07 There’s overlap.
0:25:08 Yeah, there’s overlap.
0:25:09 Yeah.
0:25:10 Just to give a couple examples.
0:25:14 So weather is low frequency, but high retention because like, you’re actually going to need
0:25:15 to know what the weather is.
0:25:16 Only once a day.
0:25:17 You know, yeah.
0:25:19 Unless you live in San Francisco, you got to check it like 20 times a day with all the
0:25:20 network.
0:25:21 Right, exactly.
0:25:22 Or if you live down here, you have to check it twice a year.
0:25:23 That’s true.
0:25:24 It’s actually the same year around.
0:25:28 That’s actually what it showed was actually more that generally people didn’t really check
0:25:29 it that often.
0:25:33 However, we were highly likely to have it installed and running after 90 days because
0:25:34 that’s, it’s a reference thing.
0:25:35 It’s so important.
0:25:36 Yeah.
0:25:41 Versus if you look at something like games or ebooks or, you know, those kinds of products,
0:25:45 like really high engagement because you’re like, all right, I’m going to get to, I’m
0:25:48 going to finish this like trashy science fiction novel that I’ve been reading.
0:25:50 I’m just going to like get to it.
0:25:53 But then as soon as you’re done, you’re like, okay, there’s no reason why I would go back
0:25:54 and read it again.
0:25:58 So the real difference is that engagement obviously varies depending on the product.
0:26:02 The type of thing it is, whether it’s a weather or ebook and retention is, are you still using
0:26:04 it after X amount of time?
0:26:06 And different companies have different cadences.
0:26:12 If the average users twice a year retention is, did they book annually other businesses
0:26:13 or did they come daily?
0:26:17 But the model behind retention is completely different and the model behind engagement
0:26:18 is completely different.
0:26:19 Right.
0:26:22 The chart that I’d love to really see is one that was a bunch of different categories
0:26:26 that’s retention versus frequency versus monetization.
0:26:30 And I think you got to be like really good at least on one of those axes.
0:26:32 So we’ve done sort of this taxonomy of metrics.
0:26:34 We’ve talked about the acquisition metrics.
0:26:38 We’ve talked about some engagement metrics, primarily frequency and engagement.
0:26:42 It’s also time, not just how frequent someone is, but just how much time do they spend,
0:26:48 time spent on site, on the piece, writing comments, not just because I mean, the number
0:26:52 of businesses that have great engagement is not high because they’re only so many minutes
0:26:53 in the day.
0:26:58 And so you’re just looking for where, okay, they’re just passing time and enjoying it.
0:27:01 And they both have obvious monetization associated with that behavior.
0:27:04 This is why Netflix is so freaking genius because when they literally invented the format
0:27:08 of binge watching, which you could not do, which is, I love it because it’s a very internet
0:27:09 native concept.
0:27:11 I mean, they’ve literally fucked up everyone else’s engagement numbers.
0:27:16 I think that’s one of the narratives on why building consumer products is much harder
0:27:17 these days.
0:27:18 And do you think it’s true or not?
0:27:21 Well, because it used to be that you were, you know, what kind of time were you competing
0:27:23 for in the first couple of years of the smartphone?
0:27:26 You were competing against literally, I’m going to stare at the back of this person’s
0:27:30 head, or I can like use some cool app that I’ve downloaded, right?
0:27:34 Versus these days, you actually have to take minutes away from other products.
0:27:35 Yes.
0:27:40 And it’s typically other behemoths because the top apps are almost all done by Facebook,
0:27:45 Amazon, Google, and, you know, breaking through just, Mark calls it the first page.
0:27:50 The people who are on the first screen are just such the incumbents.
0:27:54 And sure, most people have Facebook on the screen and YouTube on the screen and Amazon
0:27:55 on the screen.
0:28:01 You know, that competition is a big overhang right now in consumer investing because you
0:28:03 have to displace someone’s minutes.
0:28:06 Yeah, I would add one more layer to that, at least on the content side, which is I think
0:28:09 a lot of people make a lot of category errors because they think they’re competing with
0:28:11 like-minded players.
0:28:14 And in fact, when it comes to things like content and attention, you’re competing with just
0:28:15 about anything that grabs your attention.
0:28:17 It’s not just other media outlets.
0:28:18 It’s Tinder.
0:28:19 It’s Tinder.
0:28:20 It’s a dating app.
0:28:21 It’s something else.
0:28:26 I’m riding on the train for an hour. I could, you know, see one of my friends on our Facebook,
0:28:27 watch videos on YouTube.
0:28:28 Yeah, that actually changes the time blocks.
0:28:31 Xerox Park did a really interesting study on micro-weighting moments, and they’re literally
0:28:35 the little snatches of time, like two seconds here and there, that you might be weighting
0:28:39 in line or doing something so you can do a lot of snack size things in that period, which
0:28:41 is also another interesting thing to think about for how people engage with your business.
0:28:45 So it’s actually funny because there’s some businesses that have good engagement where
0:28:49 it’s one session that goes on for a while, YouTube or Netflix or something like that.
0:28:55 There are others that are multiple small sessions at an aggregate because it’s the micro opportunities
0:28:56 to.
0:28:58 And Google is the best example of this, right?
0:29:03 In fact, if you spend a lot of time on Google.com, you know, refining your searches and clicking
0:29:05 around, that means actually the service is doing poorly.
0:29:06 They failed.
0:29:07 Interesting.
0:29:10 Their goal is to get you to their advertisers as fast as they can.
0:29:14 That’s a frequency play and a monetization play ultimately as opposed to an engagement
0:29:15 one.
0:29:16 Yes.
0:29:17 That’s fascinating.
0:29:18 And then some products are more on the engagement side.
0:29:20 You have to optimize it based on how you’re monetizing it.
0:29:22 What are some of the metrics for retention?
0:29:24 I mean, is it just, should I stay or should I go?
0:29:25 Is that the retention metric?
0:29:29 And I think the big thing is the concept of churn is a tricky one.
0:29:34 In some cases, like subscription, Hulu, Netflix, and then also in the SaaS world, whether or
0:29:36 not you’re still continuing to pay or not, right?
0:29:38 And that’s really obvious.
0:29:41 The thing that’s tricky for a lot of these consumer products, especially episodic ones
0:29:44 and it’s actually less whether they’ve quote unquote churned or not.
0:29:47 It’s actually just whether or not they’re active or inactive and whether or not that’s
0:29:52 happening at a rate that you in your business strategy have decided is acceptable or not.
0:29:56 If every Halloween, you know how there’s those costume stores that like open all over the
0:29:57 place.
0:30:01 If every Halloween, you go back and you buy a costume, but you’re inactive the rest
0:30:02 of the time.
0:30:03 Have you churned or not?
0:30:04 Like it’s not clear.
0:30:07 And I would argue you’ve not churned because you’re doing exactly what they want, which
0:30:09 is to buy a costume every Halloween.
0:30:12 It seems like it makes assessing the retention of a consumer business very difficult.
0:30:14 You adjust the time period that you’re relevant on it.
0:30:20 If the average diner dines twice a year, so you can apply that metric travels a similar
0:30:24 thing Airbnb is, you know, for the average user relatively infrequent, you have to tailor
0:30:26 your look to what are they trying to do.
0:30:30 So if you’re trying to stay up with your friends and you’re doing it twice a year, yeah, that
0:30:31 doesn’t work.
0:30:32 So Facebook has got a whole different center.
0:30:36 One of the things that companies can often do is to measure upstream signal.
0:30:40 So for example, Zillow, you’re probably not going to buy a house very often, right?
0:30:43 Maybe, you know, a couple of times in your life.
0:30:47 However, what’s really interesting is they can say, well, you know, maybe folks aren’t
0:30:49 buying houses, but at least are we top of mind?
0:30:52 Are they checking the houses that are going on sale in their neighborhood?
0:30:53 Are they opening up the emails?
0:30:55 Are they, you know, doing searches, right?
0:30:57 Why do you call that upstream?
0:30:58 In the funnel.
0:31:01 You’re kind of going up in the funnel and you’re tracking those metrics as opposed to, you
0:31:02 know, purchases.
0:31:06 So even, you know, for open table, it’s like, okay, great, well, maybe if you’re not actually
0:31:10 completing the reservations, are you at least checking the app for availability?
0:31:12 What’s new restaurants where I want to dine there?
0:31:16 There’s some level of content consumption throughout this entire episode.
0:31:21 There seems to be this interesting dance between architecting and discovering like you might
0:31:25 know some things upfront because you’re trying to be intentional and build these things.
0:31:29 And then there are things that you discover along the way as your product and your views
0:31:31 and your data evolves.
0:31:34 How do you advise people to sort of navigate that dance?
0:31:38 You iterate, you develop hypotheses, you put it out there and you test the hypothesis.
0:31:41 You know, I think my product is going to behave this way and then did it.
0:31:45 Probably the most important thing is for me, marketing can be art, marketing can be science.
0:31:48 In the consumer internet, it’s more science.
0:31:53 Some companies can effectively do TV campaigns, large media budgets, things like that.
0:31:58 For me, the better companies typically just rip apart their metrics, understand the dynamics
0:32:03 of their business and then figure out ways to improve the business through that knowledge.
0:32:06 And that knowledge could feed back into new product executions or new marketing strategies
0:32:08 or new something.
0:32:13 This consideration, but it’s informed by the data at a level that, you know, on the best
0:32:16 companies is really, really deep.
0:32:23 Ultimately, you have a set of strategies that you’re trying to pursue and you pick the metrics
0:32:26 to validate that you’re on the right track, right?
0:32:29 And a lot of what we’ve talked about today has really been the idea that actually there’s
0:32:32 a lot of nature versus nurture kind of parts about this.
0:32:35 You know, your product could just be located in but high monetization.
0:32:38 And so you shouldn’t look at, you know, DA/UMAU.
0:32:41 And so you have to find really the right set of metrics that show that you’re providing
0:32:46 value to your customers first and foremost, and then really, you know, build your team
0:32:50 and your product roadmap and everything in order to reinforce that.
0:32:54 Find the loops and the networks that exist within your product, because those are the
0:32:58 things that are going to keep your engagement improving over time, even in the face of competition.
0:32:59 Growth is good.
0:33:02 Growth and engagement is really, really, really good.
0:33:03 Okay.
0:33:04 So that’s fabulous.
0:33:05 Well, thank you guys for joining the A6NC podcast.
It’s “Marketplaces Week” for us at a16z, thanks to our consumer team releasing a new index of the next industry-defining marketplaces, the Marketplace 100.
But what happens as such marketplaces and other platforms evolve over time, as do their users? This episode is a rerun of a popular conversation from a couple years ago — featuring general partners Andrew Chen and Jeff Jordan (in conversation with Sonal Chokshi) — on what comes after user acquisition: retention.
It’s all about engagement. So what are the key metrics? And if different kinds of users join a platform over time — what does that mean for engagement, and where do cohort analyses come in?