0:00:01 [MUSIC PLAYING]
0:00:03 Hey, welcome back to the Next Wave Podcast.
0:00:04 I’m Matt Wolf.
0:00:05 I’m here with Nathan Lanz.
0:00:08 And there has been an absolute ton of AI news
0:00:10 that came out recently, especially in the world
0:00:11 of large language models.
0:00:12 We got GROC 3.
0:00:14 We got GPT 4.5.
0:00:16 We got Claude 3.7.
0:00:21 Just so many new big foundation models have been released.
0:00:23 And so for this episode, we wanted
0:00:26 to deep dive into what each one is good at
0:00:28 and what each one is not good at.
0:00:29 So for that, we brought on our good friend
0:00:33 Matthew Berman, who is probably the best person we know
0:00:35 to really compare them all.
0:00:37 Because he deep dives and tests every single one
0:00:40 of these models way deeper than we test them.
0:00:44 So let’s go ahead and just dive right in with Matthew Berman.
0:00:46 Thanks so much for joining us, Matthew.
0:00:48 It’s great to have you back on the show.
0:00:48 Thanks for having me.
0:00:51 I’ve been telling people for a long time–
0:00:53 I think I even mentioned this to you last time we had you
0:00:53 on the show–
0:00:55 that when it comes to large language models
0:00:59 and trying to compare them and talk about which model is best
0:01:01 at this and which model is best at that.
0:01:03 I don’t even do that on my YouTube channel anymore.
0:01:05 I just point people to your channel.
0:01:07 I’m like, yeah, Matthew’s going to test this.
0:01:09 And he’s going to tell you which models do what better
0:01:10 than others.
0:01:12 So you’re like my go-to now when it comes
0:01:15 to comparing large language models.
0:01:17 I appreciate that.
0:01:17 That’s awesome.
0:01:18 Thanks for having me again.
0:01:21 I had a great time last time, so I’m excited to chat again.
0:01:22 Yeah, likewise.
0:01:23 So this will be fun.
0:01:26 I’m trying to figure out where the best place to dig in
0:01:28 is because so much has come out.
0:01:29 Here’s kind of the timeline of events
0:01:31 that I feel are the important events, right?
0:01:32 We got Grock 3.
0:01:36 And then a few days later, we got Claude Sonnet 3.7.
0:01:40 And then a few days after that, we got GPT 4.5, which
0:01:43 actually happens to be the day that we’re recording this episode
0:01:45 is the day that GPT 4.5 came out.
0:01:49 So we’ve all had that news in our heads for three hours now.
0:01:51 And that’s about it.
0:01:53 But maybe we start with Grock.
0:01:56 Matt, what have your thoughts been on Grock 3 so far?
0:01:57 Like how much have you played with it?
0:02:00 And what have you found it’s like really good at so far?
0:02:03 Yeah, so if I could show you my bookmarks bar in Chrome,
0:02:07 you will see that Grock now has a prominent placement
0:02:10 right next to chat GPT and right next to perplexity.
0:02:12 So the answer is I use it a lot.
0:02:16 And it has really become my go-to large language model.
0:02:16 Oh, really?
0:02:17 Yeah.
0:02:17 Same here.
0:02:20 Look, I have been a pretty die-hard chat GPT user.
0:02:21 There’s really two reasons.
0:02:23 Number one is speed, right?
0:02:26 I think speed of these models, speed of the response
0:02:29 is really underappreciated by a lot of people.
0:02:32 But it’s the same reason why you converted a higher rate
0:02:34 when a web page loads faster.
0:02:37 It just builds trust and you just get the answer more quickly.
0:02:39 And then it’s also the real time information.
0:02:43 Having access to all of the news on X in real time
0:02:45 is such a killer feature.
0:02:47 And of course, it’s a good model.
0:02:49 So I think those two factors, plus it just
0:02:53 being a fantastic model, that has made it my go-to model.
0:02:54 But that might change now that 4.5 is out,
0:02:55 but we’ll get there.
0:02:56 Yeah, yeah, yeah.
0:02:57 One thing I would say, too, is they
0:02:59 say that they’re really improving the model really fast.
0:03:02 And I believe it, because their team actually reached out to me.
0:03:03 And they’re like, what’s your feedback?
0:03:05 And I’ve been going back and forth with them,
0:03:07 giving them my thoughts on what they should be doing.
0:03:08 Oh, that’s cool.
0:03:08 Yeah.
0:03:10 And sorry if Aravind or anyone from Perplexity
0:03:12 hears this, but I’m like, this has replaced Perplexity for me.
0:03:14 And I think you guys should double down on that.
0:03:15 That’s what I was telling them.
0:03:17 I think you guys should be replacing Google,
0:03:19 replacing Wikipedia, replacing Perplexity.
0:03:21 Any question I have, I should be going to GROC.
0:03:23 They’re combining search and X data.
0:03:25 Like, no one has that, and no one ever will have that.
0:03:26 Yeah.
0:03:29 Nathan, does that make you a high-taste tester?
0:03:30 Oh, yeah.
0:03:32 Because like, I’m certainly not.
0:03:33 I get no previews at all.
0:03:34 Yeah.
0:03:35 I don’t either.
0:03:37 I don’t seem to get any early access
0:03:41 to any of the anthropic models, the open AI models,
0:03:42 the X models, none of them.
0:03:47 So yeah, I think Nathan probably has the best ins right now.
0:03:49 I was a low-level member of a lot of the different Silicon
0:03:51 Valley mafias.
0:03:52 I was not a high-level member, but I
0:03:55 was the low level of several of them.
0:03:59 The gaming one, the social media one, the crypto one,
0:04:02 the Taiwanese mafia.
0:04:04 I was kind of a member of all of those.
0:04:06 And some of the YC people, as well, I know a lot of them.
0:04:07 A lot of them, too.
0:04:10 You know, I personally think that GROC’s biggest hurdle
0:04:11 is just Elon.
0:04:14 I’m sure you’ve seen this, Matthew, on your YouTube channel,
0:04:16 because you’ve made a couple videos about GROC now.
0:04:18 When GROC came out, I made a news video about it.
0:04:21 And overwhelmingly, all of the comments about GROC are like,
0:04:24 I’m never going to touch anything Elon makes.
0:04:27 And so I almost think that, like, so many people
0:04:29 are throwing the baby out with the bathwater
0:04:32 when it comes to actually using and trying GROC
0:04:35 just because it’s Elon’s name attached to it.
0:04:39 Yeah, so whatever I post anything about GROC on X,
0:04:43 if it’s negative about GROC or Elon, I get flamed.
0:04:47 If it’s positive, everybody cheers it, shares it,
0:04:48 everything.
0:04:50 The opposite is true on YouTube.
0:04:53 I made a video about GROC 3 being really good,
0:04:54 because it is.
0:04:57 So many people commented, they won’t touch it just like you,
0:04:58 won’t touch it.
0:05:03 Elon’s AI model, bias, conservative, right wing.
0:05:04 I’m apolitic.
0:05:06 I’m trying to stay out of it as much as I can in the videos.
0:05:09 But anytime I mention anything having
0:05:16 to do with Elon X, GROC, I get the meanest comments on YouTube.
0:05:17 It’s wild.
0:05:20 I think so many people are just going to not experience
0:05:24 probably what is the best model available right now
0:05:27 just because of the Elon factor.
0:05:29 Now, when it comes to code, I don’t necessarily
0:05:31 think GROC is the best.
0:05:31 I don’t know.
0:05:34 I actually honestly haven’t tested GROC with code
0:05:36 because I mostly do code with cursor.
0:05:39 And I don’t believe GROC 3 has pushed its API out yet.
0:05:40 I don’t think.
0:05:41 I don’t think so.
0:05:43 So I’ve been mostly playing with 3.7.
0:05:44 I’ve tested it.
0:05:46 I’ve tested it a little bit.
0:05:46 It’s weird.
0:05:48 It’s not very consistent.
0:05:49 There’s sometimes where it’s amazing at code.
0:05:52 You’re like, oh, wow, that’s like a really creative solution.
0:05:53 And it’s doing something better than Claude
0:05:55 in some very limited circumstances.
0:05:57 But then other times, it just won’t follow my instructions
0:05:58 as well for code.
0:06:01 In terms of reliability, when Claude 3.7 came out,
0:06:03 that’s by far the best now for coding.
0:06:05 But it feels like there is something there.
0:06:07 I wouldn’t discount them forever.
0:06:08 They could improve that.
0:06:10 And all of a sudden, GROC 3 could be the best at coding.
0:06:11 Right.
0:06:12 Matthew, I’m curious.
0:06:14 So when it comes to GROC 3, I know
0:06:17 you have your own internal benchmarks
0:06:20 that you’ve been using on some of these tools.
0:06:22 What have you found GROC 3 is really good at?
0:06:25 And are there any things GROC 3 is just not
0:06:26 going to be your go-to for?
0:06:28 Well, let me say something first.
0:06:30 I had to throw my benchmarks out.
0:06:32 Because they were completely saturated.
0:06:35 They were absolutely annihilated by every single model
0:06:36 that comes out nowadays.
0:06:37 So I threw them out.
0:06:42 I’m currently in the process of creating a new set of benchmarks
0:06:43 and questions.
0:06:46 GROC, again, the thing I go to it for
0:06:49 is real-time information as quickly as I need it.
0:06:51 And that it’s awesome at.
0:06:53 I’ve tested it on some other things,
0:06:56 like quick coding challenges, some math challenges.
0:06:57 And it does really well.
0:07:00 But those aren’t the everyday use cases for me.
0:07:02 So there’s only so much I can test with it.
0:07:05 I’ll just prompt it with one of my benchmark questions,
0:07:05 see if it’s right.
0:07:07 It’s like, OK, yeah, it’s right or no, it’s wrong.
0:07:09 But overall, what I care about and I
0:07:12 think what most people care about is day-to-day usage.
0:07:15 Is it going to solve my problems?
0:07:16 And it does.
0:07:17 GROC3 is great at that.
0:07:21 And typically, I was using GPT-40.
0:07:24 And now GROC3 with thinking kind of replaces that,
0:07:26 although I don’t really need the thinking.
0:07:28 Yeah, it’s just really good at the day-to-day stuff
0:07:29 that I would use it for.
0:07:33 So I’m kind of split between GROC3, perplexity, and chat
0:07:34 GPT.
0:07:35 And now 4.5 came out.
0:07:37 I would say I tested it right before we got on here.
0:07:39 And I would say, OK, 4.5 has a better
0:07:41 vibe for general chat.
0:07:42 But it’s slower.
0:07:43 Oh, it’s about the vibes.
0:07:44 But when you get the response back,
0:07:46 the liked response is better, actually.
0:07:47 So that’s impressive.
0:07:50 It’s writing is better, but it’s slower.
0:07:51 So there’s a huge trade up there.
0:07:53 It feels like 4.5 is going to serve
0:07:57 something maybe for creative writing or creative work.
0:07:59 It’s probably the best for now.
0:08:01 But it is interesting that the cost is so much higher.
0:08:02 Really?
0:08:04 Yeah, I think it was 70 per million input tokens,
0:08:06 half of that for cashed input.
0:08:08 It’s funny, Nathan, that you bring up writing.
0:08:12 That is honestly something that I want to use AI for much more
0:08:15 often than I do, but it is so bad.
0:08:16 It’s so bad at writing.
0:08:20 And I haven’t had enough time to test 4.5 for writing,
0:08:21 but I really hope that it’s good.
0:08:22 It’s better.
0:08:25 Because then that’s going to be able to help me with a lot
0:08:26 of things that I do day to day.
0:08:28 I don’t script my videos, but sometimes I
0:08:31 want help writing the bullet points for them.
0:08:34 Or sometimes I want a tweet thread,
0:08:35 the initial drafts created for me.
0:08:38 I gave it probably, I don’t know,
0:08:40 over 100 pages of different notes about my game,
0:08:43 including the story and things like that and the game mechanics.
0:08:46 And it took a long time to respond, actually, like a very long.
0:08:47 I was kind of surprised.
0:08:49 This feels like a one pro when you hit into a lot of stuff.
0:08:50 4.5?
0:08:50 4.5.
0:08:52 Took a long time to respond.
0:08:52 I was surprised.
0:08:53 Like very long.
0:08:54 It was very slow.
0:08:56 Like three minutes to respond or something to all that.
0:08:59 But then its notes on the story were perfect.
0:09:02 It gave me amazing critiques of like, I love this part.
0:09:06 And it even had little emojis having color coatings.
0:09:10 It had green, orange, blue, and red and different ones.
0:09:11 It was like, like, green’s good.
0:09:12 Love this part.
0:09:14 This part probably could be tweaked.
0:09:15 This part don’t like it.
0:09:16 Here’s why.
0:09:17 And these parts are interesting.
0:09:19 Maybe you keep them, maybe you don’t.
0:09:20 The feedback was good.
0:09:21 Like it was solid.
0:09:22 So I do want to test it more for writing.
0:09:23 But my first impression is,
0:09:26 yeah, it is probably the best model for writing now.
0:09:27 That’s great.
0:09:27 Yeah.
0:09:29 Well, going back to Grock for a second,
0:09:31 have you guys played with the voice mode yet?
0:09:34 I’ve used advanced voice mode a little bit.
0:09:37 I don’t find it to be like super useful throughout my day.
0:09:39 If I’m driving, if I’m walking,
0:09:41 if there’s just not a screen in front of me,
0:09:42 maybe I’ll use it.
0:09:45 But what I found specifically with ChatGPT advanced voice mode
0:09:49 is you’ll ask a question and there’s that delay.
0:09:51 And then sometimes it repeats it and then stops
0:09:53 and then repeats it again.
0:09:56 It’s just such a high friction experience.
0:09:58 It’s not great yet.
0:10:00 I do use voice on perplexity.
0:10:01 I’m a big baseball fan.
0:10:03 Spring Training just started with baseball.
0:10:04 They’ve introduced some like new rules
0:10:06 that I didn’t realize existed.
0:10:08 I was watching one of the Spring Training games.
0:10:09 I opened up perplexity and I’m like,
0:10:11 “Hey, is there a new rule that I didn’t know about?”
0:10:13 And I just will talk to perplexity and ask it questions
0:10:15 and it will like do all the research,
0:10:17 figure out the new rules that are going on in baseball
0:10:19 and then give them back to me.
0:10:21 And I found that pretty helpful.
0:10:24 I still find myself typing my questions more often
0:10:25 than speaking them.
0:10:27 But every once in a while, I’ll be feeling lazy.
0:10:29 I’ll hit the voice button and just ask my question.
0:10:30 I find that helpful.
0:10:34 I know Nathan, you kind of almost use some of the voice modes,
0:10:36 especially advanced voice and ChatGPT
0:10:38 for like journaling, right?
0:10:40 I’m a big believer in it long-term,
0:10:41 but I have been kind of disappointed
0:10:42 like in advanced voice mode.
0:10:44 Like the demo they showed seemed amazing
0:10:45 and it seemed like they removed
0:10:46 so many different parts of it.
0:10:47 And some of the things they demoed
0:10:48 when you tried them in real life,
0:10:51 they don’t work as well as in the demos.
0:10:52 My wife’s Japanese.
0:10:53 My Japanese is getting better,
0:10:54 so I don’t have to use it as much,
0:10:56 but for like really hard topics,
0:10:58 we haven’t tried to use it to do translation.
0:11:01 And it just gets confused so easily.
0:11:02 Like as soon as you go from one language to the other,
0:11:04 like sometimes it’ll translate it properly.
0:11:06 So like, “Okay, cool, it worked in that use case.”
0:11:07 But okay, when then she talks back to me,
0:11:09 then sometimes it just totally gets confused.
0:11:10 And instead of translating,
0:11:13 sometimes it’ll like start talking versus translating.
0:11:15 It’ll start saying its own stuff
0:11:16 versus doing a translation.
0:11:18 And it’s like, “Okay, that’s super annoying.”
0:11:19 And then we just turn it off
0:11:21 every single time that’s ever happened.
0:11:23 – Yeah, I know Sam Holtman’s even talked about,
0:11:25 he wants the AIs in the future
0:11:28 to be more in line with your own beliefs, right?
0:11:31 So like, it knows your religious beliefs,
0:11:32 your political beliefs.
0:11:34 – So bring your own bias.
0:11:35 – Yeah, your own bias.
0:11:36 It will actually learn your bias
0:11:38 and sort of lean into your bias more
0:11:40 to give you more of what you want.
0:11:42 And I think that’s a really, really scary thing.
0:11:44 There was a little bit of word
0:11:45 about this happening on like Facebook, right?
0:11:49 Where Facebook was using like AI bots in the feed
0:11:52 that people didn’t even realize were AI bots.
0:11:54 And so people would post stuff on Facebook.
0:11:56 They would get like a bunch of responses
0:11:57 from these AI bots,
0:11:59 not even realize that they’re AI.
0:12:00 And they’re going, “Oh, cool,
0:12:02 I get great engagement on Facebook.”
0:12:04 And it keeps on bringing them back to Facebook
0:12:06 over and over and over again,
0:12:08 because Facebook’s where they get engagement
0:12:09 and they’re not even realizing
0:12:10 that they’re talking to AIs.
0:12:12 I think that’s going to be a bigger
0:12:13 and bigger problem as well,
0:12:17 where social media is all about getting dopamine hits, right?
0:12:20 We post our tweets because we want to get those likes.
0:12:21 We want to get those retweets.
0:12:22 We want to get those comments.
0:12:23 Every time we see one of those,
0:12:25 we get a little dopamine hit
0:12:27 and we keep on coming back for more.
0:12:30 Well, if AI gets really, really, really good
0:12:33 at giving us those dopamine hits every time we want them,
0:12:36 we’re going to go to wherever we get the most dopamine hits
0:12:38 at the highest frequency.
0:12:41 I think that’s what’s really sort of worries me
0:12:41 about the future.
0:12:44 And it ties into like the whole population collapse as well.
0:12:46 I think it gets to that point
0:12:47 where people communicate with other humans
0:12:49 less and less and less and less,
0:12:51 because they’re getting their dopamine hits
0:12:53 from fake people on social media.
0:12:55 They’re getting their conversational needs
0:12:57 met by unhinged voice chats
0:13:00 that have the same bias as me.
0:13:04 They love Trump and their, you know, the MAGA girl.
0:13:05 And I can talk to the MAGA girl
0:13:07 who has the same belief system that I have
0:13:09 or whatever, I’m not saying that’s my belief system.
0:13:11 I’m just, you know, for example.
0:13:13 And so it’s very, very concerning.
0:13:14 Like, I think you and me, Matthew,
0:13:16 are pretty much on the same page
0:13:19 where we generally lean optimistic on this stuff,
0:13:21 but there’s still quite a few things
0:13:23 that actually do scare me about this as well.
0:13:26 I’m not like the whole accelerationist
0:13:28 where I’m just like push forward as fast as possible.
0:13:30 I’m like, maybe there’s some things
0:13:32 we shouldn’t push forward as fast as possible on.
0:13:34 And that’s definitely one of those areas.
0:13:37 (upbeat music)
0:13:38 – Hey, we’ll be right back to the show.
0:13:40 But first I want to have another podcast
0:13:41 I know you’re gonna love.
0:13:43 It’s called Entrepreneurs on Fire
0:13:45 and it’s hosted by John Lee Dumas
0:13:47 available now on the HubSpot Podcast Network.
0:13:49 Entrepreneurs on Fire stokes inspiration
0:13:53 and shares strategies to fire up your entrepreneurial journey
0:13:54 and create the life you’ve always dreamed of.
0:13:57 The show is jam packed with unlimited energy,
0:13:58 value and consistency.
0:14:00 And really, you know, if you like fast paced
0:14:02 and packed with value stories
0:14:03 and you love entrepreneurship,
0:14:05 this is the show for you.
0:14:07 And recently they had a great episode
0:14:10 about how women are taking over remote sales
0:14:11 with Brooke Triplett.
0:14:13 It was a fantastic episode.
0:14:14 I learned a ton.
0:14:15 I highly suggest you check out the show.
0:14:17 So listen to Entrepreneurs on Fire
0:14:19 wherever you get your podcasts.
0:14:22 (upbeat music)
0:14:24 – Yeah, you know, Elon had said with Grock,
0:14:27 which I find kind of promising hopefully is that, you know,
0:14:29 they want to be maximum truth seeking,
0:14:30 which sounds a lot better to me
0:14:32 than when I’m hearing from open AI
0:14:33 of like bring your own bias.
0:14:35 Like we’re gonna allow you to pick your bias
0:14:38 and we’re just gonna serve you up information based on that
0:14:40 because all truth is subjective or whatever.
0:14:41 But I worried that, you know,
0:14:45 as obviously Elon has his own bias now as well, you know,
0:14:47 and, you know, I’m probably more right wing
0:14:48 than anyone on this podcast right now.
0:14:50 I’m not super right wing, but, you know,
0:14:52 after living in San Francisco for a long time,
0:14:53 that kind of did that to me.
0:14:55 And I do worry that he’ll go too far with it.
0:14:58 Grock will become like a right wing AI.
0:15:00 I do think it needs to be unbiased.
0:15:02 I’m not sure how you do that
0:15:03 ’cause there’s bias in everything.
0:15:04 Like if you pull up facts online
0:15:06 or you pull up something from the Wall Street Journal
0:15:09 or CNN or whatever, there’s bias in all of this.
0:15:11 And so since all the models are trained on that,
0:15:15 I’m just not sure how you get around the bias, you know?
0:15:17 – By the way, this is the part where I start
0:15:20 to get flabbed on Twitter and YouTube,
0:15:22 but for completely opposite reasons,
0:15:23 but I’m still gonna share anyways,
0:15:26 this is not reflective of my political opinion
0:15:29 or anything like that, but here’s a couple of things.
0:15:34 Yeah, Elon has said, “Maximally truth seeking for Grock 3.”
0:15:37 And great, like in theory, that makes a lot of sense.
0:15:39 But how do you do that?
0:15:43 Ultimately, you have to create systems
0:15:46 and maybe you’re able to create them or maybe you’re not,
0:15:48 but those systems are created by humans.
0:15:51 The original training data is created by humans.
0:15:55 The post-training techniques are created by humans.
0:15:58 The reinforcement learning is also set up by humans.
0:16:00 So there’s like a human in the loop.
0:16:04 So what he claims to be completely maximally truth seeking
0:16:06 is just what he believes.
0:16:08 And maybe not necessarily him,
0:16:09 but he’s trying to give a different perspective,
0:16:10 which, you know, fine.
0:16:13 But did you see what happened just a few days ago?
0:16:17 Grock 3 was caught having custom instructions that said,
0:16:20 don’t cite any sources that say Elon Musk
0:16:23 and Donald Trump are spreaders of disinformation.
0:16:25 And they were caught doing this.
0:16:28 And so I reported on it and of course,
0:16:30 flamed from both sides, but fine.
0:16:33 But the point is, yeah, it’s as simple as somebody
0:16:36 submits a PR and it has a little line
0:16:38 in the system prompt saying, don’t do this.
0:16:40 And then all of a sudden, fingers on the scale, right?
0:16:42 So it’s very possible.
0:16:44 It’s not only possible, it’s happening.
0:16:47 And it doesn’t matter if it’s from OpenAI or Grock
0:16:50 or Google or Anthropic, they all have bias.
0:16:53 I can’t imagine a world in which the bias
0:16:55 is removed completely, although I hope it does.
0:16:58 I just, I don’t see a path towards that.
0:17:00 – Yeah, I mean, I honestly don’t understand
0:17:02 how it’s possible.
0:17:04 Like you guys said, it’s all trained on data
0:17:05 that was created by humans.
0:17:09 It’s all basically a scraping of the entire internet,
0:17:11 which just inherently has bias.
0:17:14 Like I just, I don’t understand how it’s possible, honestly.
0:17:16 And how do you determine what is the truth
0:17:17 and what is not?
0:17:18 – That’s the question.
0:17:20 – I don’t know, maybe if the models get really smart
0:17:22 and can do like real reasoning
0:17:24 and that they can look at both sides of anything,
0:17:25 they can kind of find the middle ground
0:17:27 where there’s some truth, where there’s actual truth,
0:17:30 hopefully, but then some people will perceive
0:17:32 that real truth is not truth.
0:17:34 It’s like, well, people will perceive it to have bias
0:17:35 even if it ends up not having bias.
0:17:38 – No, but I feel like you’re getting into like this world
0:17:40 where now we’re letting AI decide what is
0:17:42 and what isn’t like ethical, right?
0:17:45 Like now we’re letting like AI sort of decide
0:17:49 what is like philosophically correct and what is not.
0:17:50 And to me, that seems weird
0:17:53 to let machines do that for humans, you know?
0:17:55 – Yeah, and, you know, it’s funny you say that,
0:17:58 I mentioned the same thing and Dave Shapiro,
0:18:01 another fellow YouTuber talking about AI said,
0:18:04 you know, actually, I would rather just completely
0:18:07 give it over to AI to make ethical decisions.
0:18:10 I think that’s what he was saying, but it’s interesting.
0:18:12 I can see both sides of it.
0:18:15 If there was a system that was completely unbiased,
0:18:18 assuming, right, yeah, okay, let them decide,
0:18:20 but how do you make that, as you said?
0:18:23 – I think you can’t allow AI to make that decision.
0:18:24 If you allow AI to make that decision,
0:18:26 I mean, that goes towards like eugenics
0:18:27 and crazy things like that.
0:18:30 Like you can’t allow AI to optimize based on
0:18:33 human performance or some crazy metric like that.
0:18:35 That would just lead to, you know, horrible things.
0:18:37 I think I totally disagree with that.
0:18:39 – Yeah, yeah.
0:18:42 Let’s shift over to Claude, ’cause Claude 3.7 came out
0:18:46 a couple of weeks ago and that one has proven to be
0:18:48 really, really good at code, it seems, right?
0:18:50 It seems like it didn’t make huge improvements
0:18:54 in almost any other areas, but it got a lot better at code.
0:18:56 From what I’ve seen so far, I mean, Matt,
0:18:58 you might have some different experiences with it.
0:19:00 I’m using cursor to write code
0:19:04 and it’s definitely gotten better at coding for me.
0:19:06 I’ve definitely noticed things that it would run
0:19:08 into these like loops where it wouldn’t write the code,
0:19:10 wouldn’t write the code, couldn’t figure out what
0:19:14 the problem was, 3.7, one prompt was 3.7,
0:19:16 now it figures out a problem that I had gone back
0:19:18 and forth 10 times prior.
0:19:21 So to me, it seems like it got a lot better at code,
0:19:24 but from most other people’s experiences,
0:19:26 it seems like it didn’t really improve
0:19:29 in almost any other areas other than code.
0:19:30 – Yeah, when I think about coding,
0:19:31 I think about Claude, right?
0:19:35 Claude 3.5 was the go-to model for a lot of coders
0:19:37 using AI assistance to help them code.
0:19:40 Yeah, cursor plus Claude, WinSurf plus Claude,
0:19:41 that was the model.
0:19:43 Now we had this huge upgrade and I agree,
0:19:44 it is a huge upgrade.
0:19:46 It also has the thinking capabilities.
0:19:49 So I’ve been doing stuff, vibe coding, right?
0:19:51 So I’m using some kind of IDE,
0:19:55 whether cursor or WinSurf plus I’m using 3.7 thinking
0:19:57 and it’s fantastic, right?
0:19:58 It is fantastic.
0:20:01 Now I’ll bring it back to what I said earlier,
0:20:03 the real world use cases,
0:20:05 the stuff that I want to use it for day to day,
0:20:08 one is coding, but a lot of other things are not coding.
0:20:11 And here’s the thing, it’s not that fast
0:20:12 and it doesn’t have web search.
0:20:14 It has no real time information.
0:20:19 So it’s essentially unusable for me outside of coding.
0:20:21 Although it’s fantastic at coding, I’ll give you that.
0:20:23 – Well, the nice thing is if you are using something
0:20:26 like cursor, cursor can actually do the web search for you
0:20:28 and then give that additional context.
0:20:31 So it’s almost like cursor will actually kind of do
0:20:33 like the perplexity thing, right?
0:20:35 Where if it needs to figure something out,
0:20:37 cursor itself goes and does the search
0:20:39 and then provides that information to Claude.
0:20:41 At least when I’ve been using it,
0:20:44 it seems to actually search the web when using cursor.
0:20:48 – Yeah, and Claude 3.7 Sonnet is available in perplexity,
0:20:49 I’ll just mention quickly.
0:20:51 By the way, perplexity just adds
0:20:53 the latest models all the time for free.
0:20:56 And I’ve not been paid by them at all.
0:20:58 I’m just a huge fan of their product.
0:21:00 So yeah, so if you wanted to try any of these new models,
0:21:03 you could go try it if you already have a perplexity account.
0:21:05 – Yeah, and their deep research is really good.
0:21:08 If we’re talking about good things about perplexity,
0:21:10 if we’re praising perplexity,
0:21:12 they just threw deep research in there as well,
0:21:13 and it’s really good.
0:21:15 – But yeah, web search is critical.
0:21:17 – Yeah, I wanted to say about Claude Sonnet.
0:21:19 Like my feeling is, you know, before it was released,
0:21:21 Claude was my go-to for just like general chat
0:21:23 for like discussing anything, right?
0:21:25 Like even like my game design document,
0:21:27 I would share that with Claude and that was my favorite.
0:21:28 Then Grock replaced that,
0:21:30 but Claude was pretty good at code.
0:21:33 Like you said, most engineers were using Claude Sonnet,
0:21:35 but not all, like I was using O1 Pro
0:21:37 and actually it was better, but way slower, right?
0:21:38 And you had to give it tons of context.
0:21:41 You could use something like repo prompt or something else.
0:21:42 But it seems like in this update,
0:21:44 they really doubled down on being the best at coding.
0:21:46 ‘Cause in terms of like general chat,
0:21:47 I feel like it actually stepped backwards.
0:21:49 Like when I use it now to chat with them,
0:21:51 like it actually got worse in this release like that.
0:21:53 I don’t like its responses as much.
0:21:55 They seem less human like, but it’s way better at coding.
0:21:57 So it feels like we’re starting to see
0:21:59 that all the AI models are finding their own like specialties
0:22:02 or at least that’s what Anthropics doing now with Claude.
0:22:04 And I do kind of wonder if all the models
0:22:05 are gonna have to do that.
0:22:06 You know, when I talked to the people at XAI,
0:22:09 I told them double down on having the best data.
0:22:11 ‘Cause you got the real-time data with X,
0:22:13 you got it with, you know, the search, double down on that.
0:22:15 And I think you’ll probably see that
0:22:16 where like these different models
0:22:17 will be the best at a thing.
0:22:21 But it seems like Chat2P is still trying to go more broad.
0:22:23 There’s trying to be like the best overall model.
0:22:26 And I’m kind of curious to see how all that ends up playing out.
0:22:27 – Yeah.
0:22:28 – It’s interesting, you know, quickly.
0:22:31 So I just pulled up the Claude 3.7 Sonnet blog post,
0:22:32 the announcement blog post.
0:22:35 And everybody knows Claude is great at coding
0:22:38 and they made this huge jump in coding.
0:22:41 But if you read it, it says,
0:22:42 in developing our reasoning models,
0:22:44 we’ve optimized somewhat less for math
0:22:46 and computer science competition problems.
0:22:50 And instead shifted focus toward real-world tasks
0:22:52 that better reflect how businesses actually use LLMs.
0:22:54 You know, now that I’m reading that,
0:22:57 maybe they met like these kind of benchmarky
0:23:00 computer science problems versus real-world coding problems.
0:23:02 But it just sounds like,
0:23:05 hey, we’re not focused as much on math and coding anymore,
0:23:07 more real-world stuff.
0:23:09 But it still is fantastic at that.
0:23:11 – Yeah.
0:23:12 – Did you guys see that they’re also,
0:23:15 it seems to be that they’re gonna be competing with cursor?
0:23:18 – Oh yeah, ’cause they released that code feature, right?
0:23:20 – Yeah, Claude code, all the top engineers,
0:23:21 I know they’ve tried it.
0:23:23 It’s very expensive to use,
0:23:24 but my understanding is like in some ways
0:23:25 it’s better than cursor.
0:23:27 So like, I’m not sure exactly how it works,
0:23:28 but I think you use it in the terminal,
0:23:31 you get full access to your entire code base,
0:23:33 and then it can just change stuff for you.
0:23:34 – Yeah, I tried it out.
0:23:35 It’s pretty cool.
0:23:35 That is interesting.
0:23:37 I think they really are doubling down on code.
0:23:38 – Yeah.
0:23:39 – I believe when I mentioned the benchmarks stuff,
0:23:40 I think they were probably talking about
0:23:42 more real-world coding versus like–
0:23:43 – I think you’re right.
0:23:44 – Benchmark coding, you know?
0:23:46 – Yeah, I think they know that most people
0:23:48 are using Claude mostly for coding.
0:23:49 – So it kind of sucks for cursor,
0:23:51 ’cause like right now everyone uses cursor to use Claude,
0:23:54 and now it’s roughly basically going to try to kill them.
0:23:55 – I don’t know.
0:23:57 I think people will probably still use cursor a lot,
0:23:59 because cursor and Windsor, if I believe,
0:24:02 are both forks of Visual Studio code.
0:24:02 – Yeah, they must be saying like,
0:24:04 oh, most of our usage is in cursor,
0:24:07 like why would we not like just be cursor then?
0:24:08 – Yeah, but I don’t know.
0:24:11 Like Visual Studio code is like pretty universal.
0:24:13 Like people use it for coding a lot,
0:24:15 and it’s what’s familiar to people.
0:24:18 So I think trying to get people to switch to a new IDE
0:24:21 might be a tough ask, unless Claude themselves goes
0:24:24 and makes their own fork of Visual Studio code.
0:24:25 So like unless Claude goes
0:24:28 and makes their own fork of Visual Studio code,
0:24:31 I have a hard time seeing like people switch over
0:24:34 to like a new IDE that’s like completely different.
0:24:36 – It’s also not an IDE.
0:24:38 It’s literally just sitting in the terminal.
0:24:41 So, you know, some people prefer that.
0:24:44 I prefer seeing the code more visually
0:24:46 and have like a nice interface to deal with.
0:24:49 So I’ve tried both cursor and Windsurf,
0:24:50 and they’re both great.
0:24:51 – Yeah, yeah.
0:24:53 To me, they feel very same-ish.
0:24:54 Like I have a hard time saying which one’s better.
0:24:56 They feel very, very similar to me.
0:24:59 – I’ve had more success with Windsurf
0:25:03 when trying to iterate on a whole code base.
0:25:05 Now cursor, I think just released,
0:25:08 they’re kind of just updated their agent feature,
0:25:09 which makes it a little bit easier
0:25:11 to operate on the whole code base.
0:25:15 And your AI coding assistant agent is able to grok
0:25:17 and search through the code and do different things.
0:25:20 That helps it work on the code base as a whole.
0:25:21 But yeah, you know what?
0:25:22 Competition’s always good.
0:25:23 – Yeah, agreed.
0:25:25 I’m trying to look for some other like really cool examples
0:25:27 here of stuff that people made with Claude.
0:25:28 I mean, these are all really cool examples.
0:25:31 We can spend 30 minutes looking at all these examples.
0:25:32 So I’m looking for the best ones right now.
0:25:35 I mean, lots and lots of really cool examples
0:25:37 of stuff that Claude just did in one shot.
0:25:38 Lots of snake games.
0:25:39 Lots and lots of snake games.
0:25:41 – I made a snake game, yeah.
0:25:43 Of course I did, and then I had–
0:25:44 – First thing I did when I tested Claude
0:25:46 was make a snake game, I still always do.
0:25:47 – Yeah.
0:25:49 – But did you guys see the actually good snake game?
0:25:50 The one where it’s like having a mental breakdown
0:25:51 as it’s escaping?
0:25:52 Did you see that?
0:25:54 – I think you were just showing that, yeah.
0:25:54 Self-aware, there you go.
0:25:56 – Oh, this one, yeah, yeah, yeah.
0:25:58 So the self-aware snake escape?
0:26:00 – Yeah, so that’s the one, yeah.
0:26:01 It’s like freaking out.
0:26:02 (laughing)
0:26:04 – It just said your brain will struggle with this,
0:26:07 and then the little text pops up on the screen
0:26:08 as the snake is moving around.
0:26:10 – Yeah, it’s like, what’s the snake thinking
0:26:11 as it’s trying to break out?
0:26:13 And it starts freaking out that it can’t break out.
0:26:14 (laughing)
0:26:16 – It says this was done with one prompt
0:26:19 plus a request to make special things happen faster.
0:26:21 So I don’t totally know what that means,
0:26:25 but that was apparently one-shotted to get that pretty wild.
0:26:27 Yeah, probably using the agent feature.
0:26:29 Actually Cursor, they tested the new Claude model,
0:26:31 and they suggested to use the agent feature.
0:26:33 And I tested it, it is pretty amazing.
0:26:35 I tried it on my game and it fixed a problem
0:26:37 I’d been trying to solve with O1 Pro
0:26:38 and hadn’t been able to,
0:26:39 but then it broke like two things.
0:26:40 So I’m like…
0:26:41 (laughing)
0:26:43 And then it was unable to fix the things that it broke.
0:26:46 So it’s like, there’s still serious limitations with this.
0:26:47 – Yeah, yeah.
0:26:52 – Yeah, I’m building a 2D turn-based strategy game right now.
0:26:54 And I’m a few hours in,
0:26:56 and now that I’m a few hours in
0:26:58 and a few thousand lines of code in,
0:27:00 it’s a little harder, right?
0:27:03 It takes longer for each iteration to add features.
0:27:05 There are more bugs popping up
0:27:07 if it changes one thing over here or something else changes.
0:27:09 Like I’ll ask it to change something
0:27:12 and then the entire game will look different
0:27:13 on the next turn,
0:27:14 even though I didn’t say anything about that.
0:27:17 So yeah, of course, there are some limitations.
0:27:18 We’re gonna get better,
0:27:20 especially as context windows grow.
0:27:22 That’s why I actually think maybe Google’s,
0:27:24 two million token context window models
0:27:26 are quite appropriate for coding.
0:27:29 I just, I don’t think anybody uses them for coding.
0:27:31 I could be wrong about that though.
0:27:32 – Well, I’ve tried using it
0:27:35 ’cause you can switch to Google’s models,
0:27:37 like their Gemini or their Gemma models
0:27:39 inside of Cursor, I believe.
0:27:40 And when I tried Google’s models,
0:27:42 they just didn’t perform as well as Claude.
0:27:45 So I always find myself going back to Claude.
0:27:46 I tried 01.
0:27:48 I haven’t really done a lot of coding with 01 Pro
0:27:49 because there’s no API, so…
0:27:51 – 01 Pro is the best in that situation right now.
0:27:52 Like, if you have a lot of context,
0:27:54 like my game project,
0:27:56 even if I remove a lot of other files
0:27:57 and just get to the basic scripts,
0:27:59 it’s like 100K of context.
0:28:00 – Yeah, but they don’t have an API yet.
0:28:02 So you can’t just use it straight in Cursor.
0:28:04 You have to use something like,
0:28:05 what’s it called, Repo Prompt?
0:28:07 – Repo Prompt, and there’s a few other things too
0:28:09 where you can combine your files into a file
0:28:10 that you just copy and pasted.
0:28:12 – Yeah, but I mean, I’ve got a PC,
0:28:13 Repo Prompt’s only on Mac,
0:28:15 so I can’t even use Repo Prompt if I wanted to.
0:28:18 But I mean, I could, I have a Mac, I just never use it.
0:28:21 So like, I can’t even use something like Repo Prompt for 01.
0:28:23 The other really good thing about the Google models
0:28:24 is like, I think they’re pretty much
0:28:26 the most inexpensive models.
0:28:27 So if you’re looking for like as cheap
0:28:29 as you can get, Google’s probably there.
0:28:31 I mean, Lama might even be a little bit cheaper,
0:28:34 but Google’s pretty damn cheap.
0:28:36 – Yeah, the people that I really trust,
0:28:38 their opinions on AI stuff,
0:28:40 they are telling me I’m missing out
0:28:42 by not using the Gemini models.
0:28:44 So there’s something there.
0:28:47 I just, you know, there’s only so much time in the day.
0:28:48 I haven’t really had a chance
0:28:50 to extensively test the Gemini models,
0:28:53 but I really should and I really need to get in there.
0:28:55 – I’m gonna give you guys a sneak peek real quick here
0:28:57 of what I’ve been building.
0:29:00 I’m building my video producer app.
0:29:02 Basically what I do is I load
0:29:04 all of these like various like interviews in here.
0:29:05 I’ve made these folders.
0:29:08 So like you’ve got like this interviews folder here
0:29:10 with just like tons of interviews.
0:29:13 Like here’s Rowan who’s interviewed Mark Zuckerberg
0:29:16 and Logan and Mustafa Solomon and Demis Hassabis.
0:29:19 I’ve basically scraped a whole bunch of like interviews
0:29:23 and panels and launch videos and keynotes
0:29:24 and, you know, Lex Friedman interviews
0:29:26 and all sorts of like interviews
0:29:29 with various like AI leaders and stuff, right?
0:29:31 And inside of each one of these,
0:29:33 I actually use OpenAI’s Whisperer
0:29:35 and it transcribes the whole thing for me.
0:29:37 So I have like the entire transcription,
0:29:39 but then I also have like videos
0:29:41 that don’t have any audio in them.
0:29:43 Like I’ve got like some B roll footage and stock footage.
0:29:46 Like here’s stock footage of like a robot.
0:29:48 And for this one, I actually use Google Gemini
0:29:50 and Gemini watches the whole video for me
0:29:52 and writes up a description of everything
0:29:53 that’s going on in the video.
0:29:57 And the idea being I just have this giant database
0:29:59 of videos where I throw the video in
0:30:02 and then I can search out anything I want.
0:30:04 So anytime like Sam Altman is mentioned in a video,
0:30:06 I can search out Sam Altman.
0:30:07 It’ll pull up all of the videos
0:30:09 that either have Sam Altman in them
0:30:10 or they mentioned Sam Altman.
0:30:13 And then I could quickly find exactly in the transcript
0:30:14 where he’s talked about.
0:30:15 – You’re building this yourself?
0:30:18 – All with cursor and Claude 3.7.
0:30:20 Dude, I would totally use this.
0:30:23 I’m using Notion for the almost same thing.
0:30:25 I’m basically just anytime I find a clip
0:30:26 that I would find useful in the future,
0:30:27 I’ll throw it in there.
0:30:29 And I just have to remember where it is
0:30:30 and what the context was.
0:30:32 This is super useful, man.
0:30:33 I would pay to use this.
0:30:35 – Yeah. So I’ve got like B roll that I shot.
0:30:37 This is actually B roll that you’re probably
0:30:39 in the background of if you look closely enough
0:30:41 ’cause this is at the NVIDIA event here,
0:30:43 the, you know, the little digits box.
0:30:44 I threw this video in here
0:30:46 and you can see it wrote this description.
0:30:47 This short video showcases
0:30:49 the NVIDIA project digits prototype
0:30:50 to compact computing device.
0:30:53 The video primarily focuses on the physical device itself.
0:30:55 So it goes into all of this detail
0:30:59 from a 31 second video of like me getting B roll digits.
0:31:00 So now if I ever, I’m like,
0:31:03 “Oh, what was that video I made that had digit in?”
0:31:04 And I need to pull that up really quick.
0:31:05 I could just search up digits, right?
0:31:08 And it’ll pull up this video as the top video.
0:31:11 So this is like sort of phase one of what I’m building here.
0:31:14 Phase two is I want to toss all of this
0:31:17 into like a rag like retrieval augmented generation model
0:31:20 where I can say, “Hey, I want to make a video
0:31:21 about Sam Altman.
0:31:23 Compile everything we know about Sam Altman
0:31:25 from all of these videos
0:31:27 and have it actually write like an outline for me
0:31:29 based on all the information that’s in these videos.”
0:31:30 – That’s so cool.
0:31:32 – So that’s what I’ve been building.
0:31:34 And this again, I’ve been working on for about two weeks now
0:31:37 and it uses like seven different APIs.
0:31:40 It’s using like the OpenAI whisper API.
0:31:42 It’s using the Gemini API
0:31:44 ’cause that can actually watch videos
0:31:46 and tell you what’s going on in the video.
0:31:49 It’s using Google’s cloud intelligence API
0:31:52 that’s actually able to like OCR any text in the video.
0:31:54 So if you’re watching like a slide presentation,
0:31:56 it can actually OCR any of the text
0:31:58 that’s in the slide presentation.
0:32:01 But yeah, it’s been a fun project to build.
0:32:03 But I run into the same kinds of stuff that Matthew,
0:32:05 you mentioned where I will go and ask it
0:32:07 to change like one feature.
0:32:09 I’ll be like, “Hey, the search isn’t working quite right.”
0:32:10 And it’ll be like, “Okay, I just fixed it.
0:32:11 I refreshed the page.”
0:32:13 And it changes the entire styling.
0:32:15 And I’m like, I didn’t ask you to touch the CSS at all.
0:32:17 I just wanted to change how the search functions.
0:32:19 Like what the hell?
0:32:21 But other than those little things,
0:32:22 I use GitHub a lot too.
0:32:25 It’s like every time a little change works,
0:32:27 I push it to GitHub so I know I can bring it back
0:32:28 if I need to.
0:32:29 – Oh, that’s smart, yeah.
0:32:31 – But yeah, it’s been fun.
0:32:35 And I’ve made less YouTube videos that I normally make.
0:32:37 I’ve only been putting out one YouTube video a week
0:32:38 for the past like month
0:32:40 because I’ve gotten so addicted
0:32:42 to playing around with AI coding.
0:32:43 It’s so fun.
0:32:44 Like I’m working on the game now.
0:32:46 I’m just like shocked that I can like build a game by myself.
0:32:48 It’s like, you know, in the past,
0:32:50 I never could imagine like one person could build a game
0:32:52 or now you’re building your own software product.
0:32:55 And Matt, you’re kind of like a no-code guy, right?
0:32:59 Now you’re using, you went from no-code to using AI to code
0:33:00 and you’re actually able to build a whole product.
0:33:02 I mean, it’s just, everything’s changing.
0:33:03 – Yeah.
0:33:05 And I mean, the thing is I’m learning as I go too, right?
0:33:07 ‘Cause like when you use these models,
0:33:10 it’ll explain to you what it did, what it changed.
0:33:12 You know, why something broke.
0:33:14 3.7 has been really, really good at that.
0:33:16 I don’t know if it’s 3.7
0:33:18 or if it’s like the agent feature inside of cursor.
0:33:21 But when it fixes stuff, it’ll explain the problem.
0:33:23 It’ll say you were running into this problem
0:33:24 because this, this and this was happening
0:33:26 or there was a conflict with this and this
0:33:29 or you know, it was sending the wrong information
0:33:30 through the API or whatever, right?
0:33:32 It gives you that information.
0:33:34 So although I don’t actually know
0:33:36 how to like type out the code myself,
0:33:39 I’m getting a lot better at troubleshooting
0:33:42 why problems are happening within the code.
0:33:45 I don’t know how to like actually change the code.
0:33:48 Like I don’t know what to write to make it work myself,
0:33:50 but I’m starting to pick up on like,
0:33:52 oh, I think this might conflict with this
0:33:54 as a result of learning
0:33:56 as it explains to me all of these problems.
0:33:59 So anyway, that’s what I’ve been working on.
0:34:01 But there is one last topic I do wanna shift in.
0:34:02 – Let’s talk about it.
0:34:04 – I wanna shift over to GPT 4.5
0:34:08 because as of today’s recording, GPT 4.5 came out.
0:34:10 Before we hit record, I asked Matt,
0:34:11 I’m like, what were your thoughts on that launch today?
0:34:13 And he’s like, I’ll save it for the recording.
0:34:15 I was like, all right, let’s save it for the recording.
0:34:16 So let’s start there.
0:34:20 What are your thoughts on the GPT 4.5 launch?
0:34:21 – So I think it looks cool.
0:34:24 I haven’t obviously tested it extensively.
0:34:25 It came out just a handful of hours ago,
0:34:28 not even since recording.
0:34:30 So Nathan, you mentioned it’s really good at writing.
0:34:32 So I’m excited to test that out.
0:34:34 But let me just talk about a couple of things
0:34:37 that I noticed from the live stream.
0:34:40 So one, it’s the largest model that they’ve ever made.
0:34:44 And it took new innovations on both the pre-training
0:34:47 as well as the serving of it, the inference,
0:34:48 to actually be able to serve this model.
0:34:51 And if you use it, it is pretty slow, right?
0:34:53 So I found that pretty interesting.
0:34:55 It’s a world knowledge model,
0:34:58 meaning it is not a thinking model,
0:35:00 but in terms of just questions and answers
0:35:02 that you would use kind of day to day,
0:35:03 it’s really good at that.
0:35:05 And it’s much better than GPT 4.0.
0:35:07 And then the last thing,
0:35:08 and I think this flew under the radar a bit,
0:35:10 and I wanna get your guys’ thoughts on this.
0:35:13 They said that it was such a massive model
0:35:17 that they actually trained it across multiple data centers,
0:35:19 not in a singular location.
0:35:21 So when you think of the GROC model,
0:35:24 it’s the Colossus data center, 100,000,
0:35:27 something like 200,000 GPUs in a single place.
0:35:28 They didn’t have that.
0:35:29 They being open AI, they don’t have that.
0:35:32 And so they had to split it up.
0:35:35 And I don’t think any other company has done
0:35:38 parallel training across multiple data centers
0:35:40 at this level before.
0:35:42 I think that it really flew under the radar
0:35:44 and unlocks the ability for companies
0:35:49 that don’t have the money or resources that an XAI does
0:35:51 to go out and spread their model out
0:35:53 and still get a massive model trained.
0:35:55 And I thought that was just fascinating.
0:35:57 – Yeah, I didn’t catch that.
0:35:58 I mean, it’s not something that I picked up on
0:36:00 until you just mentioned it.
0:36:02 So I mean, that is really interesting.
0:36:05 So you’re saying that like, basically,
0:36:07 there was separate physical locations
0:36:09 where the training was happening simultaneously
0:36:11 in separate locations.
0:36:15 – Yeah, ’cause they probably literally could not get
0:36:19 a large enough data center to train this one model
0:36:23 in a concentrated location like Colossus with GROC3.
0:36:26 So they had to split it up and do it in parallel.
0:36:28 And again, that really hadn’t been done before
0:36:30 at this level.
0:36:31 – Yeah, well, I mean,
0:36:33 we’ve got Project Stargate coming as well.
0:36:34 That’s gonna be the data center
0:36:36 where eventually they’ll be able to do it all
0:36:38 from one data center, I believe.
0:36:40 But yeah, no, that’s really, really interesting.
0:36:43 I personally found the actual presentation
0:36:45 a little bit underwhelming.
0:36:48 They didn’t really show off any sort of capabilities
0:36:51 that were like, oh, we’ve never seen that before.
0:36:53 – Considering this was like hyped up for like a year
0:36:55 of like, oh, Orion’s coming.
0:36:57 – Well, and then also Sam Altman,
0:36:59 he posted on Twitter a couple of days ago
0:37:02 that a lot of people who have used GPT 4.5
0:37:05 have gotten that feel the AGI moment from it, right?
0:37:08 But anybody really feel an AGI moment from their demo.
0:37:10 I don’t know, I’m reading into it a little too much.
0:37:13 But the other thing I realized too is like,
0:37:15 you can always tell when OpenAI doesn’t see it
0:37:16 as that big of an announcement,
0:37:18 when Sam doesn’t show up for the announcement.
0:37:19 – Right.
0:37:22 You know, I think there’s more there than people realize.
0:37:25 And so many people accuse me of being overly optimistic.
0:37:28 So bear with me, or at least take what I’m gonna say
0:37:29 with a grain of salt.
0:37:30 I really do think that there’s more there.
0:37:34 First of all, this is the first version, right?
0:37:35 And we’re gonna get lots of different upgrades over time.
0:37:37 We’re gonna get a turbo version.
0:37:40 But I think in terms of just a baseline model,
0:37:42 it’s a lot better at the kind of general Q and A.
0:37:45 And so from that, when you have all of that world knowledge
0:37:47 baked into this model,
0:37:49 how do you think the thinking versions,
0:37:52 the O1s, the O3s, how do you think those are the created?
0:37:54 It’s taking that foundation model,
0:37:56 using reinforcement learning,
0:37:59 and then kind of eliciting that thinking behavior from it.
0:38:02 So now we’re gonna have this incredible foundation model
0:38:05 to build the thinking models on top of.
0:38:07 Maybe that’s what O3 Pro is.
0:38:09 – And it’ll be crazy expensive.
0:38:11 – Oh, that’s another thing is the cost, yeah.
0:38:11 – You combine those two.
0:38:13 It’s like, okay, this model is already very expensive
0:38:15 before you do the thinking.
0:38:17 And so when you do, it’s like, it makes me wonder,
0:38:20 is that gonna lead to like the $2,000 a month plan
0:38:22 or something like this?
0:38:23 You know, there’s also another angle to think of it,
0:38:26 which is, it’s all about the vibes, right?
0:38:28 That was like a huge theme of the announcement of the vibes.
0:38:30 It’s a warm model.
0:38:32 It’s a high EQ model.
0:38:33 – They were trying to compete with Claude.
0:38:34 – Right, and so–
0:38:35 – You know, ’cause before Claude, everyone said like,
0:38:38 Claude had a better vibe than chat to BT, you know?
0:38:40 – Right, and so like, if you think about it
0:38:42 from that perspective,
0:38:44 they’re really positioning this model
0:38:48 to be a true kind of AI personal assistant.
0:38:50 And I emphasize the word personal.
0:38:53 It is there to help you.
0:38:55 It is there to know you.
0:38:58 And I think this is really more interesting
0:39:00 than people are giving it credit for.
0:39:03 Because when you think about systems like Siri,
0:39:04 what it could be,
0:39:08 it’s probably based on something like this type of model.
0:39:10 – Yeah.
0:39:11 – One thing I just wanted to point out too
0:39:13 is I don’t know if you guys caught this tweet,
0:39:16 but he basically said that the new model
0:39:18 is a giant expensive model,
0:39:19 and they really wanted to launch it,
0:39:21 but they ran out of GPUs, right?
0:39:24 And this kind of comes back to Matthew’s point earlier
0:39:26 of how they had to like train it, you know,
0:39:30 in separate locations with this parallel processing.
0:39:33 They’re literally like reaching the end
0:39:36 of the available GPUs to be able to process this stuff.
0:39:38 And I just found that really, really interesting.
0:39:39 So I wanted to share that real quick.
0:39:41 – Open AI is GPU poor.
0:39:45 – Yeah, I mean, they’ve got SoftBank now behind them.
0:39:47 – Who would have thought?
0:39:48 – I want to share two things that kind of show
0:39:50 that there is something special here.
0:39:52 So here’s something I saw that’s pretty interesting
0:39:55 from a professor or he’s a biomedical scientist.
0:39:56 He said, “It appears to be remarkable
0:39:57 in medical imaging diagnosis.
0:40:00 It was the only model that perfectly diagnosed
0:40:01 this ultrasound image.”
0:40:03 So like in terms of like looking at ultrasound,
0:40:06 apparently it’s by far the best model out there.
0:40:08 So there probably are things that we’re going to find
0:40:09 about this model that are special
0:40:12 that when you first use it are not apparent.
0:40:13 – Yeah, good point.
0:40:13 No, I agree.
0:40:15 I’m sure it’s a lot better model
0:40:18 than I feel like their presentation led on to be.
0:40:20 Part of the problem was like,
0:40:22 we all watched the live stream, I’m sure.
0:40:25 They seemed like they were kind of nervous.
0:40:26 They were kind of uncomfortable.
0:40:29 Sam wasn’t on the live stream this time.
0:40:31 I just don’t think it was presented very well, honestly.
0:40:32 I think that was probably the problem.
0:40:34 – Yeah, here’s another thing I saw
0:40:35 that was super interesting.
0:40:37 Like the hallucination rate is like way lower
0:40:38 with like 4.5.
0:40:42 So it’s like 37.1% hallucination rate
0:40:45 versus 4.0 being 61%, you know,
0:40:47 0.3 mini being 80.3%.
0:40:48 It’s like dramatically lower.
0:40:50 So that’s a, yeah, that’s step forward.
0:40:51 ‘Cause like obviously for big companies,
0:40:53 one of the big problems with using AI models
0:40:55 is hallucinations, right?
0:40:56 So yeah.
0:40:56 – Yeah, absolutely.
0:40:57 That’s crazy.
0:41:01 0.3 mini on simple QA has an 80% hallucination rate.
0:41:03 That seems insane to me.
0:41:04 – Seems right.
0:41:05 ‘Cause like 0.3 mini is like really fast,
0:41:07 but I’ve seen it make lots of weird mistakes.
0:41:09 Like when I tried to use it for coding and stuff,
0:41:10 it’s like, oh, you’re a genius.
0:41:12 And the next I’m like, oh, you’re a total moron.
0:41:15 It’s just like, it’s responses are like all over the place.
0:41:16 – I don’t quite understand that benchmark.
0:41:19 I feel like an 80% hallucination rate
0:41:21 just makes, would make something unusable.
0:41:23 I mean, if you’re only getting accurate information
0:41:25 20% of the time, I don’t know.
0:41:28 I don’t totally understand how this one is come to, I guess.
0:41:29 – I think we would have to see what the actual questions
0:41:33 are in the simple QA benchmark to understand the context
0:41:34 of why it’s scored so poorly.
0:41:36 But then, but then just, you know,
0:41:40 I guess maybe remember it’s mostly for math, science,
0:41:42 like basically STEM, right?
0:41:44 Things that have verifiable answers.
0:41:45 – Absolutely.
0:41:46 – I think it’s cool.
0:41:47 I think it’s a good start.
0:41:49 – Yeah, no, I’m excited about it.
0:41:51 I mean, like, hey, for content creators like us,
0:41:53 all this news is amazing, right?
0:41:54 ‘Cause we get to talk about it.
0:41:56 We get to, you know, keep on sharing what’s coming out.
0:41:59 We get to play with it all and get access to it
0:42:01 and show what it’s capable of
0:42:03 and put it through its motions and stress test it.
0:42:06 And I think, you know, I could speak for myself.
0:42:07 I think I could speak for Matthew.
0:42:08 That’s what we love doing, right?
0:42:10 Like we love playing with the stuff and stress testing it
0:42:12 and figuring out what it really can do.
0:42:15 And that’s what’s most exciting to me, I think.
0:42:19 You know, GPT-5 is probably only six, eight weeks away.
0:42:22 I mean, it’s not that far off, apparently.
0:42:24 So this is just the beginning.
0:42:27 It’s been a crazy couple of weeks.
0:42:28 – Absolutely.
0:42:28 – So who won?
0:42:29 Who won the last week?
0:42:30 – I don’t know.
0:42:33 I mean, they all have different pros and cons, I guess.
0:42:34 Right?
0:42:35 – I was most surprised by Grock.
0:42:36 Right, like out of all the three,
0:42:39 like I feel slightly disappointed by 4.5.
0:42:40 I would say it would be my general feeling.
0:42:42 Not like majorly disappointed,
0:42:43 but like Orion’s not as big of a deal
0:42:45 as I was hoping it was going to be.
0:42:45 – Yeah.
0:42:47 – Grock really impressed me.
0:42:50 Anthropic Clawd, you know, 3.7, amazing.
0:42:52 Kind of what I expected in terms of improvement.
0:42:54 So I would say Grock’s the biggest surprise
0:42:55 out of all the three.
0:42:56 – Yeah.
0:42:57 – I mean, personally, I’ve gotten the most use
0:43:01 out of Clawd 3.7 because I’ve been really, really going
0:43:03 down the coding rabbit hole lately,
0:43:05 but that’s just a very like anecdotal thing for me, right?
0:43:08 Like that’s the use case that I’ve found the most valuable
0:43:10 in the moment is I’m doing a lot of coding
0:43:12 and 3.7 is great for that for me right now
0:43:14 in this moment in time.
0:43:18 – Yeah, with the caveat that I haven’t tested GPT 4.5 much,
0:43:20 I gotta give the crown to Grock 3
0:43:23 over this last wave, these last couple of weeks.
0:43:25 It went from I never used Grock 2
0:43:28 to now it is my go-to model for a lot
0:43:30 of use cases on my day-to-day tasks.
0:43:32 So definitely have to give it to Grock 3 there.
0:43:33 – Yeah.
0:43:35 Now we just need that damn API.
0:43:36 – Yes, yeah.
0:43:37 Then we’ll really–
0:43:38 – It’ll be interesting to benchmark it
0:43:39 when the API comes out, right?
0:43:41 Like see how it actually compares
0:43:42 to all these other models.
0:43:43 – Yeah, yeah.
0:43:44 Well, cool, Matthew.
0:43:45 This has been amazing.
0:43:47 You know, anybody listening, make sure you check out
0:43:50 Matthew’s YouTube channel over at MatthewBurman.
0:43:51 You’ve got an amazing newsletter.
0:43:52 I’m subscribed to it.
0:43:54 It’s the forward future newsletter, I believe.
0:43:55 – Forward future.
0:43:57 – Everybody needs to go check those out anywhere else.
0:44:00 You want people to go check you out and follow you online?
0:44:04 – Yeah, come check me out on Twitter @MatthewBurman.
0:44:07 Come flame me for my opinions on politics,
0:44:08 even though I don’t share them.
0:44:08 – All right, you’re asking for it?
0:44:09 – Yeah, yeah.
0:44:10 – Well, you called it Twitter.
0:44:11 That’s like, you’re kind of like your DM.
0:44:12 – All right, you’re right.
0:44:13 I already offended one side.
0:44:15 – So like, look, I’m on your side, guys.
0:44:16 – Well, cool.
0:44:17 This has been super, super fun.
0:44:19 – Thank you for having me, guys.
0:44:19 Thank you.
0:44:20 – Great having you back.
0:44:23 I’m sure you’ll be back on if you want to be.
0:44:24 We’d love to have you again.
0:44:26 It’s always fun to chat with you
0:44:29 and really appreciate you spending the time with us today.
0:44:30 – Yeah, thank you, guys.
0:44:33 (upbeat music)
0:44:35 (upbeat music)
0:44:38 (upbeat music)
0:44:40 (upbeat music)
0:44:43 (upbeat music)
0:44:46 (upbeat music)

0:00:06 Hey, welcome to the Next Wave Podcast. I’m Matt Wolff. I’m here with Nathan Lanz. And
0:00:10 today we’re going to talk about probably one of the hottest topics of 2025. And that’s
0:00:16 the topic of AI agents. Today we’re bringing on a special guest, Dmitri Shapiro. He is
0:00:22 the CEO of Mind Studio. And Mind Studio is a tool that makes it insanely easy for you
0:00:26 to build your own AI agents. So in this episode, not only we’re going to talk about AI agents
0:00:31 and what they’re useful for, we’re going to actually jump into Mind Studio with Dmitri
0:00:37 and show off him actually building some of these agents in real time and making them
0:00:41 work. It’s pretty impressive stuff. And I don’t want to waste your time. So let’s just jump
0:00:46 right in with Dmitri.
0:00:50 Before we get into talking about Mind Studio and all the cool stuff that it can do, let’s
0:00:54 get a little bit of background on you. I know you and I are practically neighbors. You live
0:00:59 maybe three miles from me. We’ve met up for coffee before. And when we hung out, you told
0:01:03 me about all of the sort of stuff you did before you built Mind Studio. And I just found
0:01:08 it fascinating because you worked at MySpace. You worked at Google. You did all sorts of
0:01:12 really, really cool stuff. And you’ve been sort of in the thick of the AI world for quite
0:01:17 a while. So can you give us that quick rundown of what you’ve done in AI and in the tech
0:01:18 world so far?
0:01:23 Ah, totally. Yeah. Great to see you again. And I love being your neighbor. These days
0:01:30 I describe myself as an old nerd. I started writing code in 1984 when I was 14 years old
0:01:34 and I went to see the movie War Games. And I came out and was like, I got to get a computer.
0:01:38 And we were like dirt poor so we couldn’t afford a computer. But my high school had two
0:01:44 Apple 2e computers. And so I spent the rest of my high school years instead of being a
0:01:49 normal teenager in this like computer lab. And you’re mostly trying to like write code
0:01:54 and cause havoc and whatever, you know, way that I could start World War III. Mostly like
0:02:01 show my friends, you know, versions of like very primitive viruses. And then I studied
0:02:05 electrical engineering, got a degree in electrical engineering. I’ve never done a day of it always
0:02:10 done software. And then, you know, had a long career in software before, you know, founding
0:02:15 Mind Studio. I spent four and a quarter years at Google on the main campus. There I was
0:02:21 on the product side. I started on Google plus, when Google felt that Facebook was an existential
0:02:27 threat. And like all the efforts went into standing up what they called the social spine
0:02:32 or the social backbone of Google. That’s what was called Google plus, make all of Google
0:02:39 social. And my job was to build the social graph, which like the most important part
0:02:44 of any social network. And even though Google has tremendous amount of information on users,
0:02:49 it was actually struggling to put together like things like you may know, you may like
0:02:54 feed rankings and things of that sort. And so my job was to run product on these like
0:02:59 three mostly machine learning teams that were crunching through all this Google data and
0:03:03 trying sort of like compute these lists and also creating a signal to abuse because whenever
0:03:08 you open up social networks, you know, abuse goes crazy. And in all of that as a function
0:03:13 of that before that, I was the chief technology officer of MySpace Music. This is the joint
0:03:19 venture between MySpace and the record labels. So I did that before that I built a venture
0:03:23 back company called VO Networks, which was one of the major competitors to YouTube in
0:03:28 the first five years of YouTube’s life. In fact, we had video recommendations before
0:03:32 YouTube had video recommendations. So that was very early. And of course, that was all
0:03:37 machine learning stuff. Before that, I built a venture backed cybersecurity company enterprise
0:03:43 software cybersecurity company called Aconyx, raised $34 million for Aconyx raised $70 million
0:03:49 for VO. Here so far, we’ve raised $36 million, like raised $140 some odd million dollars
0:03:53 in my career. And then what are before that I spent five years at Fujitsu and other big
0:03:58 Japanese companies like building their web team. So I’m an old nerd. I love building
0:04:03 products. This is the most fun I’ve ever had, which I think should say a lot because I’ve
0:04:07 been involved in a lot of like really, I thought like really interesting and fun things but
0:04:13 like, wow, what a time to be alive and working on products to be able to like leverage like
0:04:18 these new capabilities. It’s crazy. Yeah, absolutely. You raised a lot of money over
0:04:22 three different projects, right? Was most of that from Western VCs like Silicon Valley
0:04:28 VCs or like strategic partners or like, where did most of that come from? Both. So Aconyx,
0:04:34 I raised $34 million. The vast majority of it was from VCs, top tier and VCs like Mandel
0:04:39 Ventures was in there, which looked top tier VC and sort of second tier VCs at VOI raised
0:04:47 $70 million, a lot from VCs, but also like Intel, Adobe, the two former heads of Viacom,
0:04:52 John Dolgen and Tom Preston, all of Sifrin, Bretonham law firm, which like the most important
0:04:57 law firm in entertainment, Michael Eisner, the former head of Disney, Michael was on
0:05:01 my board of directors. There was a lot of strategic time Warner was an investor. So a
0:05:08 lot of strategic synod deal along with VCs. Here it’s primarily VCs and lots of angels,
0:05:13 lots of like senior Googlers. Because of my time at Google, I knew who I was, I laughed,
0:05:17 I started doing interesting things. They all sort of showed up and said, can I ride with
0:05:20 you? Yeah, I’ve heard there’s like a huge angel network of like ex Googlers. Yeah. And
0:05:25 so yeah, so it’s a bunch of nerds, you know, people like me are mostly nerds.
0:05:32 Well, so now you’ve got mine studio and you know, the big buzzword of 2025 is agents, right?
0:05:36 Like everybody’s talking about agents that you hear open AI and Google and all of these
0:05:39 companies basically saying agents, agents, agents, this is what you’re going to hear
0:05:45 the most about in 2025. And mine studio to some degree is basically a tool that helps
0:05:50 you build these agents. But I’m curious, like, I feel like everybody kind of defines agents
0:05:55 differently. Like how do you define agents? What are you calling? What you’re building
0:05:59 at mine studio agents? Like, where do you stand on all that?
0:06:05 Yeah, I think there’s a tremendous amount of sensationalism and hype and noise. And I’m
0:06:11 mostly allergic to all of it. And so even though we call them agents, we’re kind of
0:06:14 allergic to the word agent because it’s it’s so overloaded.
0:06:18 Yeah, but it’s kind of the word that people know and it’s the buzzword right now, right?
0:06:21 So I’m not sure what else to call these things. For a while there, we call them workers just
0:06:26 to differentiate from agents. But here’s how we think of them, I think differently than
0:06:30 majority of the people in when they use the term agent, I think kind of the popular definition
0:06:37 of it is an AI that can sort of direct itself and do what it needs to do in sort of quite
0:06:44 amorphous situation. It has access to tools. And it has a goal and it goes off and does
0:06:48 that. Like, magically, it’s intelligent. And it’s able to do that like a human would like
0:06:52 I’ll figure it out, right? Basically, it takes the approach of like, give me some task,
0:06:56 I’ll figure out like what to do about it. And as a nerd, like I’m excited about that kind
0:07:01 of vision. And certainly, we’re all moving in that direction. But at the same time, we
0:07:08 think that’s sort of a bit too much autonomy to be just given to AI and that like for most
0:07:13 things that you want to do, you generally want to be able to be in like more control.
0:07:18 You want it to do the heavy lifting and the scaling, but not necessarily to do like all
0:07:23 the strategy, for example, for you and like choosing what paths to take. And you’re sort
0:07:27 of telling it where to go and it goes there, rather like it, it takes you for a ride to
0:07:33 wherever it is that it decides to go. And so we think of these things as being workflows
0:07:37 that you can say, I want to get something done, I want to create something that does something.
0:07:41 And well, how would you do that? Well, you create a workflow like same way that you would
0:07:44 sort of like onboard a human to do it, you would explain to this thing, here’s what I
0:07:50 want you to do. Like every weekday morning at 9am, I want you to go and search the web
0:07:55 and search these other databases, looking for developments and this thing that I care
0:08:00 about. And you can be quite fuzzy in saying what the thing is, you can be quite precise.
0:08:05 And then if you find that kind of situation, if you find interesting information there,
0:08:09 do some thinking, you know, synthesize it, evaluate it from my standpoint, you know what
0:08:14 I want. And then sort of create a nice report for me and sort of nudge me, send me an email,
0:08:18 send me an application, tell me, hey, something just happened in the world that you should
0:08:22 know about, I’ve thought about it for you, here you go. And you can just set it on a
0:08:26 schedule that’s like one of the ways that you can deploy these MindStudio AIs and it
0:08:29 just runs and does that for you. By the way, that’s amazing.
0:08:35 Like I’ll give you like an example here, like, so Google Trends is a really sort of important
0:08:40 living data set of what are people searching for? Like what does the world care about? Or
0:08:45 what is like the world that you care about? What do they care about? And more specifically,
0:08:49 like how is that changing over time? Like what’s becoming more popular, what’s becoming
0:08:54 less popular? So the information is there on Google Trends. But the interface for you
0:09:00 to like load it in your browser and like periodically go look at Google Trends, that takes a lot
0:09:04 of work. And so you’re going to look at like one or two keywords, but you’re certainly
0:09:09 not going to look at 100 keywords. You certainly wouldn’t look at 1000 keywords. But you see
0:09:15 with the MindStudio AI agent, you can look at 1000 keywords. And you don’t have to do
0:09:19 any of it. You just have to go and say, you know, when you build the agent, like, this
0:09:23 is what I want to look at, it can generate the keywords or you can generate the keywords.
0:09:29 And it can at scale, every day or every hour, every minute you decide, go and look at the
0:09:34 world and synthesize it and look for those changes you want. If you’re trading stocks,
0:09:37 that’s an unfair advantage, right? If you’re starting companies as well, that’s a big thing
0:09:41 with like product studios is like, it’s an unfair advantage in everything. Yeah, exactly.
0:09:47 Yeah. And so again, before now, like that, how would you do that? But now that’s actually
0:09:53 really straightforward, you know, or you find something, you know, online that you’re reading,
0:09:57 you’re consuming, but it’s hard to tell whether that thing is accurate, you know, more and
0:10:02 more things are inaccurate, misinformation, disinformation. Now we’ve got a little button
0:10:06 you can press and like, we’ve got a Chrome extension. And you can have these agents
0:10:10 triggered from any web page. And so you can push a button and basically takes that web
0:10:16 page and then does deep research, like deep Google searches using like very sophisticated,
0:10:21 you know, Google dorks, Google search operators and composite all the data evaluates and can
0:10:24 return back to you like, these things are true, these things are misleading, these things
0:10:29 are false. And if you’re making business decisions on that or, you know, personal decisions,
0:10:33 it’s valuable to be able to have that kind of superpower. That’s actually kind of the
0:10:38 way I’ve mostly been describing these things when I talk about them. Is that called a superpower?
0:10:42 Like AI superpower, because that’s exactly what they are. And they give you an unfair
0:10:47 advantage if you’ve got the right ones, and you use them in the right way.
0:10:52 Hey, we’ll be right back to the show. But first I want to talk about another podcast
0:10:56 I know you’re going to love. It’s called entrepreneurs on fire. And it’s hosted by John
0:11:01 Lee Dumas available now on the HubSpot podcast network. Entrepreneurs on fire stokes inspiration
0:11:05 and share strategies to fire up your entrepreneurial journey and create the life you’ve always
0:11:11 dreamed of. The show is jam packed with unlimited energy, value and consistency. And really,
0:11:16 you know, if you like fast paced and packed with value stories and you love entrepreneurship,
0:11:21 this is the show for you. And recently they had a great episode about how women are taking
0:11:26 over remote sales with brook triplet. It was a fantastic episode. I learned a ton. I highly
0:11:30 suggest you check out the show. So listen to entrepreneurs on fire wherever you get
0:11:33 your podcasts.
0:11:38 Yeah, well, let’s dive in and actually take a look at that mind studio and some of the
0:11:41 stuff that you’re building. You know, if anybody who’s listening on audio, we are going to
0:11:46 be sharing this on YouTube as well. So you can go and check out what Dimitri’s sharing.
0:11:51 But I know you guys just rolled out a pretty new feature that works inside of Chrome where
0:11:56 you can build all these automations directly inside of Chrome. And so I thought it’d be
0:12:00 kind of cool to show off some of the types of agents that you’re building right now and
0:12:04 some of the like really simple workflows that you’ve built to, you know, just make your
0:12:05 life easier.
0:12:10 For sure. For sure. Yeah. So I’ve installed a Chrome extension. This thing is an alpha
0:12:13 now and soon we’ll be in the app store. It’s been submitted. And it’s here. I can drag
0:12:19 it around. It can be collapsed or it can be expanded. And these are things almost like
0:12:24 triggers for AI agents. So here’s a TLDR AI agent, here’s an agent called what is this?
0:12:29 This one’s called alternatives, right? And I can trigger them. Or if I want to have these
0:12:34 things collapsed, and I can click on this button, and it gives me the side drawer. And
0:12:39 in it, you get, you know, all the agents that are sort of available in our agents store,
0:12:43 you get all the agents that you’ve created yourself, you’ve got agents that you’ve started.
0:12:48 So these are like your favorites. And so you can very quickly, again, take an agent and
0:12:53 trigger it. And so like what could we do here? Well, one thing we could do here, for example,
0:12:57 is do like a TLDR. And so again, I can trigger it from here, or I could have triggered it
0:13:01 from the extension right here that I’ve got. And it instantly, you know, again, grab the
0:13:05 content on this page and gave me this quick summary. And this quick summary was just a
0:13:10 quick AI model looking at this thing. And it did this and extracted these entities. We
0:13:13 also have a thing called who’s involved, which sort of more sophisticated, you’ll go grab
0:13:19 the page, and then we’ll find like all the people and entities on that page, and ask
0:13:24 me like what people or entities I’d like to know more about, like this theoretical business.
0:13:29 I can click this, and this is going to kick off like a really sophisticated workflow that
0:13:34 could take in this case, I think takes about two minutes to run, which is again, generating
0:13:38 Google search queries over here. Again, you can sort of see it working. It’s executing
0:13:41 those Google search queries on Google news, Google images, blah, blah, and then it’s
0:13:45 going to create a report for us. And by the way, we don’t have to wait, we can say notify
0:13:49 me when finished. And this thing is going to send me an email with this like really
0:13:54 sophisticated profile executive summary, you know, of this person here that I’ve done.
0:13:57 And again, we don’t have to stay on this page, we can keep moving. So you know, here you
0:14:01 are, I’ve got a little thing over here where I’ve got a bunch of buttons, one of them is
0:14:06 YouTube TLDR, right, I click this thing, it’s going to grab the transcript, it’s going to
0:14:11 process it and it’s going to give me a TLDR. Now your video is only 33 minutes here. And
0:14:16 so I could have watched it and you know, two times speed, that’s nothing at all. But some
0:14:21 videos are, you know, three and a half hours, you know, all of it is also fine tunable by
0:14:28 me. And so if I want my TLDRs to be different, I simply go and I edit this AI agent that
0:14:33 I have, and then I can use it. Gotcha. So many of these things like when you were just
0:14:36 showing you had the little like sort of tray on the right side where you had a whole bunch
0:14:41 of little like apps that you built. So my understanding is most of those apps are like
0:14:47 pre built templates that are available in my studio that you can just add to your sidebar,
0:14:50 but then also you can create your own and you can also modify some of the ones that are
0:14:55 already pre built. Is that kind of the idea? Yeah, let’s talk about that. So again, if people
0:15:00 sign up and they don’t yet have anything started, they haven’t built anything, they will see
0:15:03 on this agents tab, they’ll see these agents. It’s almost like a little like app store for
0:15:08 agents. It’s like an app store for agents. Exactly. Sorry, these ones right here. And
0:15:12 there are more agents coming again. This is an alpha now. You should expect hundreds
0:15:17 beyond that thousands. And then on any one of these agents, you can open it up and it’s
0:15:21 got sort of a like a detailed page and it tells you how much it costs to run this on
0:15:26 average about six cents. In inference, this one takes about 45 seconds. But again, it
0:15:32 returns back like these really awesome summaries like reports in a sense that are publishable,
0:15:37 that are shareable. They’re like pieces of content in a sense with just clicking a button.
0:15:42 And so you can run it right from here or you can run it from the extension. Gotcha. Click
0:15:47 a button, YouTube TLDR and run it. Now, the other thing you can do with these things is
0:15:53 if you click here, you can say make a copy. And when you make a copy, you’ve just made
0:15:58 a copy of that agent and you have access to like all of the it’s not code. You’re like
0:16:03 forking it basically. You’re forking it. Exactly. So agents are made out of these things called
0:16:07 workflows and these workflows are basically these things where you can add these little
0:16:12 nodes to. And so it’s got a start block. And so let me make this a little bit smaller is
0:16:16 take a look at what this one is doing. It’s actually quite simple. It’s got a start block.
0:16:20 You can configure the start block to again, how does it run? These things can be run on
0:16:25 demand meaning you trigger them manually. They can be run on any kind of a schedule that you
0:16:30 specify. If you click email here, it gives you the custom email address. And whenever
0:16:36 you send an email to the email address, it takes that as input and triggers workflow
0:16:41 and passes it variables. Or you can trigger it as a browser extension, which is what this
0:16:46 was. And then you get access to these launch variables as we call them. So when this thing
0:16:52 gets called by the browser extension, the browser extension passes at the URL, the metadata
0:16:57 in code, the metadata, our name, description, image, etc. The full text, which is visible
0:17:01 to the users is stuff you can see on the page. If user is selecting anything highlighting
0:17:06 anything or right clicking anything, it gets passed in this user selection variable. And
0:17:10 then like all the raw HTML CSS JavaScript behind the scenes or graphs that and that’s
0:17:16 available to us. And we can reference that using variables. And here we’ve got different
0:17:20 blocks that you can add. And I’m going to go here, you can see there’s many different
0:17:26 types of blocks, like fetch YouTube captions. Like this is how you can grab the captions
0:17:31 from a YouTube channel, or, you know, fetch the channel and metadata about it, or fetch
0:17:34 all the comments. This is all if you want to customize it, right? Like I assume you’re
0:17:39 like a regular person would not have to do, you know, expand its capabilities. Exactly.
0:17:43 Yeah, there’s a whole bunch of pre built ones that you can just use by default. You can
0:17:47 go build one from completely from scratch if you want, or you can take one that already
0:17:50 exists and sort of fork it and iterate on it yourself.
0:17:56 Right. And so, okay, so this is like one, let’s take a look at another one here quickly.
0:18:01 Let’s take a look at this researcher that we have that people love. This one takes an
0:18:05 average of four, almost four and a half minutes to run again, you can run it from here. If
0:18:08 you run it from here, it says, what do you want to research? And I can say I want to
0:18:14 research Matt Wolf and go off and do the research there, or I can run it from, you
0:18:20 know, Chrome extension, right? And so I could be whatever on on CNN. And then I can click
0:18:24 this thing and I can go to researcher and run it from here. And then again, it’s going
0:18:28 to do with that, you know, people index or did to which is like, grab a bunch of things
0:18:32 off this page and say which of these do you want me to research and I’ll present you
0:18:37 like a menu, you know, emerging technologies and AI, investigating recent developments
0:18:41 in artificial intelligence, but I can click this and this thing is going to kick off this
0:18:46 like very sophisticated workflow. Let’s take a look at what that workflows is actually
0:18:51 doing. Again, I’m going to say make a copy, which means work this, you make a copy of
0:18:55 this. And so, as I said, these things are made up of these workflows. The main workflow
0:19:00 here right here is called main. And it’s made up of these automation. So what is this thing
0:19:05 doing? Well, when it gets triggered, it gets passed into it, a variable called request,
0:19:10 that’s the user’s request. And then the first thing it does is it calls a model in this
0:19:15 case, 3.5 sonnet. And again, we support over 50 AI models with new ones being added all
0:19:20 the time, all the major providers, including DeepSeek and Google, OpenAI, Meta, Google,
0:19:25 Mistral, whatever, they’re all multimodal. So you can call any model you want. And then
0:19:30 in this case, it says analyze this request and figure out a plan to use Google search
0:19:35 in order to do that, and then store that plan in a variable called plan. And then move on
0:19:40 to the next step. And in this step, it says, here’s the plan that was made here. Using
0:19:45 this plan, create a bunch of Google search queries to be able to research this plan.
0:19:50 And then it uses another type of block, which is a run workflow block that triggers another
0:19:56 workflow on this, this workflow called run query, and it passes into it an array of queries.
0:20:02 And then it runs this workflow here over and over again, and searches Google, and then comes
0:20:08 back with all of the data from that. And then it normalizes everything. And then it generates
0:20:13 the report. And then it runs another workflow to generate images. And then it outputs the
0:20:17 thoughts and report and emails it to you, and it’s available for you whenever you want.
0:20:21 And so again, you don’t need to know any of the stuff in order to use it, right? In order
0:20:25 to use it, you just push this button and say run, and it creates the stuff for you. But
0:20:26 you can edit it and do it.
0:20:29 Yeah, the matriarch camera, when you talked about the Google trends thing, that was like
0:20:33 honestly, the thing that I’ve found the most interesting out of everything you’ve said,
0:20:36 like just imagining like how that would work. And I can imagine so many different scenarios
0:20:37 where that’d be useful.
0:20:42 We’re monitoring your competitor’s websites and messaging. Again, the AI can just be
0:20:46 checking all your competitor websites. And as they change their messaging, which they
0:20:51 do, the AI can be sort of analyzing like, which way are they going? What might it be
0:20:52 and why?
0:20:53 Yeah, why are they doing that?
0:20:55 Well, the stuff that like you wouldn’t have time to do and do it.
0:20:58 It’s going to get kind of funny though, if every company is like following trends like
0:21:01 that, like, oh, based on some data, we changed our website copy. And then, oh, well, they’re
0:21:04 changing theirs. And then they’re all like changing it.
0:21:05 Right.
0:21:09 Well, it’s funny because it’s like whenever like open AI or Google or Microsoft or one
0:21:15 of these big companies like makes a change to their terms of service, it’s news, right?
0:21:18 People are like tweeting about it and things like that. And it’s like, you can actually
0:21:23 have one that watches for changes of terms of service on AI related companies to see
0:21:26 if now all of a sudden this company is going to spy on me, even though they told me they
0:21:31 weren’t originally, you know, right, we’re simple to do use case trivially easy to build.
0:21:36 You can build it in five minutes doing this. Okay, so I’m on your profile on Twitter X.
0:21:41 And so these things can be contextual. And so when I’m on Twitter, the agents that are
0:21:46 relevant to Twitter can show up right here. And so one way I could get to know Matt Wolf
0:21:50 is by going through and like spending a bunch of time like looking through your profile,
0:21:55 but ain’t nobody got time for that. And so I’ve got a thing called profile analyzer and
0:21:58 it just grabs that then the AI is going to analyze you. And I could build up a bunch
0:22:05 of things, you know, your primary focus, sort of content themes, content types, key behavioral
0:22:10 insights, you know, so again, like a bunch of stuff about you, I’m in control, I can
0:22:15 modify this thing to do something different because, you know, I want different types
0:22:20 of things, or there might be conversations that are happening on Twitter. And again,
0:22:25 I can try to like read through this and understand this thing. But that’s really hard to do.
0:22:30 But I can just click this little button, which is X conversation analyzer, and it can do
0:22:35 like all this work for me. It give me again, these TLDRs. And these are artifacts, like
0:22:40 I can share them, I can save them, I can send them, I can do whatever I want with them in
0:22:41 order to be able to do that.
0:22:45 Yeah, one cool thing too that I don’t think you mentioned, but I know I’ve seen it playing
0:22:50 around with Mind Studio is that the sort of visual workflow that you were showing, you
0:22:55 can actually use natural language, and tell it what workflow you want, you can and using
0:22:59 natural language, it will actually go and automate the building of those workflows as
0:23:00 well.
0:23:01 Oh, that’s awesome.
0:23:06 Well, again, we can make yours here by creating agent, there’s actually, I wouldn’t even use
0:23:10 this to do it, but like for somebody novice, they could do it. So again, we’ve got, you
0:23:14 can start from a blank project, or you can start from this thing called Mind Studio Architect,
0:23:18 where it says, what do you want me to build? And I’ll build it for you. By the way, this
0:23:25 Mind Studio Architect was built using Mind Studio. It’s just a Mind Studio agent whose
0:23:31 job it is to be the architect of Mind Studio, so it knows how Mind Studio works. It knows
0:23:35 all the blocks that Mind Studio has, it knows the kind of the design patterns that Mind Studio
0:23:37 prefers.
0:23:40 And then if you tell it what to do, it says, I know how to follow these instructions. And
0:23:49 so again, we could say every day, monitor my competitor’s website and check terms of
0:23:58 service when they change, notify me with a summary of change. I think it seems pretty
0:24:02 straightforward. I don’t need to, again, you can do that and say generate. And this thing
0:24:06 is going to go off and it’s going to create a plan of like how it’s going to build this
0:24:09 thing, because there’s a bunch of ways you could build it. You can say, I don’t like
0:24:12 this plan. Choose another plan. Or you can say, great, this plan looks good enough for
0:24:18 me. Push go, and then it’ll build all of the scaffolding for you. And many times, first
0:24:26 time it builds it, it runs well and you don’t need to fix anything. So that happens. Sometimes
0:24:30 it makes some weird error and you’ve got to trouble shoot it a little bit. And we’ve got
0:24:35 a little button where you can ask people for help and they can, there are more experience
0:24:36 than you in doing it.
0:24:39 So you can do that. But while it’s doing that, let’s open up another tab. And I just want
0:24:42 to start another one here quickly. I want to show like kind of the simplest one that
0:24:48 I think will help people grok this maybe better than like some of these ones that I’ve been
0:24:52 showing. So I’m just going to go to build. I’m going to say create new agent and my development.
0:24:56 I’m not going to use this. I’m just going to say open blank project. So the system prompt,
0:24:59 you can add these or not. I tend not to do them. But if you want to do them, this again
0:25:03 sort of tells the system like what you want it to be. You can write it yourself or again,
0:25:07 you can use another agent that we have here to generate a prompt.
0:25:10 I’m still thinking about the whole Google Trends thing. Sorry, my mind stuck on that.
0:25:15 I know, you know, another HubSpot podcast, My First Million, they’ve had so many episodes
0:25:18 talking about, Hey, if you want to start a new company, one of the main ways to do it
0:25:19 is to look at Google Trends.
0:25:25 You can just automate, you can be automating, I’m glad that that connected. Yeah.
0:25:30 Because that is a profound superpower. Yeah, it sounds like you can actually leverage
0:25:36 Google Trends, leverage that data at scale. It’s profound. It’s an insanely unfair competitive
0:25:38 advantage with anybody who can’t do that.
0:25:41 Yeah, sorry to take you off track. I just like that was still in my mind. I’m like,
0:25:42 I want to go try that.
0:25:46 Okay, so let’s build the Chrome extension-y thing. Okay, I’m going to go here, start.
0:25:51 I’m going to say, you know, browser extension, again, we now get these variables. I’m going
0:25:55 to say plus, and I’m just going to say generate text. This means call a model. In this case,
0:25:59 I’m just going to change a model just to change it. Why not? We’re going to use Gemini 1.5
0:26:06 Flash, and we’re going to tell this model, you know, create a TLDR of this. Okay. And
0:26:10 then in double curly braces, that’s as complex as it gets. We’re going to pass it this variable
0:26:16 here that we get again called full text. Full text means the full text of the page from which
0:26:19 we call this. There it is. It’s case sensitive. And here we can either display it to user
0:26:23 or save it to variables. I’m just going to leave it display to user because we’re done.
0:26:28 We just made this agent that does a TLDR. Again, if we want it to be fancier and we want it
0:26:32 specifically say how we want it to create the TLDR, what should it look for? What should
0:26:37 it buy us for? Are you just have this in plain English over here, right? And then, you know,
0:26:43 you can click here and you can give it a name. I’m just going to call this maths TLDR. And
0:26:47 then you can give it an icon if you want. And then you can just say publish. And so again,
0:26:52 we just built this thing. It exists. It’s got a URL. We can run it. We can run it manually
0:26:56 and just give it a URL, but that’s no fun. That’s not how we built it. We want to run
0:27:03 it from whatever, this one here. And so we can go here and we can open our extension
0:27:07 right here. And like a fast way to find it because I’ve got so many of them is to simply
0:27:11 do a search, maths TLDR. And then I can just run it from here. And again, we built this
0:27:15 thing in what less than a minute right here. And you know, this says Microsoft was made
0:27:19 a major breakthrough in this. Okay, so that’s not a very good TLDR. We could go back and
0:27:25 say no, no, like these are specific things that I want out of my TLDR and do I’m not
0:27:28 going to demonstrate that because anyone can do that. You can play around and then you
0:27:32 could save and you go back and you test it again, or you’ll go back here and test it
0:27:36 again. It’s that easy to sort of like iterate on these things to create custom tools for
0:27:37 yourself.
0:27:42 Yeah. And then you can just go and test different models to maybe Gemini 1.5 flash tends to
0:27:47 give you smaller responses where if you were to use, you know, GPT 40 or sonnet 3.5, it
0:27:49 will give you a much deeper response.
0:27:53 Absolutely. In fact, let’s do that. Let’s use sonnet here instead. We’re going to publish
0:27:57 that and we’re going to rerun the exact same thing. And again, like it, it’s super easy
0:28:01 to rerun these things. I should take the one we just made. So I don’t have to look for
0:28:09 it. Matt’s TLDR. I can click on this thing. And over here, I can say add to start. Okay.
0:28:13 So this one didn’t do much more on the TLDR, but you can change models. Again, if you give
0:28:17 it some more instructions, it doesn’t matter a much better job. Another way to get it to
0:28:21 do stuff like that, that’s even better. I can say generate prom. You know, we like to
0:28:24 refer to this thing that usually is assistant, but you can say anything you want, write it
0:28:34 in any form you want. Assistant generates TLDRs for human when provided with content.
0:28:38 And I could say from the web, or I can say content, whatever, and like to say generate.
0:28:42 And this thing will go off and it’ll like generate the prompt that will be much more
0:28:48 specific about what a TLDR should have. And that tool will now bias the TLDR that we get
0:28:54 on that. But the point is like all of this stuff is like really easy to do and can play
0:29:00 around with. Yeah, for sure. Now, what’s up with the one that you started using the automation?
0:29:03 You use the natural language to build that one. Let’s check on that one. Let’s check
0:29:08 that out. Okay. So here’s what it did. It built the terms of service change monitor.
0:29:12 And again, this is the plan automatically monitors competitors terms of service, page
0:29:17 for changes, analyzing differences, sending email notifications. It says it’s going to
0:29:23 take, you know, is input the URL in the notification email. And then here are the variables it’s
0:29:28 going to use. It doesn’t need any custom functions that it needs to write. This is what the automation
0:29:32 is going to look like. It’s going to scrape the terms of service. And going to clean and
0:29:37 normalize content, check if it’s the first run, you know, or if it’s got data about it
0:29:41 already, save the version, you know, and the first run analyze change. Again, we say great,
0:29:46 whatever this means, sounds good to me. And then you say go, and it’ll go off and it’ll
0:29:52 build that entire workflow, we’ll get that workflow again, it might work with just we
0:29:56 have seen to put in the URL, because we didn’t tell it what URL, we could have told it what
0:30:01 URL. Okay, so workflow generation is complete. I can, you know, watch a YouTube tutorial
0:30:08 on how to use it, or we can close and start editing it. Again, it built the system prompt.
0:30:13 It built the automations for us that we can then look at and we can edit. And so this
0:30:18 is using our scrape URL block, and it’s taking in here again, a variable called URL that
0:30:23 we need to define. So here, it says this thing is scheduled, and it gets past these like
0:30:28 launch variables. Or if it’s the same URL, we can just put it in here explicitly. So
0:30:34 we can say, you know, CNN.com slash terms of service, you get the point of that.
0:30:35 Right, right.
0:30:38 Yeah. Dimitri, I’m kind of wondering, like the more you talk about this, like, long term,
0:30:42 I’m curious, like, how does this, you know, obviously for open AI agents, it’s going to
0:30:47 be very, very important. Like, you know, and I’m kind of curious, is your thesis that
0:30:52 even if they really, really focus on agents, and of course, they get amazing at it, that
0:30:54 there’s still going to be a benefit of having like a community of people sharing their different
0:30:59 use cases and different templates, is that kind of the idea behind Mind Studio?
0:31:03 Yeah, look, we believe that models will continue to get better and better. And certainly model
0:31:10 providers like open AI will have sort of consumer interfaces along with API interfaces and those
0:31:16 consumer interfaces, you know, today are primarily conversational, so chat GPT, and it’s got
0:31:21 access to, you know, tools, function calling, etc. And like, in your chat, you can sort
0:31:25 of go back and forth with it and like does stuff for you. And there’s a lot of use cases
0:31:30 for that where like ad hoc, you want to do something, and you do that. But for, you know,
0:31:34 many other things, and I propose most other things, you will say, well, I wish like something
0:31:39 just like I could build it and it just ran and it did these things like monitor Google
0:31:44 trends. I don’t want to constantly ask chat GPT to do it for me, or talk to it. Right.
0:31:48 I want it to only notify me, by the way, does it in the background, it should only nudge
0:31:52 me if it found something interesting. Most of the time it won’t. And so just to be there
0:31:56 in the background, I’ve got a dashboard, I can see these things running, I can see what’s
0:32:01 scheduled, etc. As for like the capabilities of the models themselves to sort of be at
0:32:06 a place where, again, you kind of don’t need anything like this, and you just tell them
0:32:11 I want you to be this thing and it sort of does it, I struggle from a product side to
0:32:18 see how you would sort of have the human feel enough in control without sort of creating
0:32:22 something that they could understand. Again, you might generate the workflow for them just
0:32:28 like our architect did, but I think humans still being able to say, Oh, I want to put
0:32:33 like a checkpoint in here. Yeah, modify how it works. And for this, or I want to be able
0:32:38 to change this in some way. You know, I didn’t show you guys our debugger, which is awesome,
0:32:43 like extraordinarily powerful. So like our instrumentation for like building and running
0:32:50 these things is dramatically deeper than any of the model providers themselves offer. And
0:32:53 so if you’re going to build anything with models, we think this is the no brainer way
0:32:58 to be it’s it’s the most instrument that it’s the fastest it’s the easiest gives you access
0:33:02 to all those models, models go up and down. Yeah, I was going to say it feels like that’s
0:33:05 a big advantage to you right is like being agnostic because I mean, yeah, in the last
0:33:09 month we had deep seat come out now grok’s amazing. There could be a lot of options
0:33:13 out there. Well, as you’re building these workflows, again, you want to with for each
0:33:18 step where you want to use intelligence, you might want to call a different model, right?
0:33:23 There you know, Gemini 1.5 flash is super fast at being able to do a bunch of stuff,
0:33:28 but it’s not a very good writer. Like Claude is a much better writer. So you might get
0:33:32 the Gemini to do a bunch of analysis data collection, all of that and then sort of hand
0:33:38 that off to Claude to put together and do or like generating images or like whatever else
0:33:43 is doing here. So like this ability to analyze different models for each part of the step,
0:33:49 use the right model based upon quality, latency, so low latency and cost. Yeah, and you probably
0:33:52 could suggest that eventually too, right? Like this model is better for writing here.
0:33:57 I want to use this one. Yeah, it’s like suggested in those like detail pages that each one of
0:34:02 them has you can read them. But sort of what I was going is like, these models aren’t reliable.
0:34:06 These model providers are still building their infrastructure. They’ve got a bunch of usage.
0:34:10 So they go down periodically, in fact quite frequently. And if you’re just like standardized
0:34:15 on only open AI and all you’ve got is there, you’re kind of stuck your AI and doing it.
0:34:18 Here you can just fail over to any of these other models that could also do the job. They
0:34:22 might not be preferred. They might be a little bit more expensive. They might be a little
0:34:25 bit slower, but at least you’re not down in your workings like all of those kinds of things
0:34:30 become really important when you get serious about like building these things. So are those
0:34:35 like redundancy mechanisms or those built in already like try Claude 3.5 if that doesn’t
0:34:41 work, then go do GPT 4.0 or whatever it doesn’t automatically do it. So like can notify you
0:34:46 that models are down, which we do. And then you can manually go do that. But coming soon,
0:34:49 it will do it intelligently and just like do that for you. That’s cool. Like here’s
0:34:54 my tier list, you know, for writing, for coding, for whatever you sit in this order, the infrastructure
0:34:58 is there. You can now just manually choose a different one takes 10 seconds. But soon
0:35:02 it will be autonomous for sure. Yeah, I did have a question. This was something you showed
0:35:06 off on the screen earlier in the conversation, where you showed, all right, it’s going to
0:35:10 cost like six cents to run this, right? How does that work? Is it like a bring your own
0:35:16 API key? Is it like you guys bill based on the usage? Where does that six cents come
0:35:20 from? Yeah, so you don’t need to bring your own API key. Although if you want to call
0:35:25 your own models, we allow again function calling an API call. You can call any model including
0:35:30 on premise, private cloud, etc. The vast majority of people that use mine studio use
0:35:35 our API keys that are built in so you don’t need any API keys. So you instantly have access
0:35:41 to all of the models from all of their providers on our billing. And we simply pass the costs
0:35:48 on to you based on usage. Okay, we take a tiny markup 2.9% and then it reduces with scale
0:35:55 and goes down to like 0.15% of overage over what the model providers charge us. It’s kind
0:35:59 of like Stripe. We’re going to see ourselves very much in that light. We’re sort of like
0:36:05 what Stripe is to payments or Twilio is to telephony or moxes to video. Mine studio is
0:36:10 to intelligence. Make sense. Make sense. Have you ever considered doing it more like cursor
0:36:14 where like you have like a monthly and that’s basically baked in like you’re kind of assuming
0:36:18 they’re going to use the API to my mouth with some limit. And then you go over that it’s
0:36:21 like, okay, you got to either pay us more or use your own API key. Yeah, I would like
0:36:24 that with cursor. Then I don’t have to worry about doing something and like, oh, right now
0:36:27 I’m paying something. I’m like, oh, I’m not paying anything. I’m just using it. You know,
0:36:30 and so that’s kind of feels nice to me. I don’t know. Great. Well, I’ll break the news
0:36:40 on your show here. We’re about to add that here. I can predict the future when this publishes.
0:36:44 It might already be there. Yes, exactly. That makes sense. But so I wanted to mention sort
0:36:50 of another thing. So we’ve got this map as you noticed that this is like an app store.
0:36:55 And these agents are put in there today by us. We created all of the agents that are
0:37:00 available for you to use. You just pay metered usage form. They’re free to use, but other
0:37:06 than metered usage, you can make copies of them and modify them and launch new ones,
0:37:11 etc. And we will adding more to it. The other thing that we’re doing is we’re opening this
0:37:16 up to the world just like the app stores opened up to the world that people can show up and
0:37:20 say, I’ve got some use cases that I think are awesome, and that lots of people are going
0:37:26 to want. I’m going to build them in my studio. And again, as you can see, it’s easy to build.
0:37:30 And then I’m going to submit them to the store. And I want to make them available to other
0:37:35 people to just use by venturing also like charge form, obviously, we’ll implement that
0:37:40 as well. And so we’re actually looking for people that want to show up and start building
0:37:45 these things. And we’re venture backed. And so we’re willing to pay like bounty. So there’s
0:37:51 an opportunity for enterprising folks to get in early and fill up this app store with all
0:37:57 kinds of amazing utilities that now can be triggered from a browser extension. If you’re
0:38:04 building good ones, we’ve got, you know, praise and money to make your life better. And we’re
0:38:08 also looking to hire people full time to do this. There’s like great opportunities for
0:38:11 people to start building these things and take an advantage.
0:38:15 Very cool. When you say app stores, they’re going to be like a revenue share or?
0:38:19 Absolutely. Yeah. You’ll be able to charge for these things and you’ll be able to earn
0:38:21 revenue from the work that you’ve done. Yeah.
0:38:25 Yeah. Very cool. By the way, people are already building these things and putting them behind
0:38:29 paywalls, which you can do. And so their entire businesses that have been started that simply
0:38:34 build a bunch of Mind Studio AI agents, put them behind the paywall and then charge monthly
0:38:37 access to all of those agents.
0:38:41 Yeah. I remember early on, I think when we first talked, you know, 18, 19 months ago or
0:38:45 something, that was kind of like the original sort of selling point of Mind Studio is go
0:38:47 build agents and then sell access to them.
0:38:51 Yeah. So people took us up on that and are doing it. And so, yeah, you can make money
0:38:57 with them already. By the way, we also now have over 150 integration partners that make
0:39:04 money by again, they’re experts in Mind Studio and they are building AI agents for organizations.
0:39:08 So we’ve got over 150,000 of these things that have been built and deployed. Again, some
0:39:13 are just like experiments, they’re toys, but many are like mission critical, really important
0:39:18 AI agents that sit at, you know, government agencies, the His Majesty’s Revenue and Customs,
0:39:26 the British IRS, uses Mind Studio and recruiting, you know, Service Now uses it in sales, demo
0:39:30 automation, a company called Advanced Local, which is a giant publishing company, has got
0:39:35 over 80 of these running that are automating newsrooms and newspaper publishing. And so
0:39:41 some of those are built by the people inside these companies, others, these companies hire
0:39:46 our partners to do professional services work and build them. So again, there have been
0:39:53 entire actually agencies that have like started off simply to do Mind Studio implementation
0:39:58 or organizations that want the skill. That’s where all the budgets are flowing, you know,
0:40:02 in enterprises, they’re all flowing towards this like AI digital transformation.
0:40:08 Right. And we make it really easy for them to get this giant ROI out of very little work,
0:40:09 you know.
0:40:10 Very cool.
0:40:11 Very cool.
0:40:14 Absolutely. Is there anything else about Mind Studio that we should touch on that we haven’t
0:40:18 touched on yet, or even something outside of Mind Studio that’s really, you know, got
0:40:23 your attention right now that you’re really excited about just in the AI or tech world?
0:40:26 That’s a deep question. It was like another podcast, right? That’s another hour.
0:40:32 Yeah, no. I mean, I’m assuming your head’s pretty much always in Mind Studio playing with
0:40:36 it, making it better. So, you know, you may not be going and playing with all the other
0:40:39 tools and, you know, stuff that’s out there.
0:40:43 Well, I get to watch your videos and get you doing it. I don’t want to make it political
0:40:47 because there’s a bunch of interesting stuff there. Like what’s a fact checker? Well, fact
0:40:53 checker is a really important tool that, you know, citizens of democratic countries need
0:40:58 to be able to understand what’s reality. And now that tool is a click of a button. It has
0:41:00 broad implications of what that means.
0:41:05 Yeah, yeah. Well, there’s a site called Ground News. I don’t know if you’ve heard of it or
0:41:09 not, but it’s like a news website that curates all of the latest news from like all of the
0:41:15 various news sources. And then when you’re actually looking at the site, it’ll tell you,
0:41:19 does this article and this news source tend to lean left? Does it tend to lean right?
0:41:24 Does it tend to be a little more center? And it sort of like analyzes every piece of news
0:41:29 and tell you like where that news falls on the political spectrum. And I feel like with
0:41:34 Mind Studio, you can build that, right? Go to any news website, click a button. It’ll
0:41:38 tell you whether this site tends to lean left, tends to lean right and actually look for
0:41:42 bias in the news article for you, right? Like you don’t need to get political. It doesn’t
0:41:47 matter what sort of political leanings you have. That just seems helpful to anybody.
0:41:49 Pick your bias.
0:41:53 We have AI agents that have already been built that do specifically that, that are sentiment
0:41:58 analyzers and bias analyzers. You push a button and it looks at the article and says, Hey,
0:42:01 here are things that might be inaccurate. Here are things that are misleading. Here
0:42:05 are things that are clearly biased where they’re showing only one perspective. There’s others
0:42:08 that have a counterpoint and so it’ll be like, great, what’s the opposite of that? Like what
0:42:13 would be the counterarguments? And so again, these superpowers that you now get with click
0:42:19 of a button that AIs can do really easily. But once they’re like made so easy to take
0:42:23 along with you, that’s kind of the big unlock. Yeah. Yeah. Wherever you are, you now can
0:42:29 like instantly trigger them and get that instead of like going to chat GPT and asking it to
0:42:30 do something.
0:42:31 Right.
0:42:35 Super cool. Now with the little like Chrome extension to any plans to make that available
0:42:39 on mobile as well, like will I be able to be reading a news article on my phone and do
0:42:40 the same kind of things?
0:42:45 Yeah. We’re going to have a native app. Probably I’m guessing in a quarter, which will allow
0:42:50 you to basically make it a shared target. So you can like intent out from any application
0:42:54 to any of these agents and get callbacks on that. So like it instantly sort of enables
0:42:58 anything that can do share intents to now have
0:42:59 Gotcha.
0:43:03 Available for it. There are some, I can’t sort of announce them yet, but there are medium
0:43:10 sized social networks that are starting to integrate mind studio agents into their interfaces.
0:43:15 So they will be within their native mobile apps. There will be buttons on their posts
0:43:20 that allow you to fact check things or like do other things. And those are simply mind
0:43:24 studio agents that are doing that work. And so you’ll see, you know, certainly a lot
0:43:26 more of that kind of stuff that you can do.
0:43:30 Well, cool. This has been a really, really interesting conversation. I mean, I love,
0:43:33 I love actually getting to see all this stuff in person. I’ve been playing around with Mind
0:43:38 Studio for a little over a year now and have messed with it quite a bit, but it’s always
0:43:42 really cool to see how like the person who’s actually building the tools would actually
0:43:48 use it as well. So I really appreciate all the insights and you showing off how to use
0:43:49 it.
0:43:54 The website is mindstudio.ai. Is there anywhere else people can follow you? You on X, you
0:43:58 on LinkedIn, where’s the best place to sort of get to know you?
0:44:03 I’m on all of them. I prefer LinkedIn and please feel free and add me on LinkedIn. I’m easy
0:44:08 to find Dimitri Shapiro. And yeah, that’s probably the best place to follow me, but
0:44:14 I’m also on X and Facebook and Reddit and everywhere else. You gotta be everywhere.
0:44:19 Very cool. Well, thank you once again for joining us today. This has been a really cool conversation.
0:44:23 Really, that’s listening. If you haven’t already, make sure you subscribe on YouTube
0:44:28 for more interesting conversations like this, especially if you want to get the visual element.
0:44:32 And if you prefer audio, you’d like listening in your car or wall out walking your dog.
0:44:36 We’re available wherever you listen to podcasts. So subscribe to us there. And thanks again,
0:44:38 Dimitri. It’s been an amazing episode.
0:44:39 Great to chat.
0:44:57 [Music]
0:44:59 [Music]
0:45:09 [BLANK_AUDIO]

0:00:04 [MUSIC]
0:00:06 Hey, welcome back to the Next Wave Podcast.
0:00:06 I’m Matt Wolf.
0:00:08 I’m here with Nathan Lanz.
0:00:13 Today, we’re going to talk about the crossover of software and AI.
0:00:15 We’re bringing on a guest expert,
0:00:17 Dan Shipper to talk all about it.
0:00:19 Dan is the CEO of every.
0:00:22 He has all sorts of companies underneath him that he’s incubated.
0:00:25 He’s an investor. He works with Sequoia.
0:00:28 He has all sorts of knowledge in the software space,
0:00:31 and he’s going to break down his exact process
0:00:33 for deciding what products to build,
0:00:35 and what tools he used to build them,
0:00:37 and exactly how he brings them to market.
0:00:38 It’s a great discussion,
0:00:41 so let’s just go ahead and dive right in with Dan Shipper.
0:00:44 Thanks so much for joining us today, Dan. How are you doing?
0:00:46 I’m doing great. Thanks so much for having me.
0:00:49 Yeah. I’m excited to really dig in with you and what you’re doing over at
0:00:52 your various companies that you’re working on.
0:00:54 Let’s go ahead and lay the groundwork.
0:00:57 Tell us a little bit about your company and what you’re building,
0:00:59 and what is every?
0:01:00 Yeah. That’s a good question.
0:01:01 We’ve been around for five years,
0:01:02 so it’s evolved significantly over time.
0:01:05 The way that I talk about what we do right now is,
0:01:07 I call us a multimodal media company.
0:01:10 We publish writing, we publish podcasts,
0:01:12 we publish a video, and we publish software.
0:01:13 We do all that using AI,
0:01:17 and we’ve gone in that direction because I think that there’s
0:01:20 this really interesting thing happening where the line between
0:01:24 being a writer and being a builder is starting to blur,
0:01:27 and they’re moving into this overlapping Venn diagram,
0:01:30 because now that you can write in English and turn it into code,
0:01:32 writing is building.
0:01:35 Also, because software is so cheap to make,
0:01:37 I think that software is becoming content.
0:01:41 As a media company, what we focus on at every is,
0:01:42 we have a daily user,
0:01:45 every day we write a long-form essay about what comes next in tech.
0:01:49 We’ve got about 10 regular writers who do that.
0:01:51 I write every Friday a column called
0:01:53 Chain of Thought on the intersection between AI,
0:01:56 creativity, psychology, philosophy, all that stuff.
0:01:58 Our job is to explore the future,
0:02:02 and the best way to do that in my opinion is always to be using
0:02:05 new tools and experimenting with things and building things and all that stuff.
0:02:08 What we found is that when we experiment with something,
0:02:10 if we release it to the audience,
0:02:13 people love it, it makes the writing better.
0:02:18 Sometimes those experiments become real products and should be real products.
0:02:21 What we’ve done is then made them into real,
0:02:24 actual products that we often start by using them internally,
0:02:25 and then we’ll post them to the audience,
0:02:28 and then we bundle them into the subscription so you pay one price,
0:02:31 and you get access to all of our content and all of the software that we make.
0:02:33 We’ve done four incubations so far.
0:02:37 The most recent one, which we just launched about three or four weeks ago,
0:02:39 right before Christmas is called Cora,
0:02:41 and it manages your inbox with AI.
0:02:43 Basically, what happens is as emails come in,
0:02:45 it decides if you need to respond to them.
0:02:47 If you do need to respond to them,
0:02:48 if they’re from a human,
0:02:50 it makes it to your inbox and you see it immediately.
0:02:53 If not, it automatically archives it and then twice a day,
0:02:56 you get a beautiful brief that
0:02:59 summarizes all the things you need to know about what’s going on in your inbox,
0:03:00 but you don’t need to actually do anything with,
0:03:04 and it just saves a ton of time because 80-90 percent of your emails are like that.
0:03:09 We have a bunch of other products that are in that vein for different parts of your work life.
0:03:11 That’s the basics of every.
0:03:13 >> That was Cora, like C-O-R-A?
0:03:15 >> C-O-R-A, yeah.
0:03:16 >> Okay, cool.
0:03:17 >> Yeah. It’s been going really well.
0:03:18 We’ve got like 10,000 people on the wait list,
0:03:21 we’re taking off like 20 people a day and just starting to ramp it up.
0:03:24 >> I’m on the wait list, so I can’t really talk about it, I haven’t tried it.
0:03:27 >> I’ll get you off the wait list, you should have told me, I know a guy.
0:03:31 >> Cool. That’s fascinating thing about writing is becoming software,
0:03:31 is that what you said?
0:03:33 >> Writing is building and software is content.
0:03:35 >> Yeah, okay. I’ve been noticing that too,
0:03:36 because it’s part of this doing this podcast,
0:03:38 I’ve been trying to get more hands-on with AI,
0:03:39 like I actually started building some stuff.
0:03:45 Recently, I started using all the different AI tools to develop a game prototype just for fun.
0:03:46 Maybe it becomes a real thing, I don’t know,
0:03:50 but it is fascinating that I am now talking with AI,
0:03:53 and kind of writing my game design document with AI,
0:03:56 using my voice, using something like whisper flow,
0:03:58 chatting with it, having it update the document,
0:04:00 and then I feed that document back into
0:04:01 01 Pro when I’m coding on it,
0:04:05 and it’s just amazing that that conversation I’m having with AI,
0:04:08 then forms the actual code and software that is created.
0:04:09 That’s never existed before.
0:04:10 >> Isn’t that so cool and like,
0:04:13 so you can code while you’re walking around and just like,
0:04:17 you know, recording to your voice and Kiran who is our GM of Cora and he built Cora,
0:04:20 like that’s how he started the first Cora MVP.
0:04:21 He built that in a single night,
0:04:24 and he did it by going and take a walk,
0:04:26 and he just like recorded himself being like,
0:04:27 this is how I want the app to work,
0:04:30 and he threw it into 01 and just like, it’s awesome.
0:04:31 I think like- >> It’s so cool.
0:04:35 >> There is a big difference between you can build a prototype,
0:04:37 and then actual like production-grade software.
0:04:38 >> Yeah.
0:04:39 >> But even the production-grade software,
0:04:42 it’s just changed what you can do and how fast you can do it,
0:04:43 and it’s really fun.
0:04:44 >> Yeah.
0:04:45 >> I really like what you said there.
0:04:47 So writing is building, that makes sense to me, right?
0:04:50 Like everything is kind of becoming prompting to get what you want, right?
0:04:53 If you want to write software, you prompt it,
0:04:54 the software can come into existence.
0:04:56 I mean, even now, like we’re talking about,
0:04:59 you can even speak it into existence and then it actually becomes software.
0:05:02 Can you speak more to the software is content?
0:05:03 What do you mean by that?
0:05:07 >> Yeah. So software is much cheaper to make now,
0:05:10 and one of the things we found is that if we release
0:05:12 a little experiment as a piece of software,
0:05:16 like all of the experiments we run are much better at
0:05:21 attracting audience and emails than any given essay or YouTube video or whatever,
0:05:25 because it’s this totally new format of thing, right?
0:05:29 So in the same way that on Instagram,
0:05:30 you can take pictures and TikTok,
0:05:33 you can take videos on YouTube, you can take videos,
0:05:35 and those are different from like Hollywood films,
0:05:36 but they’re still videos.
0:05:39 I think that we’ve been living in the world of like software is
0:05:42 like a Hollywood movie like you need a lot of money and time to make it.
0:05:44 Now, with all these tools,
0:05:49 it’s expanding beyond people who are professional developers and anyone can make a little prototype,
0:05:53 and that switch is like anyone can write a blog post,
0:05:55 anyone can do a tweet, it’s going to be the same thing.
0:05:57 There aren’t really platforms yet for this,
0:06:00 but I think there’s going to be like a YouTube of software development,
0:06:02 maybe like Replet is that to some degree,
0:06:04 and it’s not for professional coders,
0:06:06 it’s for people who just grew up prompting,
0:06:08 and that’s happening right now.
0:06:10 That definitely makes a lot of sense.
0:06:14 I also feel like this is something we’ve actually talked about on the show in the past too,
0:06:18 is like I almost feel like software could be the next SEO,
0:06:24 because I feel like just writing blog posts and writing content is so easy now,
0:06:27 like anybody can do it because of tools like Claude and chat,
0:06:29 GPT and Gemini and everything that’s available out there.
0:06:34 I think the value of written content is starting to go down,
0:06:35 and when it comes to SEO,
0:06:38 one of the goals of SEO is to bring somebody to your site
0:06:40 and keep them on your site as long as possible.
0:06:44 It almost feels to me like really good software that people find value in,
0:06:48 will be the new thing that keeps them around longer on your platform
0:06:53 and help your site rank better and help you actually grow your business a little more.
0:06:57 I’m curious if you have the same thoughts or if you would disagree with that.
0:07:00 That’s really interesting for one,
0:07:04 like I love words and I think the written word is always going to be super, super valuable.
0:07:06 I do think there’s something to what you’re saying,
0:07:08 which is that to some degree,
0:07:10 a lot of the content that gets made right now,
0:07:15 like I just saw this thing about HubSpot’s traffic and HubSpot’s traffic is just going down,
0:07:19 and so a lot of the SEO type content that’s made right now,
0:07:22 is probably better served by a chat GPT.
0:07:25 It’s like, okay, how do I ask for a raise?
0:07:30 There’s zillions of those SEO articles and that’s much better from chat GPT,
0:07:32 especially the more generic stuff.
0:07:36 I still think that for every, for example,
0:07:39 the audience that we have wants to hear from me,
0:07:42 and wants to hear from other writers that they know,
0:07:47 and are not looking for a generic fact or a generic process,
0:07:49 and that’s a different thing.
0:07:50 That’s a definitely smaller audience,
0:07:54 but it’s still incredibly important and valuable.
0:07:56 I think on the tools front,
0:08:00 you may actually see some of the same commoditization because you don’t need to go use
0:08:04 this ass tool when artifacts just spins up a little UI for you,
0:08:06 and you can just do whatever.
0:08:10 That may also happen with software to some degree,
0:08:12 and then I think you also have to wonder,
0:08:14 “Okay, if software tools are the new SEO,
0:08:18 what is the thing that is pushing people to those tools?
0:08:20 Is it Google still?
0:08:21 Is it chat GPT?
0:08:25 If it’s chat GPT, how and why is chat GPT letting those things surface and rank?”
0:08:33 Right. But I do think that providing people with rich multi-modal experiences,
0:08:34 that’s not just text, it’s not just video,
0:08:35 it’s maybe it’s tools or whatever,
0:08:39 that is I think going to be a really legitimate way to build an audience and build a following.
0:08:42 Yeah. I totally agree too that when it comes to the world of content,
0:08:46 I think as AI becomes more and more prolific,
0:08:50 and everybody’s using AI to try to gain the system through SEO and things like that,
0:08:53 I think it becomes more and more important to have your own voice,
0:08:56 and to actually be a voice that people can trust,
0:08:58 and actually put your face out there.
0:09:02 There’s a lot of stuff you’re seeing on YouTube now of like faceless video channels,
0:09:05 and people are claiming it’s all the rage right now to make faceless video channels.
0:09:07 But in my mind, if that’s the direction everybody’s going,
0:09:12 the people that actually put their face on YouTube are the ones that are going to rise and stand out.
0:09:15 The sacrifices we make for fame.
0:09:18 You’re in San Francisco, right Dan?
0:09:21 No, I’m in New York. I go there often, but I live in New York.
0:09:22 I lived in San Francisco for about 10 years.
0:09:24 I’m out in Japan now. That’s one thing.
0:09:27 But nice being here is, you know, even if the podcast blows up in the future,
0:09:31 I don’t have to worry about fame. Like, nobody will notice who I am.
0:09:34 That’s why I wear this like media mogul hat that HubSpot gave me.
0:09:36 I think it’s hilarious because obviously I’m not a media mogul,
0:09:38 but it’s kind of funny where you’re in Japan.
0:09:41 You know, I don’t know if you know this is a HubSpot-owned podcast.
0:09:43 I’m not sure if you’re aware of that. You were talking about HubSpot, this traffic.
0:09:46 I am aware because the emails are from HubSpot.
0:09:50 Oh, shoot. Yeah, yeah, I did not put that together.
0:09:53 Sorry, I didn’t. I mean, that is one reason to do this podcast, right?
0:09:55 Now, this is even I think kind of like what’s part of our page too is like, you know,
0:09:59 having faces is going to be very important in the future versus just generic content on,
0:10:02 you know, text content that anyone can throw out and AI is going to get better and better at that.
0:10:04 So SEO is going to be harder in the future.
0:10:08 But I do wonder why would chat to BT send you to other tools in the future?
0:10:11 If it gets so easy that they could just like create it for you on the fly?
0:10:14 With like what we’re seeing right now, like I’ve been using O1 Pro a lot and like,
0:10:16 yeah, seeing how good it is at software.
0:10:20 I can’t imagine in another two or three years that any idea you have
0:10:26 AI is now just like, cool, here it is. Here it is right now in the window and just use it.
0:10:29 Totally. It’s interesting because when it comes to the whole like HubSpot conversation,
0:10:32 I do think, you know, maybe HubSpot’s traffic is going down.
0:10:36 But, you know, podcasts like this and what they’re doing in like other media spaces,
0:10:39 I think is like, they’re sort of answer to that, right?
0:10:42 Like you’ve got podcasts like the next wave and my first million and marketing,
0:10:46 it’s the grain and the hustle and they’re starting to do all their newsletters now
0:10:48 with the hustle daily and, and all that kind of stuff.
0:10:51 So it’s like maybe less people are going to the HubSpot website,
0:10:54 but HubSpot’s becoming like this larger media brand.
0:10:54 Totally.
0:10:57 So I don’t think there’s anything wrong with saying that, you know,
0:11:01 HubSpot’s traffic is dropping because they obviously have like, you know,
0:11:03 backup plans that they’re building as well.
0:11:05 But just wanted to throw that out there.
0:11:09 I feel like I just like went to someone’s dinner party and like spit in soup or something like that.
0:11:11 It’s okay.
0:11:14 I knew in one part of my brain that this is a HubSpot podcast.
0:11:19 And then, but anyway, um, yeah, I’m not trying to imply that HubSpot is going to die or anything
0:11:20 like that.
0:11:23 It’s just the whole SEO game is like changing rapidly.
0:11:23 Right.
0:11:29 And yeah, it seems like shows like this are a really good bulwark against those kinds of changes,
0:11:31 which have been happening for a long time.
0:11:32 It’s not just AI.
0:11:37 And I think there is that interesting dynamic that, you know, you can see open an AI trying
0:11:42 to react to or trying to head off, which is like, you know, they do deals with publishers
0:11:45 and they cite their sources and all that kind of stuff.
0:11:49 And I do think that they would like to keep a vibrant ecosystem and app partners and all
0:11:50 that kind of stuff.
0:11:57 And I also think you can see with Google’s example that like, over time, the pressure
0:12:03 is to keep people on the platform longer and add more ads or like do the like instant answers
0:12:04 or whatever.
0:12:10 And I think AI broad use case general purpose AI consumer apps are going to have that same
0:12:10 pressure.
0:12:15 Maybe it’s not now, but in like 20 years, like when Sam Altman is like, you know, on
0:12:20 his island somewhere and like there’s another CEO, that’s what they’re going to have to do.
0:12:24 I mean, maybe we’ll just be living in a post capitalist utopia because we’ll have a GI
0:12:24 or whatever.
0:12:28 But like assuming that that’s not the case, that is a thing and that is a pressure.
0:12:30 And I think that’s a really interesting thing.
0:12:34 But also like all of these new tools open up new opportunities.
0:12:40 So like it seems likely to me that maybe it spins up a whole UI for you for like a specific
0:12:45 thing, but like the kind of UI that it makes and the way that it does the UI, like we’ve
0:12:48 been talking about is a content format is a perspective.
0:12:53 And it would be surprising to me if they didn’t allow brands or businesses or whatever to
0:12:58 like make available those tools or those powers within their app in some way, sort of like
0:12:59 an app store type thing.
0:13:00 And they’ve done that like repeatedly.
0:13:03 They’ve tried to do app store type things for a long time.
0:13:07 It hasn’t quite worked yet, but it’ll probably work eventually.
0:13:11 So it’s not like the death knell of like all independent tools or media or whatever.
0:13:15 It’s just like it’s changing and there will be new opportunities in the old ones.
0:13:17 You have to like move on to the new ones.
0:13:17 Yeah, totally.
0:13:22 Hey, we’ll be right back to the show.
0:13:24 But first I want to have another podcast I know you’re going to love.
0:13:29 It’s called Entrepreneurs on Fire and it’s hosted by John Lee Dumas available now on the HubSpot
0:13:30 podcast network.
0:13:35 Entrepreneurs on Fire stokes inspiration and shares strategies to fire up your entrepreneurial
0:13:38 journey and create the life you’ve always dreamed of.
0:13:41 The show is jam packed with unlimited energy, value and consistency.
0:13:46 And really, you know, if you like fast paced and packed with value stories and you love
0:13:48 entrepreneurship, this is the show for you.
0:13:54 And recently they had a great episode about how women are taking over remote sales with
0:13:54 Brooke Triplett.
0:13:56 It was a fantastic episode.
0:13:57 I learned a ton.
0:13:59 I highly suggest you check out the show.
0:14:02 So listen to Entrepreneurs on Fire wherever you get your podcasts.
0:14:10 Would you be open to like sharing a little bit of like your workflow for actually building
0:14:11 new software?
0:14:13 Like how do you validate an idea?
0:14:16 And then what sort of tools do you use to actually build the idea?
0:14:19 And, you know, can you walk us through some of that?
0:14:20 Totally.
0:14:27 I think the really cool thing about the current AI landscape is it has like completely reset
0:14:30 the table for like what tools are possible.
0:14:35 And so that means that there’s a lot, a lot, a lot of low hanging fruit because we’re all
0:14:37 just like figuring out our workflows, right?
0:14:38 It’s all new.
0:14:43 And that wasn’t the case two years ago because it was like the apex of the B2B sas boom and
0:14:45 like every low hanging fruit had been picked, right?
0:14:45 Yeah.
0:14:47 There’s a clear way you do things.
0:14:49 It was all like it’s all been figured out.
0:14:49 Yeah.
0:14:50 Now it’s totally different.
0:14:55 So what I do and like sort of the philosophy is first I use these tools every day.
0:15:00 And by these tools, I mean like the more general purpose tools like Claude and ChatGbt and Cursor
0:15:02 and Windsurf and like all that kind of stuff.
0:15:08 And what ends up happening is you end up finding workflows that start to work pretty well for you.
0:15:12 But once you find a workflow and a ChatGbt type tool,
0:15:15 it’s kind of a pain to just do it over and over and over again.
0:15:16 It’s not built for that.
0:15:18 It’s built for this like general purpose thing.
0:15:19 Right.
0:15:21 And that is always an opportunity to me.
0:15:25 So like a really good example of that is we have this app called Spiral.
0:15:28 And Spiral helps you automate a lot of repetitive creative work.
0:15:33 You know, for me, it’s like doing my podcast descriptions or my podcast tweets or my YouTube
0:15:36 titles or like that kind of stuff where it’s like I have a format.
0:15:38 I know basically what it should be.
0:15:40 And it’s like really important that I get it right.
0:15:42 But it’s also like fairly repetitive.
0:15:47 And what I realized at some point, I guess in like May that Claude was like really good at that.
0:15:51 Like if I gave it a few shot prompt with some examples and I was like, okay,
0:15:53 I want you to take a podcast transcript and turn it into a tweet.
0:15:56 And like here’s examples of me doing that in the past.
0:15:57 Here’s a new transcript.
0:15:58 Can you like do a tweet?
0:16:00 It would like get me 80% there.
0:16:03 And then I would just have to like do some modifications.
0:16:06 But like that prompt was like really hard to do all the time.
0:16:08 Like I didn’t want to have to redo that all the time.
0:16:13 And if I didn’t want to have to do it, like I just knew that my team wasn’t going to do it.
0:16:18 And so I just spent like two days basically making an app called Spiral where you can like,
0:16:19 you make a spiral.
0:16:22 You say like, I want to turn podcast transcripts into tweets.
0:16:26 You feed it some examples and then you have like a shareable widget that you can anytime
0:16:32 thereafter like generate more and more and more tweets without having to like reprompt anything.
0:16:33 And that’s going really well.
0:16:36 We have a full-time GM for that on that product.
0:16:36 His name is Danny.
0:16:37 He’s really talented.
0:16:41 We have almost 10,000 users for it and it’s growing really well.
0:16:45 And that’s an example of like you find this like use case in the general purpose tool
0:16:48 and then you can like unbundle it into like its own app.
0:16:53 And the way that I do that is it’s sort of the way Nathan, you mentioned earlier is like
0:16:56 they use cursor or Windsurf.
0:16:58 I kind of go back and forth between the two.
0:17:03 They have these like really, really good agentic building environments now where you just
0:17:06 type in a chat and it’s like, it just goes off and does stuff.
0:17:08 And then it’s like, it’s made like five files and it’s like,
0:17:10 I have no idea how it works, but it’s fine.
0:17:12 Sometimes it’s fine.
0:17:14 Quick question.
0:17:17 Do you actually know how to code yourself or are you letting AI write all the code?
0:17:18 Okay. You do know how to code.
0:17:19 Okay.
0:17:22 I do know how to code, but I am letting AI write all the code because I’m very lazy.
0:17:25 Yeah, but at least you’re able to do a little bit of extra troubleshooting
0:17:28 if the AI doesn’t figure it all out for you.
0:17:28 Yeah.
0:17:29 Okay.
0:17:32 And like, you know, what’s really nice is like 01 and 01 Pro.
0:17:37 Like sometimes these tools don’t have tool use and all the agent stuff uses tools.
0:17:41 Honestly, when 01 or maybe 03 has tool use, which I think is coming very, very soon,
0:17:43 it’s going to completely change everything again.
0:17:47 But I’ll often like, I’ll start in 01 or 01 Pro.
0:17:48 I’ll say make the plan.
0:17:49 I’d paste the plan into cursor.
0:17:51 Cursor will do stuff.
0:17:52 Then it will get stuck and have a bug.
0:17:55 And then I’ll just paste it back into 01 and then paste it back.
0:17:58 And then every once in a while, I’m like, I’m bored of copy pasting.
0:18:00 Like a lot of when I figure out how this actually works.
0:18:06 But also for me, like to be really clear, like my job right now is I’m mostly
0:18:08 prototyping new things and writing about them.
0:18:10 And then sometimes they become products, but once they’re actual products,
0:18:14 we have actual engineers who like actually are like in the code a lot.
0:18:17 And so I could be, you know, reading the code or whatever,
0:18:19 but I’m not doing that right now because it’s not really my job.
0:18:21 Yeah.
0:18:25 So I want to ask this to you because there’s been a lot of discussion around
0:18:27 like SaaS companies having no moat, right?
0:18:30 I’m sure you’ve heard that a million times over the last couple years.
0:18:32 What are your thoughts on that?
0:18:36 So let me preface this with saying like, I personally, like if I have a problem
0:18:39 I need to solve, it’s like a fairly simple problem.
0:18:43 I’ve stopped going out and looking to find if there’s a software product for it.
0:18:47 I will literally open up cursor and tell cursor what I want it to make.
0:18:49 And it’ll write a little Python script for me.
0:18:51 And it solves that recurring problem for me.
0:18:55 I know right now I’m probably like abnormal and doing that.
0:18:58 I don’t think most people sort of have that kind of workflow.
0:19:02 But I’m wondering if it’s going to get to a point where when somebody has a problem
0:19:05 they need to solve instead of turning to SaaS products and paying,
0:19:07 you know, some sort of recurring subscription,
0:19:11 they’re just going to go to one of the AI chat bots that are out there.
0:19:14 Maybe it’s ChatGPT, your Claude, or one of these tools and just say,
0:19:16 this is a problem I’m trying to solve.
0:19:19 And it says, okay, here’s a little script I wrote that solves the problem.
0:19:22 So I wonder what the future of SaaS looks like.
0:19:27 I mean, I think you’re definitely right in the sense that there are certain little things.
0:19:29 There are maybe like a thousand little things that come up every day
0:19:33 that like maybe I would have like looked for a tool for before.
0:19:38 Probably I would have just like suffered through like doing them manually or whatever
0:19:41 that now ChatGPT just does it for me and it’s amazing.
0:19:46 But I think there’s also a big difference between that and like a fully featured like SaaS app,
0:19:51 like for a power workflow, which I just think that things that are more specialized
0:19:55 for those kinds of users are going to win even when you have AI,
0:19:59 even if you’ve theoretically have AGI and that there are room for those kinds of tools,
0:20:01 whether they look like traditional B2B SaaS,
0:20:04 or whether they look a little bit more like an AI native version of that,
0:20:07 whatever that looks like, I think it’s going to be more like that.
0:20:10 And I think there’s also a really interesting question about like
0:20:13 how much SaaS is actually the right business model.
0:20:17 What I’ve been seeing a lot more of and I think is probably a little bit more of a
0:20:23 AI native business model is paying per completed task or paying for the value that you get,
0:20:29 basically, because when you have AI, you can pay for the job to be done versus just like the time
0:20:30 or by the month.
0:20:36 And I think that’s really interesting because SaaS is such a much better business model
0:20:41 that if you are currently a SaaS company, you’d probably rather stay a SaaS company.
0:20:44 And that means that there’s a lot of opportunities for startups
0:20:47 that don’t necessarily need to keep that model.
0:20:49 Yeah, performance-based SaaS would be incredible.
0:20:50 Like, yeah, that’s an amazing idea.
0:20:51 Yeah, I think that’s going to happen.
0:20:53 I mean, you see it already in certain places,
0:20:56 like the LLM APIs or all paper use or whatever,
0:20:58 but I think that’ll filter up through the stack.
0:21:01 I feel like I’m your worst guest ever.
0:21:05 I’m like SEO is dying and SaaS is dead.
0:21:10 I’m so sorry.
0:21:14 I mean, we’ve got other tools that I think are cool.
0:21:17 We have a call called Sparkle, which is a file organizer.
0:21:21 So basically, it just sits on your desktop and just automatically organizes your desktop,
0:21:24 your downloads and your documents with AI, which is like really cool and very useful.
0:21:27 Everything is f***ing messed for me.
0:21:30 So like, but now it’s not and I didn’t have to do anything.
0:21:30 That’s amazing.
0:21:32 I’m always like trying to think about like how to organize my files.
0:21:34 I’m like, oh, one day I’m going to organize them all.
0:21:36 They’re going to be, and I was like, wait a minute,
0:21:37 AI is going to eventually do that for me.
0:21:39 So I was like, it’s awesome that you’re doing something like that.
0:21:40 Yeah, it’s just a matter of time.
0:21:43 Another thing that we’ve been working on,
0:21:44 it’s actually one of our writers, Michael Taylor,
0:21:47 has been working on and we’re kind of doing it with him,
0:21:52 is AI is quite good at acting like a particular kind of person,
0:21:54 like simulating a particular kind of person.
0:21:59 And Michael built this tool called Rally that we’ve been using internally.
0:22:02 I think we’re going to make like every column out of it
0:22:06 and then maybe launch it as a product that’s still TBD.
0:22:08 And it really depends on if Michael wants to do it with us.
0:22:09 But I think it’ll be really cool.
0:22:12 And basically what it does is like, for example,
0:22:15 if we’re coming up with headlines for a podcast,
0:22:19 what we can do is we spin up an audience of like a thousand every people
0:22:21 or a thousand hacker news people.
0:22:24 And then we can be like, okay, here’s my three titles.
0:22:25 Which ones do you like?
0:22:27 And it will just like survey the crowd
0:22:30 and then tell us what the crowd of LLMs thinks.
0:22:32 And it’s really cool. It works.
0:22:34 I thought you were going the opposite side.
0:22:36 So I actually thought it was going to be like the person
0:22:38 who’s helping you create the taglines or whatever.
0:22:42 You fed it a bunch of books from some famous person who’s an expert at that.
0:22:44 But you’re doing the opposite, like the audience side.
0:22:45 Yeah, that’s incredible.
0:22:45 Exactly.
0:22:47 So it’s sort of like wisdom of the crowds,
0:22:50 like ask the audience type stuff, which like now you just have to try stuff
0:22:52 where you have to do A/B testing and it takes a long time.
0:22:54 You need a big audience or whatever.
0:22:56 And if you use this, you don’t have to.
0:22:58 And it’s still very early and it remains to be seen.
0:23:03 But like I feel quite confident that the results will align to some degree
0:23:05 with like what you would actually find out in the world.
0:23:06 And that’s really cool.
0:23:09 Wow.
0:23:11 So I actually do have one other sort of little rabbit hole.
0:23:12 I do want to go down with you.
0:23:16 So I believe you do some investing outside of every as well, right?
0:23:17 So you have some investments.
0:23:19 I believe you work with Sequoia Capital as well.
0:23:20 Is that correct?
0:23:22 What are you looking for in investments?
0:23:24 I’m assuming you’re probably focused more on AI.
0:23:25 I could be totally wrong.
0:23:26 I’m just sort of guessing.
0:23:28 But what are you looking for in investments?
0:23:30 Like how do you find something that stands out to you?
0:23:37 The investments that I’ve done usually are places where I’ve had a particular thesis
0:23:38 I’ve been writing about and thinking about.
0:23:42 Or I just know the founder and I know they’re really awesome.
0:23:45 And I just like want to get into whatever they do.
0:23:47 Yeah, I do invest in AI.
0:23:52 I have found honestly that a lot of AI companies are just raising at such gigantic
0:23:57 valuations that I haven’t actually pulled the trigger on that many deals,
0:24:00 even if I really like the person or really like the thing they’re building.
0:24:04 Because it’s just like a $50 million seed round is just like crazy to me.
0:24:09 But like generally, it’s like, is it a thesis in a space that I care about
0:24:10 or like an idea that I care about?
0:24:13 And I have a bunch of articles that sort of lay out all that stuff.
0:24:17 And I can give you a little preview about the things I care about right now.
0:24:19 Like for example, one is the future of science,
0:24:23 like how AI might change how we do science is a big one for me.
0:24:25 Another one is yeah, just really like the founder.
0:24:27 And then the last one is just do I use the product?
0:24:28 Do I like it?
0:24:30 I think the proof’s in the pudding a lot in those situations.
0:24:33 And you can tell immediately when someone’s built something awesome
0:24:35 because you just feel it.
0:24:35 Yeah, yeah.
0:24:37 So when you say like you have a thesis on it,
0:24:40 you’re kind of saying like this is something I wish was in the world,
0:24:42 but then you actually find something that does that.
0:24:44 That’s something that you’d want to invest in.
0:24:45 Yeah, exactly.
0:24:45 Very cool.
0:24:46 All right.
0:24:47 I assume some way the world’s changing and then health,
0:24:50 a product would fit into that changing world.
0:24:50 Yeah.
0:24:51 I love that stuff.
0:24:54 I especially get really excited about a lot of the science stuff.
0:24:56 Like I was really nerding out about like that matter gen
0:24:58 that just came out from Microsoft the other day.
0:25:01 I think that’s something that’s really cool and really exciting.
0:25:03 And I think there’s a lot more stuff that’s going to pop up
0:25:06 sort of in that world, you know, this year.
0:25:08 I think we’re going to see a lot of sort of materials,
0:25:11 science sort of things come out of AI this year.
0:25:13 And so that to me is super, super exciting.
0:25:17 But on that note, I really, really appreciate you hanging out with us today.
0:25:20 Where should people go after listening to this episode?
0:25:23 Is there a specific social media account you want to shout out?
0:25:25 Your website, let us know where to check you out.
0:25:26 Thanks so much for having me.
0:25:32 You can find me on X at Dan Shipper, S-H-I-P-P-E-R or at every also on X.
0:25:34 And then the website is every.to.
0:25:35 You can find our newsletter.
0:25:38 You can find all of our products, the courses we offer,
0:25:39 all that kind of stuff.
0:25:40 And it’s been awesome chatting with you guys.
0:25:41 Yeah, it’s been great.
0:25:42 Awesome.
0:25:43 Yeah, thanks so much, Dan.
0:25:44 Really, really appreciate you.
0:25:46 And thanks everybody to listening to this episode.
0:25:50 If you haven’t already, make sure you subscribe wherever you listen to podcasts.
0:25:51 We’re also available on YouTube.
0:25:53 If you want to see our faces, actually talk to each other.
0:25:55 And thanks so much for tuning in.
0:25:56 We’ll see you in the next one.
0:25:59 [MUSIC PLAYING]
0:26:02 [MUSIC PLAYING]
0:26:06 [MUSIC PLAYING]
0:26:08 (upbeat music)

0:00:02 (upbeat music)
0:00:06 – Hey, welcome back to the Next Wave Podcast.
0:00:07 I’m Matt Wolf.
0:00:08 I’m here with Nathan Lanz.
0:00:11 Today we’re joined by Logan Kilpatrick,
0:00:15 who is the Senior Project Manager over at Google DeepMind.
0:00:17 And the day we’re recording this episode
0:00:19 is the same day that Google just released
0:00:22 a whole bunch of new AI tools.
0:00:26 Gemini Flash 2.0, Gemini Flash 2.0 Lite,
0:00:30 Gemini Flash 2.0 Pro, all sorts of really, really cool stuff
0:00:32 coming out of Google right now
0:00:34 and Logan’s gonna break it all down.
0:00:36 And you’re gonna get a pretty grand overview
0:00:39 of where the AI world is headed according to Google.
0:00:41 So let’s just go ahead and dive right in
0:00:43 with Logan Kilpatrick.
0:00:44 – Thank you so much for joining us.
0:00:46 It’s probably a really busy day.
0:00:47 So I really appreciate you taking the time
0:00:48 to join us today.
0:00:50 – Yeah, I’m excited to catch up with you both
0:00:51 and talk about all things Gemini
0:00:53 and what’s happening in the AI world.
0:00:55 – Well, so this is actually your second time on the show.
0:00:58 So we’ve already kind of dove into some of the backstory
0:01:00 and introduced people to you in a past episode.
0:01:02 So let’s just jump straight into it.
0:01:06 Can you break down what is 2.0 Flash, 2.0 Flash Lite,
0:01:08 2.0 Pro, like what are the differences?
0:01:10 What’s better about these models
0:01:12 than what was out prior to them?
0:01:14 – I think this is an exciting moment for us
0:01:16 just because of like the amount of effort and work
0:01:18 that’s gone into bringing Gemini 2.0
0:01:19 actually into the world.
0:01:21 And Matt, you were there, Nathan,
0:01:23 I don’t remember if you were there at I/O last year,
0:01:27 but we announced 1.5 Flash and the long context
0:01:29 and a bunch of this other stuff last May.
0:01:30 So like literally less than a year ago.
0:01:33 And for the last year-ish, Flash has been this
0:01:36 like wild success story for us
0:01:38 of building a model that developers really love.
0:01:41 And a lot of that is rooted in like the right trade-offs
0:01:44 of like cost, intelligence, performance, capabilities.
0:01:46 And if you look at the 1.5 Flash model
0:01:48 and you think about like, how do we do this better?
0:01:50 It’s like, you have to make it more powerful.
0:01:52 You have to give it more capabilities.
0:01:55 You have to do that all while not making it cost
0:01:57 a lot more money for developers.
0:02:00 And it feels like we pulled a rabbit out of the hat
0:02:01 to a certain extent with 2.0 Flash
0:02:04 because like the actual cost for developers,
0:02:06 it was historically like seven and a half cents
0:02:08 per million tokens.
0:02:09 Now it’s 10 cents.
0:02:11 So the blended cost is actually less for this model.
0:02:13 And we did all that while like,
0:02:15 this model is actually better than Pro.
0:02:16 It has all these capabilities.
0:02:18 It’s natively agentic.
0:02:21 It has search built into it as code execution built into it.
0:02:23 – Yeah, it’s just exciting for me as a developer.
0:02:27 Like I think ultimately you remove the cost barriers
0:02:29 and all these other things for people to build
0:02:30 really cool stuff.
0:02:31 And like that’s what enables the world
0:02:33 to make really cool products.
0:02:34 So I’m super excited.
0:02:36 So that’s the sort of headline for Flash
0:02:37 is better, faster, cheaper,
0:02:40 which continues to be sort of the tagline.
0:02:41 I need to get my Gemini t-shirts
0:02:43 that say better, faster, cheaper on them.
0:02:46 – I read that it’s better than GPT-40,
0:02:47 but like 20 times cheaper.
0:02:49 Is that like roughly correct?
0:02:51 – Yeah, which is just crazy to me.
0:02:53 And I think like we’ve got a lot of work to do.
0:02:56 I think one of the dimensions of the Gemini story
0:02:58 is like we continue to put out really great models.
0:02:59 I think we need to do a great job as well
0:03:01 of like going and telling the world
0:03:03 about this technology that we’re building.
0:03:05 ‘Cause I don’t think people really understand
0:03:07 and actually for a lot of developers,
0:03:08 the cost is the reason in many cases
0:03:10 they don’t build the stuff that they want to build.
0:03:13 It’s like, I can’t afford to put this thing into production.
0:03:14 It’s too expensive.
0:03:17 So I think Flash is really important in that story.
0:03:18 But we also landed 2.0 Pro.
0:03:22 We also landed an even cheaper version of Flash Light,
0:03:23 which sort of has the capabilities
0:03:24 pared down a little bit,
0:03:26 but makes it so that we can keep delivering
0:03:30 on that like frontier cost performance sort of trade often.
0:03:31 And that model is in preview
0:03:33 and it’ll go GA in the next few weeks
0:03:35 as we iron out the last few bugs.
0:03:38 And I think Pro gives me a lot of excitement
0:03:43 about those whole narrative of pretraining, being dead.
0:03:45 An interesting sort of realization
0:03:47 I had after our conversation with Jack Ray,
0:03:49 who’s one of the co-leads for the reasoning models
0:03:53 in DeepMind is there’s this non-linear amount
0:03:57 of extra effort it takes to make the models
0:03:59 continue to get better.
0:04:00 Like you look at like, okay,
0:04:03 what does 3% mean on some benchmark?
0:04:06 Like you think 3% and we think of like the normal world
0:04:10 where 3% is like actually 3% and like in the model world,
0:04:14 3% is actually like a 25% increase
0:04:16 in like the amount of efforts that went into this.
0:04:19 But also that 3% is like the difference
0:04:22 between unlocking a bunch of capabilities
0:04:25 and a bunch of use cases that like just didn’t work before
0:04:28 because like the thing failing 3% to 4% of the time
0:04:31 versus not is the difference between you putting AI
0:04:33 into production at your company and not.
0:04:35 So it actually matters a lot.
0:04:36 And that’s why I think we continue to push
0:04:37 on that frontier.
0:04:41 – So what’s the big difference between like these new models
0:04:44 and the last ones as far as like how they were created,
0:04:46 is it like more parameters that are being trained on?
0:04:49 Obviously, the big narrative like you just mentioned
0:04:52 with things like deep seek and O3 and things like that
0:04:55 from open AI are the sort of what happens at inference, right?
0:04:58 When somebody enters a prompt, it does all of this thinking
0:05:00 and that’s really what they’re sort of like pushing on
0:05:02 is like the next sort of breakthrough.
0:05:03 Like what sort of breakthroughs,
0:05:06 what changed between the last models and this one
0:05:07 to make this one so much better?
0:05:09 – Yeah, there’s two dimensions of this.
0:05:12 One, it’s a story of the really difficult work
0:05:14 of doing algorithmic improvements and breakthroughs.
0:05:16 And I think like the team at DeepMind
0:05:17 stuff way beyond my understanding
0:05:20 as far as how they’re able to make this continue to work.
0:05:23 So I think there’s like core fundamental research
0:05:24 advancements that are happening.
0:05:26 And there’s a lot of like data efficiency wins as well,
0:05:27 which is also exciting.
0:05:30 But as far as like new capabilities of these models,
0:05:34 I think the two big ones is when Gemini was first announced,
0:05:36 it was announced as this model that’s natively multimodal.
0:05:39 And it was natively multimodal in the sort of input sense
0:05:42 that it could really understand the videos,
0:05:45 audios, images that it was being given.
0:05:47 And that was one of the main differentiators.
0:05:50 Today, the model’s actually capable of doing that
0:05:51 except on the output sense,
0:05:53 which I think was a huge jump for us.
0:05:54 And it actually requires again,
0:05:58 a bunch of like non-trivial amount of engineering work
0:06:00 in order to make the models capable of doing that.
0:06:02 I had an interesting conversation a few weeks ago
0:06:04 with someone on our research team,
0:06:05 which reminded me of this.
0:06:07 Someone asked a question of like,
0:06:09 why does it matter if the models are capable
0:06:12 of natively outputting these multimodal capabilities?
0:06:14 Like we have really great text-to-speech models.
0:06:16 We have great speech-to-text models.
0:06:18 We have great image generation models.
0:06:20 Like why is it cool that the model can do this natively?
0:06:23 And there’s all of these really great examples,
0:06:26 like a calculator versus like, I don’t know,
0:06:29 an AI model that has access to code execution.
0:06:31 Like the code execution version can really like solve
0:06:33 these problems in a really complicated way
0:06:36 that you wouldn’t otherwise be able to,
0:06:38 or at least that the effort is required
0:06:40 as you on the user of the models.
0:06:43 And I think that’s the world of these custom domain-specific
0:06:45 models, like image generation and audio generation,
0:06:48 versus the native capability really feels like
0:06:50 the model can just do the heavy lifting for you,
0:06:52 which is really interesting.
0:06:55 – So right now, can Gemini actually output like an image
0:06:57 if I give it a prompt to generate an image?
0:06:59 Does it generate an image right now?
0:07:00 – Not accessible to everyone yet.
0:07:01 And I think this is the gap.
0:07:03 So we have it internally and folks are using it
0:07:05 in our early access program
0:07:07 and we should get you both early access
0:07:08 to play around with it and test it out.
0:07:11 And we’ll roll it out more broadly soon,
0:07:12 which I’m excited about.
0:07:15 But that sort of same line of thinking
0:07:17 is what takes us to like native tool use as well.
0:07:19 And like native tool use is available to everyone.
0:07:22 And it’s like the model was trained,
0:07:25 knowing how to differentiate questions
0:07:27 that it should go and search the internet for
0:07:28 or questions that it needs to use a tool
0:07:30 like code execution for.
0:07:33 So you get like all of those like silly examples
0:07:34 where the model would be like,
0:07:36 let me try to solve this math problem,
0:07:38 which I know I’m not going to be able to solve
0:07:40 just because you asked me to with code execution,
0:07:42 like it knows it needs to use that tool.
0:07:44 And there’s a whole bunch of verticals
0:07:45 where like the performance goes up
0:07:46 significantly because of that.
0:07:49 – So Gemini is actually generating the image.
0:07:51 It’s not going and calling upon like,
0:07:53 imagine three to generate the image.
0:07:55 It’s actually Gemini who’s creating that image
0:07:57 when it does generate an image.
0:07:57 – Exactly.
0:07:59 And I’ll push on getting you both access
0:08:00 after this conversation
0:08:02 because I think the world knowledge piece
0:08:04 really highlights like why this matters.
0:08:06 And there’s like a bunch of examples
0:08:09 that I played around with of like pictures of a room
0:08:11 and like having the image change
0:08:15 based on these like really complex nuanced prompts
0:08:16 around moving objects in certain ways.
0:08:19 Like things that if it doesn’t have world knowledge
0:08:21 and understand like it understands physics
0:08:22 and understands all these things
0:08:25 that again require the world knowledge piece.
0:08:28 And I think it’s actually there’s some interesting trends
0:08:30 of what is the outcome of like being able
0:08:32 to take other domain specific models
0:08:35 and bring them into these LLMs that have world knowledge.
0:08:38 I think there’ll be some really cool capabilities
0:08:39 that like we’re not thinking of today
0:08:41 that this is going to enable,
0:08:43 which yeah, it gets me excited for people.
0:08:45 – Yeah, it seems like that’s kind of required
0:08:46 for this also to work in like everything
0:08:48 from like gaming to robotics
0:08:50 to it actually have an understanding of the world.
0:08:52 – Oh yeah, 100%.
0:08:54 – Yeah, yeah, I mean, I actually had the opportunity
0:08:57 to go out to London, go visit the DeepMind offices
0:08:59 and got to play with Astra on the phone.
0:09:02 And I mean, that was like the first taste I got
0:09:07 of like an actual useful AI assistant out in the real world.
0:09:10 And I don’t know if that was using Gemini 2.0 Flash
0:09:13 or if that was actually using the full Gemini 2.0 yet
0:09:17 at the time, but it was definitely a very, very impressive model
0:09:20 to actually see the tool use in real life
0:09:21 and just be able to walk around
0:09:24 and it understand images and understand video
0:09:26 and understand audio and understand text.
0:09:29 And it was all built from the ground up
0:09:33 to understand that stuff as opposed to like look at an image,
0:09:36 use OCR to figure out the text on the image
0:09:38 and then pull in the text or listen to the audio,
0:09:41 transcribe the audio to text and then use the text.
0:09:43 It’s actually understanding what it’s seeing,
0:09:46 what it’s hearing, which I think is like one
0:09:50 of the major differentiators about like what Gemini is doing
0:09:52 that you don’t see the other models doing yet.
0:09:54 So it’s super, super impressive.
0:09:56 – Yeah, I think the other piece of this is
0:09:58 other than just the raw capabilities
0:09:59 from a complexity standpoint,
0:10:01 it means that like developers building stuff,
0:10:04 you don’t have to go and like do a ton of scaffolding work
0:10:05 in order to like make this happen.
0:10:07 It’s like the overall complexity of your application
0:10:10 when the model is just able to sort of take in a bunch
0:10:12 of things and put out a bunch of things
0:10:13 makes life incredibly easy
0:10:16 and you don’t have to deal with like frameworks
0:10:17 on frameworks on frameworks.
0:10:19 So much of the agent world is like,
0:10:21 hey, the models actually aren’t that good
0:10:22 at doing some of these things.
0:10:24 And like the way that we supplant that
0:10:27 is by like building a bunch of scaffolding and frameworks,
0:10:29 that’s where a lot of developers are focused today.
0:10:30 And actually there’s going to be a moment
0:10:31 where like all of a sudden the model capabilities
0:10:33 are just like good enough that it kind of works.
0:10:35 And then people are going to be like,
0:10:36 well, why do I have all this scaffolding
0:10:38 that’s doing these things for me
0:10:40 that the models can just do out of the box now?
0:10:42 So it’ll be interesting to see how that plays out.
0:10:45 – Yeah, I know like 2025 is going to be sort of
0:10:46 the year of the agent, right?
0:10:49 That term has already been thrown around quite a bit.
0:10:50 But I also feel like everybody kind of has
0:10:52 a different definition of an agent.
0:10:54 You know, some people will look at something
0:10:56 that like you can build over on make.com or Zapier
0:10:58 where it’s tying different tools together
0:11:00 using APIs as an agent.
0:11:02 But I’m curious, does Google and DeepMind,
0:11:04 do they actually have like an internal definition
0:11:06 of an agent that they’re shooting for?
0:11:08 Do you think we actually have agents now
0:11:09 based on their definition?
0:11:11 Where does Google stand on agents?
0:11:14 – I actually don’t know what our like formal definition
0:11:15 of agents are.
0:11:18 I have tried a bunch of the agent products
0:11:20 and historically haven’t been super impressed
0:11:21 at what they’re capable of.
0:11:22 I think we’re just not there yet.
0:11:25 The thing that I want and I think a lot of users
0:11:28 want this as well is just the models to be proactive.
0:11:32 And like all of the products of today that build on AI
0:11:35 require me to basically change my workflows
0:11:38 or put in extra work or put in extra effort
0:11:41 like through this sort of guys of like,
0:11:42 oh, this is actually going to save you time
0:11:43 if you do this thing.
0:11:45 And like really what I want is like the models
0:11:47 to just be like looking at the stuff
0:11:48 that I give them access to
0:11:50 and like coming up with ways to be useful
0:11:51 and like save me time.
0:11:54 And like, yeah, it’s going to get some things wrong
0:11:56 but like I don’t want to have to be the one
0:11:57 in the driver’s seat all the time.
0:11:59 And it feels like today’s AI agents,
0:12:01 again, because the models aren’t good enough
0:12:03 like have to be proactive.
0:12:05 Like it requires the proactiveness of the human.
0:12:08 And I think once that role reversal switches,
0:12:11 I think that’s where we see like billion agents scale
0:12:14 deployments like all of a sudden just like happening
0:12:14 and working.
0:12:17 And this is also where like things get crazy
0:12:17 again with compute.
0:12:19 Cause again, like actually if you look at
0:12:21 how compute is being used today,
0:12:23 it’s in a lot of cases like this one to one correlation
0:12:27 between like a human input and the token output.
0:12:30 And I think the future is thousands and thousands
0:12:32 of X more usage of AI happening
0:12:34 by the agents themselves than by humans,
0:12:36 which will be fascinating to see play out.
0:12:40 – Hey, we’ll be right back to the show.
0:12:42 But first I want to talk about another podcast
0:12:43 I know you’re going to love.
0:12:45 It’s called entrepreneurs on fire.
0:12:46 And it’s hosted by John Lee Dumas
0:12:49 available now on the HubSpot podcast network.
0:12:51 Entrepreneurs on fire stokes inspiration
0:12:54 and share strategies to fire up your entrepreneurial journey
0:12:56 and create the life you’ve always dreamed of.
0:12:59 The show is jam packed with unlimited energy,
0:13:00 value and consistency.
0:13:02 And really, you know, if you like fast-paced
0:13:04 and packed with value stories
0:13:07 and you love entrepreneurship, this is the show for you.
0:13:09 And recently they had a great episode
0:13:12 about how women are taking over remote sales
0:13:13 with Brooke Triplett.
0:13:15 It was a fantastic episode.
0:13:16 I learned a ton.
0:13:17 I highly suggest you check out the show.
0:13:19 So listen to entrepreneurs on fire
0:13:21 wherever you get your podcasts.
0:13:26 – Yeah, no, I think what you described
0:13:28 is sort of what I envision of an agent
0:13:29 is almost like predictive.
0:13:32 Like it sort of figures out what you need before you need it
0:13:34 and then make suggestions based on that.
0:13:36 So I think that’s a world that I’m really excited
0:13:37 to get into.
0:13:39 But I do want to touch on the word compute
0:13:41 that you just mentioned for a second
0:13:43 ’cause obviously there was a bit of like a, you know
0:13:45 a freak out, so to speak in the US
0:13:47 when that deep seek model came out
0:13:49 and everybody thought that, well,
0:13:52 this deep seek model uses a lot less compute
0:13:55 than these other models that have been trained.
0:13:57 So therefore, you know,
0:13:59 NVIDIA GPUs are no longer necessary.
0:14:02 And then we saw NVIDIA sort of lose some market share
0:14:03 as a result of it.
0:14:05 But I’m just curious, like what are your thoughts
0:14:06 on the compute?
0:14:07 Because I saw all of that happening
0:14:09 and I thought it was so bizarre.
0:14:11 I was like, this seems like a bullish sign
0:14:13 for NVIDIA to me, not a bearish sign.
0:14:15 Like what’s going on here?
0:14:16 But I’m curious, like what’s your take on
0:14:17 what happened there?
0:14:20 – Yeah, there’s a lot of complexity to that story
0:14:22 and parts of the story that I don’t want to touch on
0:14:25 but the thing that I do want to touch on is like
0:14:26 if you look around the world,
0:14:28 I think this example that I just gave of like
0:14:31 who is in the driver’s seat of using AI today?
0:14:34 And like today it’s humans that are in the driver’s seat
0:14:36 and like we’re just inherently bounded
0:14:38 by the amount of humans who are using AI
0:14:39 because, you know, it just takes a while
0:14:42 for technology to assimilate into culture
0:14:44 and real use cases and all that stuff.
0:14:46 You know, we’re on this exponential right now.
0:14:48 I think as soon as agents start to take off
0:14:51 that exponential becomes a straight line up into the right.
0:14:54 Like I think it’s gonna be pretty profound
0:14:56 because like again, the challenge is
0:14:59 the human process just doesn’t scale
0:15:02 and the agent process is going to scale
0:15:03 which is going to be really interesting.
0:15:05 Like I have 10,000 emails I haven’t read
0:15:06 in the last three months.
0:15:07 – Right, I was thinking emails.
0:15:08 First thing I was thinking of,
0:15:10 I wanted to handle all of that for me.
0:15:11 I don’t want to think about any of it.
0:15:12 – Yeah, it’s gonna be wonderful.
0:15:14 And really like go in and find the things
0:15:16 like I know there’s things that I should be doing
0:15:18 that would create value.
0:15:19 – Missed opportunities.
0:15:20 – 100%.
0:15:22 And there’s like so many of those things
0:15:24 where if you also like think about like
0:15:26 what’s the economic value of all that work?
0:15:29 Like one of the crazy frames of mind
0:15:31 that I look at the world through is like,
0:15:33 you look at the world and like the world is just filled
0:15:34 with all this inefficiency.
0:15:36 And like it’s beautiful in many ways,
0:15:39 but like it’s also this really cool opportunity
0:15:41 if you can be the one to create something
0:15:43 that sort of makes people more productive
0:15:46 and explicitly makes them more productive
0:15:47 at maybe the things they don’t want to do.
0:15:49 And I feel like that’s the other part
0:15:51 of this agent story and this compute story,
0:15:53 which is a lot of the products
0:15:55 that I see people building are actually going
0:15:59 after the things that people really like doing.
0:16:02 And like maybe like shopping is this sort of
0:16:03 tongue-in-cheek example of this.
0:16:05 ‘Cause some people really like shopping
0:16:05 and some people don’t.
0:16:07 But like if I told my girlfriend
0:16:09 that we could never go shopping again together
0:16:13 and we could never go try out different experiences
0:16:15 and go check out the vibe of different stores,
0:16:17 like there’s so much of this is like
0:16:19 such a fundamental part of the human condition actually
0:16:22 of like going and seeing these different places
0:16:24 and like it’s really baked into who we are.
0:16:27 And that’s such like a traditional example
0:16:29 of like what agents are going to do for people.
0:16:30 And it’s odd to me.
0:16:33 I feel like people have like kind of misconstrued
0:16:35 what the value creation is going to be
0:16:36 in some of these examples.
0:16:37 – I mean, I think that’s like the bias
0:16:39 from like a, you know, Silicon Valley nerds
0:16:40 building this stuff, right?
0:16:41 – Yeah, yeah.
0:16:42 – I don’t want to shop.
0:16:43 I just want to be automated for me.
0:16:46 And it’s like, that’s just like a small subject
0:16:46 of the world.
0:16:47 Like, you know, when I shop,
0:16:49 I just like go in and get what I want.
0:16:50 There’s like two options.
0:16:51 Okay, I like this one better.
0:16:51 I get it.
0:16:54 But my wife, she just loves checking out stores
0:16:55 and different shops.
0:16:56 And like she would hate the idea of like skipping
0:16:58 all of that would just make no sense to her at all.
0:17:01 – Yeah, I think there’s an underlying story here,
0:17:02 which is like, first of all,
0:17:03 there’s a lot of variance in human preference,
0:17:07 but also there’s like ways about going about a certain task
0:17:09 that like make it interesting or not.
0:17:10 And like, I like shopping too,
0:17:13 if it’s like me getting to do it on the terms that I want.
0:17:15 And like, it will be interesting to see like,
0:17:17 how can agents and some of these products
0:17:19 like actually help create that experience?
0:17:21 And this goes back to this deep seek narrative
0:17:23 around like the value creation
0:17:24 happening at the application layer.
0:17:26 And it really does feel like this is true.
0:17:28 Like if you look back two years ago,
0:17:29 the narrative was, you know,
0:17:32 all these companies are just rappers on top of AI.
0:17:33 There’s no value creation.
0:17:35 All the creation of the value creations in the tokens,
0:17:38 like don’t spend your time thinking about these companies.
0:17:41 And it’s so funny how quickly this like flip-flops
0:17:43 back and forth between like,
0:17:45 now all the value creation is at the application layer
0:17:48 and like LLMs are this commodity thing
0:17:49 that no one should think about.
0:17:51 I enjoy watching it all play out.
0:17:51 It’s fun.
0:17:52 – So how do you think it’s gonna play out?
0:17:53 Because like right now,
0:17:55 Google’s kind of focused on developers, right?
0:17:57 More than consumers, correct?
0:17:58 – So on one hand,
0:18:01 like I spend all my time thinking about developer stuff.
0:18:02 Google’s got a ton of other people
0:18:04 who are doing consumer stuff.
0:18:05 I think a good example of this is like,
0:18:10 Gemini is both in search and through the Gemini app,
0:18:13 like, you know, deployed across like billion user scale.
0:18:15 Like there are literally billions of people
0:18:18 who are interacting with outputs and the models themselves,
0:18:20 which is crazy to think about.
0:18:22 And like it’s a very consumer forward use case
0:18:23 for those folks.
0:18:25 And I think it’s also like still incredibly early
0:18:27 for Gemini in search.
0:18:29 And there’s some interesting stories around that stuff.
0:18:32 But I mean, I personally am incredibly bullish
0:18:35 on the infrastructure layer and like the infra tooling.
0:18:38 And I think like actually a good example of this.
0:18:41 And you see this in some of Sam’s recent tweets
0:18:44 about sort of open AI and the pro subscription
0:18:46 are sort of good examples of this,
0:18:48 which is builders at the application layer
0:18:52 have a lot of tension with building more AI.
0:18:53 And like actually back to the thread
0:18:56 of having AI be more proactive.
0:18:58 This is why I believe in flash so much.
0:19:00 And I believe in the direction that we’re going
0:19:02 as far as like reducing the cost for developers
0:19:04 while continuing to push the performance frontier
0:19:07 is because the story of AI to me is a story
0:19:09 of like the actual infrastructure,
0:19:11 incentivizing developers to not use it.
0:19:14 Like you literally have an economic incentive
0:19:16 not to use AI because it costs you money.
0:19:18 And like the more AI you build into your product,
0:19:19 the more expensive it is.
0:19:21 And like the more margin pressure you have
0:19:23 as an application builder.
0:19:24 – It’s scary too, right?
0:19:25 To build an app and like all of a sudden
0:19:27 you get like a gigantic bill
0:19:28 because people are using your thing
0:19:30 and you haven’t figured out how to properly monetize it yet.
0:19:31 As you know, someone who creates companies,
0:19:33 it’s kind of intimidating.
0:19:33 – Yeah, 100%.
0:19:35 That is like the realist reaction
0:19:38 and like also just like the truest reaction developers
0:19:40 have to the cost of the technology.
0:19:43 So back to the point of like where the value creation happens.
0:19:46 I think the nice thing for infrastructure providers
0:19:48 is you have a fixed margin.
0:19:51 So like, you know exactly how much money you’re gonna make
0:19:54 by providing some infrastructure at the application layer.
0:19:56 You’re like, you’re constantly incentivized
0:20:00 to almost like not add additional stuff.
0:20:01 And I think this has been the story
0:20:03 for like the chat GBT plus and pro subscription
0:20:06 is like they built a subscription for $20 a month
0:20:08 and they realized, hey, we actually can’t give people
0:20:09 all of these things anymore.
0:20:11 We have to make something different.
0:20:13 And like even at $200 a month,
0:20:16 it’s like not a break even scenario for them yet.
0:20:19 So it’s super interesting to see that play out
0:20:21 and it’s a lot of food for thought
0:20:22 for people who are building stuff to be like,
0:20:25 are there new economic incentive mechanisms
0:20:28 that you can create as you’re building a product
0:20:31 more so than just like charging a $20 a month subscription.
0:20:33 And like the one example I can think of of this
0:20:35 that has made me think is,
0:20:37 I don’t know if you all are familiar with OpenRouter,
0:20:40 but it’s a surface that lets you sort of swap
0:20:41 in and out different language models.
0:20:44 OpenRouter’s product leaderboard,
0:20:46 I’m pretty sure they give you discounts on tokens
0:20:49 and stuff like that for showing you in certain ways
0:20:51 and some metadata passing back and forth
0:20:53 so they can understand like how people are generally
0:20:54 using AI models.
0:20:57 So like some interesting things like that.
0:21:00 Alex Atala, who’s the CEO who previously worked on OpenSea,
0:21:03 he has this quote, which always rings in my head,
0:21:06 which is like usage is the ultimate benchmark.
0:21:08 How many people are using your model or your thing
0:21:10 is like the proof point of success,
0:21:11 not all these other benchmarks
0:21:12 that people are chasing after.
0:21:14 So super interesting platform.
0:21:15 – Do they actually publish like a leaderboard
0:21:17 similar to like what LMSYS does?
0:21:18 – Exactly. – Okay.
0:21:21 – Checking it out right now at openrouter.ai
0:21:23 and it’s a forward slash rankings.
0:21:24 – Oh, cool.
0:21:26 I’m curious, a little bit of a topic shift here,
0:21:29 but I know you’re a proponent of open source
0:21:31 and Google obviously has their Gemma models.
0:21:34 Are there any updates, any idea of what’s going on with Gemma
0:21:35 and what we can expect next
0:21:37 out of the open source side of things?
0:21:39 – Yeah, I think that this is also the piece
0:21:42 that makes me excited about what we’re doing at Google,
0:21:45 which is it really is the exact same research
0:21:46 that powers the Gemma and I models
0:21:48 that ends up making the Gemma models.
0:21:53 And Gemma two was, I think it’s like the second most downloaded
0:21:55 open source model in existence, which is awesome to see.
0:21:57 And Gemma three is definitely going to happen.
0:21:59 I think the timeline is soon.
0:22:02 So you’ll hear more and y’all should do an episode
0:22:03 with some of the Gemma folks
0:22:04 ’cause there’s lots of cool stuff coming.
0:22:05 – Yeah, for sure.
0:22:07 – They’ve been doing a lot of like interesting fine tunes
0:22:08 for different use cases.
0:22:10 Like they have a version for RAG,
0:22:12 they have a version for vision.
0:22:16 I think they’re probably gonna do some agent stuff as well.
0:22:18 So like there’s lots of really cool explorations happening
0:22:20 on making those open models.
0:22:21 – Super cool.
0:22:23 I wanna talk about Imagine three,
0:22:26 which I just learned is actually pronounced Imagine three.
0:22:27 Like I’ve been calling it Imagine.
0:22:29 I always thought it was like Image Generator.
0:22:32 So like Image Gen, but I heard you pronounce it Imagine.
0:22:34 So now I’ll start saying it that way.
0:22:37 But you mentioned that there’s some updates with that as well.
0:22:38 What can you tell us about that?
0:22:39 – Yeah, and you’re in good company.
0:22:42 Don’t worry, I swear, like 50% of the meetings I’m in,
0:22:44 I hear Imagine, 50% I hear Image Gen.
0:22:47 So yeah, there’s no conclusive answer.
0:22:48 I think it’s Imagine,
0:22:50 but someone should correct me if that’s not the case.
0:22:54 So we released the Imagine three model
0:22:57 across a couple of services late December,
0:22:58 and then have been doing a bunch of work
0:23:00 in the last few months to bring that to developers.
0:23:03 So Imagine three should be available to developers
0:23:07 in the API and is the frontier Image Generation model
0:23:10 across like quality and a bunch of like human ranking
0:23:12 benchmarks, which as an aside comment,
0:23:15 it’s super interesting that if you look at text models,
0:23:18 I think one of the reasons the world has so much success
0:23:20 hill climbing on making models better
0:23:22 is like there’s a definitive source of truth
0:23:24 in some of the tasks that the models can perform.
0:23:26 I think with Image models, it’s like actually not the case.
0:23:28 It’s really hard to eval
0:23:30 and like you actually need humans in the loop
0:23:31 to do a lot of those evals,
0:23:34 or it’s like artistic and stylistic stuff
0:23:37 that’s like hard to put a finger on
0:23:38 which of these two things are better.
0:23:40 So a lot of those evals use human Raiders
0:23:41 and human benchmarks.
0:23:43 So there’s some degree of error,
0:23:46 but yeah, it’s been exciting to see like the models available
0:23:47 in the Gemini app.
0:23:50 It’s available to enterprise customers
0:23:51 and now it’ll be available to developers
0:23:52 to build with, which is awesome.
0:23:55 I think this like gen media future
0:23:56 is going to be super exciting
0:23:59 and VO hopefully sometime in the future as well.
0:24:01 – Yeah, yeah, VO is awesome.
0:24:02 I have access to it, early access to it.
0:24:04 It’s super fun to play with.
0:24:06 Is the best way to use the Imagine 3
0:24:07 inside of the image effects?
0:24:09 Is that still kind of like the easiest way
0:24:11 for a consumer to just go and play around with it?
0:24:12 – Exactly.
0:24:13 I think it’s also available for free
0:24:16 to folks in the Gemini app.
0:24:17 I think if you ask to generate an image,
0:24:19 it’ll just do it through the Gemini app,
0:24:22 but ImageFX gives you a little bit more controllability
0:24:23 and stuff like that.
0:24:24 So there’s a few more like features
0:24:26 that are built into ImageFX.
0:24:28 So that’s definitely a place that it’s publicly available.
0:24:29 – Super cool.
0:24:32 Yeah, I know you and Nathan sort of before we hit record,
0:24:34 we’re nerding out a little bit about, you know,
0:24:37 the whole like sort of text application concept.
0:24:38 He wished there was a way
0:24:40 that you could have Unity open on the screen
0:24:42 and then actually have an AI sort of like assist you
0:24:44 with like where to click and what to do next
0:24:46 while you’re building a game in Unity.
0:24:48 And I think both you and I in Unison went,
0:24:51 “You can do that in AI studio right now.”
0:24:52 So I thought it might be kind of cool
0:24:55 to pick up where we left off on that conversation
0:24:57 and talk about some of the cool stuff
0:24:59 that’s available inside of AI studio
0:25:02 that maybe a lot of people don’t even realize exists
0:25:04 and probably definitely don’t realize
0:25:06 you could use most of it for free still right now too.
0:25:07 – Yeah, cool.
0:25:08 So everything in AI studio is free,
0:25:10 which I don’t think people realize.
0:25:12 Like the entire product experience,
0:25:14 there is no paid version of it.
0:25:15 There’s a paid version of the API,
0:25:17 which hopefully developers can scale with
0:25:18 and do all that fun stuff.
0:25:22 But all of our latest models end up in AI studio for free,
0:25:24 including the experience that powers
0:25:27 this like real-time multimodal live experience,
0:25:29 which if folks haven’t played around with it,
0:25:33 aistudio.com/live lets you do things
0:25:36 like share your screen or show your camera
0:25:38 and ask all these different questions
0:25:39 and interact with the model.
0:25:40 There’s a bunch of different voices.
0:25:43 There’s a bunch of different modalities to choose from.
0:25:45 But back to the conversation of like,
0:25:46 what will agents look like?
0:25:48 What do we want out of agents?
0:25:49 One of the limitations for agents
0:25:51 is you have to build all this scaffolding
0:25:54 for the agent to be able to see the things that you do.
0:25:55 Like to see my email and my text
0:25:57 and my et cetera, et cetera,
0:25:59 my personal laptop and my work laptop
0:26:01 or my phone and my watch,
0:26:03 like incredible amount of work to make that happen.
0:26:05 Except if you have a camera,
0:26:07 all of a sudden all of it just works.
0:26:10 And like you can sort of make the determination
0:26:12 of being able to show like the information you want to show
0:26:15 and share the stuff that you want to share.
0:26:18 It’s definitely more of a showcase of what’s possible.
0:26:20 And that’s why we put it in the API
0:26:22 because like, we don’t have the answers ultimately.
0:26:24 Like developers should go and build these products.
0:26:25 But I do think Matt,
0:26:27 you mentioned the Astra experience earlier.
0:26:29 I think like the multimodal live API
0:26:32 sort of gets to what Astra does at the core,
0:26:36 which is like be able to be this co-presence with tools.
0:26:38 And again, through a simple API, which is really exciting.
0:26:40 I think the piece that the multimodal live API
0:26:42 doesn’t have that will build,
0:26:44 that I think the Astra experience did have
0:26:46 is this notion of memory,
0:26:48 which again is like critical for agents.
0:26:51 Like I don’t want agents to just forget that,
0:26:54 I prefer sitting in window seats instead of aisle seats
0:26:55 or whatever it is.
0:26:57 Like you want all that context to be retained
0:26:59 as agents are making decisions for you in the future.
0:27:02 And I think that’s gonna require this sort of memory layer
0:27:04 which we’re working on building, which is exciting.
0:27:05 – Yeah, yeah.
0:27:07 And I mean, project Astra even sort of remembered
0:27:10 between sessions too, that like I was in London, right?
0:27:12 So I was talking to it while in London,
0:27:16 asking for restaurants to go check out things like that,
0:27:19 close that session, started another session later on.
0:27:22 And it remembered that I was over in London,
0:27:24 it remembered all the previous conversations.
0:27:27 So it wasn’t just like memory in the terms that like,
0:27:29 you can plug into the custom instructions on open AI
0:27:32 and it’ll remember your name and stuff like that.
0:27:35 It was actually like remembering the past conversations
0:27:37 and bringing in that as additional context,
0:27:38 which I thought was really cool
0:27:40 ’cause that’s really helpful to just like remember
0:27:41 the past conversations you had.
0:27:43 – I think this is an infrastructure problem.
0:27:45 And I think we didn’t talk about this explicitly,
0:27:47 but like one of the other narratives
0:27:49 over the last like year and a half has been like,
0:27:51 not enough AI in production.
0:27:54 You know, it’s kind of this demo toy thing
0:27:55 and no one really uses it.
0:27:58 I think a lot of this is ’cause it’s just like taking a while
0:28:00 for companies to build the infrastructure
0:28:02 to actually put AI into production.
0:28:04 And I think memory is this example of,
0:28:05 there aren’t a bunch of companies
0:28:07 that are like building this memory as a service.
0:28:09 And if you are building this, like let’s talk,
0:28:11 I’d love to hear about it and hear about what you’re building,
0:28:14 but I think there’s a lot of opportunity still to be built
0:28:17 around that, around memory, like as a service for folks.
0:28:18 You could also start to think about like,
0:28:21 there’s so many interesting ways to explore this.
0:28:23 Like where does all your personal context
0:28:24 already live today?
0:28:26 Like how does that, whoever that provider is,
0:28:30 plug into the world of where all the other memory services
0:28:30 are going to be.
0:28:32 So I think there’s a lot of like really, really interesting
0:28:36 directions that need to be built for memory specifically.
0:28:37 – Logan, I’m curious.
0:28:39 So you were talking about earlier the models
0:28:40 that a lot of people don’t realize and all of it’s free,
0:28:41 like an AI studio.
0:28:44 Like why do you guys hide it in AI studio?
0:28:46 Recently, I talked to a bunch of different people
0:28:48 about DeepSeq and they were talking about how amazed
0:28:49 they were by it.
0:28:52 And I was like, yeah, but like you can get the same stuff
0:28:55 but better for free on AI studio right now.
0:28:57 And they didn’t know.
0:28:58 And it’s like, there’s a lot of people who don’t know.
0:29:00 And so I was like, you got to communicate that better
0:29:02 somehow or like, I think you guys should have like,
0:29:05 you know, its own website or something like outside of Google,
0:29:06 like a new product where you guys just like,
0:29:08 hey, here’s the new frontier
0:29:10 and here’s what we’re pushing and Google is still there
0:29:12 and it uses some of the tech, but we have a new thing.
0:29:14 And that’s that personal opinion, but you know.
0:29:15 – Yeah, yeah.
0:29:17 No, I think you’re spot on and for what it’s worth.
0:29:20 Like I think we get this feedback pretty consistently.
0:29:23 I think some of this is a factor of just like the state
0:29:27 of the world and the challenges that we have as a product.
0:29:30 Like I think in one hand, like we are a developer platform.
0:29:32 Like we’re not building the front door
0:29:33 for Google’s AI surfaces.
0:29:36 Like that’s not the product that I’m signed up to build.
0:29:37 That’s not the product that like we’re sort of
0:29:38 directionally building towards.
0:29:40 We’re really focused on like,
0:29:43 how do we enable builders to get the latest AI technology?
0:29:47 The Gemini app formerly barred is the sort of front door
0:29:49 to Google’s AI technology.
0:29:51 And I think from a consumer standpoint
0:29:52 and also from an enterprise standpoint
0:29:55 and like workspace and other places,
0:29:57 there’s all this interesting organizational work
0:29:59 that’s happened at Google over the last couple of years.
0:30:02 I think like one of the cool stories is like operationalizing
0:30:05 Google DeepMind from doing this sort of foundational research
0:30:07 to being an organization that builds
0:30:09 the world’s strongest generative AI models
0:30:11 and like actually delivers those to the world
0:30:12 sort of as a product.
0:30:15 And then now bringing the product surfaces
0:30:18 that are the front line of delivering those to the world,
0:30:21 the Gemini app Google AI studio into DeepMind
0:30:23 so that we can sort of continue to accelerate.
0:30:24 Like all of those things to me are like
0:30:26 directly the right stuff for us to do.
0:30:27 I agree with you.
0:30:28 I think we need to put the models
0:30:30 in front of the world as soon as possible.
0:30:33 And I think having a single place to do that makes sense.
0:30:34 And it should probably be the Gemini app
0:30:37 that probably shouldn’t be AI studio.
0:30:39 But at the same time I say that like
0:30:40 we also want to be a surface
0:30:42 to sort of showcase what’s possible.
0:30:44 So there’s a lot of like tension points
0:30:46 but I do think, I fundamentally do think
0:30:47 we’re going to get there.
0:30:49 The Gemini app is moving super quickly
0:30:50 to like get the latest models.
0:30:52 Like they just shipped 2.0 flash.
0:30:54 They shipped 2.0 Pro today.
0:30:55 They shipped the thinking model today.
0:30:57 So I think that Delta between the Gemini app
0:30:59 and AI studio is sort of going away.
0:31:01 Which yeah, I’m excited about
0:31:03 because like the consistent feedback
0:31:04 is people don’t like that Delta.
0:31:06 They want to have a single place to go to
0:31:07 to sort of see the future.
0:31:09 – I always saw the AI studio is sort of like
0:31:12 the playground to test what the APIs are capable of.
0:31:15 In the same way like open AI has their open AI playground
0:31:18 and you can kind of go and mess with some of the settings
0:31:20 and see what the output will look like
0:31:21 before using it in your own APIs.
0:31:24 That’s kind of how I always saw the AI studio.
0:31:26 Because like once you get into it
0:31:28 if you’re not very technically inclined
0:31:29 you might get a little overwhelmed
0:31:32 seeing things like what models should I be using?
0:31:34 What is the temperature?
0:31:35 You know, things like that.
0:31:36 People aren’t necessarily going to know
0:31:37 like how to play with that.
0:31:39 Like what should I set my token limit to?
0:31:40 Things like that.
0:31:42 I don’t really feel like general consumers
0:31:43 want to mess with.
0:31:45 They just want to go to a chat bot
0:31:46 and ask their question, right?
0:31:50 It feels very tailored towards developers to me.
0:31:52 And then the Gemini app feels like, all right
0:31:53 this is their front user interface
0:31:55 that they want the general public
0:31:57 to go be using at least.
0:31:58 – Yeah, I got a ping yesterday actually
0:32:01 from someone saying why is the chat experience
0:32:02 have all this stuff in it?
0:32:04 Like why are there all these settings and stuff?
0:32:06 And I was like, you’re in the wrong place.
0:32:08 Like Gemini app is the place that you need.
0:32:09 And then they responded right away on Twitter
0:32:12 and they’re like, yes, this is much better for me.
0:32:13 I don’t want to see all that complexity.
0:32:15 But I do think about this a lot
0:32:17 is like how do people actually show up?
0:32:19 How are they finding their way into these products?
0:32:22 I do think the Gemini app is like this very large front door.
0:32:24 So it tends to capture most of these folks.
0:32:27 Like it’s literally built into the Google app on iOS
0:32:28 and all this other stuff.
0:32:29 Versus like you actually kind of have to do
0:32:31 a little bit of searching to find AI studio
0:32:34 which probably makes sense in some cases.
0:32:35 Awesome.
0:32:36 – Logan, you said you were super excited
0:32:38 about text application.
0:32:39 You were talking about like Lovable, Bolt,
0:32:41 other companies like that.
0:32:42 Like what are you excited about in the space
0:32:44 and where do you think that kind of stuff is going?
0:32:48 – Yeah, I think just like being able to democratize access
0:32:51 to people building software and like creating things.
0:32:52 There’s a ton of people in my life
0:32:54 and I don’t live in the Bay Area.
0:32:56 So there’s a disproportionate amount of people
0:32:57 who aren’t in tech where I live.
0:33:00 But the proportion of like people with interesting ideas
0:33:01 I actually think is the same.
0:33:03 I think it’s just like the actual tools themselves
0:33:05 that they have to go and execute on those ideas
0:33:08 that I think is like much less distributed
0:33:10 in places outside the Bay Area, New York
0:33:11 and other places like that.
0:33:15 So I think this frontier of text to app creation
0:33:18 is gonna be so, so interesting to see play out.
0:33:19 And yeah, there’s a ton of companies
0:33:22 that are having like lots of like actual real early
0:33:24 commercial success and traction today.
0:33:28 Which I think, again, this is one of those examples
0:33:31 where like sometimes there’s use cases that don’t work
0:33:33 and then all of a sudden like the model quality
0:33:35 just gets good enough for you build the right
0:33:37 sort of couple of things from a product experience.
0:33:39 And then all of a sudden it clicks
0:33:40 and like now this thing is possible.
0:33:43 And to me it feels like text to app creation
0:33:46 like has had that moment and it’s now possible.
0:33:47 And I think it’ll take a while
0:33:50 and there’ll still be a bunch of other things to hill climb on.
0:33:52 But I think especially now with like reasoning models
0:33:54 and the ability for them to like keep thinking
0:33:57 and writing more code and doing all that work.
0:33:59 Like I think the complexity of the apps
0:34:01 is also going to continue to go up on this exponential.
0:34:03 So and actually replicate just how they’re launched.
0:34:05 I think today or yesterday of like this
0:34:07 a similar sort of product of text apps.
0:34:09 I think there’s more and more players showing up
0:34:10 in this space.
0:34:13 I would assume that like probably 50% of products
0:34:15 or something like that that are building with AI
0:34:17 have this type of experience.
0:34:19 And you could think about like, you know
0:34:20 how does that translate to someone
0:34:23 who’s doing something very, very domain specific?
0:34:25 I think there’s a lot of like companies
0:34:28 that try to build extension ecosystems or connectors
0:34:31 or like, you know, all these other like side cars
0:34:32 of their product.
0:34:34 You can imagine like you just let your users create those.
0:34:36 Like here’s the sort of generic set of APIs
0:34:39 that talk to, you know, your email client, for example.
0:34:42 And like here’s a text box and like go build
0:34:44 the sort of product experience you want.
0:34:45 Like it’s sort of in your hands.
0:34:48 And like that’s a crazy world that you could totally customize
0:34:49 it however you want to them.
0:34:50 Yeah, it doesn’t feel that far away.
0:34:52 My new email client that’s got like, you know
0:34:54 80s style video game stuff.
0:34:56 You know, it’s like a mixed end with the email client.
0:34:57 That’d be so cool.
0:34:58 You know, I’ve been loving that concept.
0:35:00 We’ve talked about this a couple of times on the show
0:35:03 of like, I’ve gotten in the habit now of like,
0:35:05 when I have like a little problem or a bottleneck
0:35:07 that I need to solve instead of going and like searching out
0:35:10 if there’s like a SaaS company that already exists
0:35:11 that has that product for me.
0:35:14 If it’s simple enough, I’ll just go prompt
0:35:15 that software into existence.
0:35:17 And I have like a little Python script
0:35:20 that runs on my computer to solve the problem for me, right?
0:35:22 Like I made a little script where I can input
0:35:24 my one minute short form videos into it.
0:35:26 It automatically transcribes them
0:35:27 and then cleans up the transcription
0:35:30 and adds like proper punctuation and stuff.
0:35:31 I created another mini app
0:35:33 where I can drag and drop any image file.
0:35:35 Doesn’t matter what type of file format it is.
0:35:37 It’ll convert it to a JPEG for me
0:35:39 so I can use it instead of like my video editing.
0:35:42 And these are probably softwares that exist.
0:35:43 I could go and hunt them down on the internet
0:35:46 and maybe pay five bucks a month to use them,
0:35:49 but I could just go use an AI tool,
0:35:50 prompt the tool that I need.
0:35:53 And 15 minutes later, I have something on my desktop
0:35:56 that I don’t need to go pay anybody else for anymore.
0:35:57 Maybe it connects to an API,
0:36:00 like my transcription one connects to the OpenAI whisper API.
0:36:03 So it is costing me like a penny every time I use it,
0:36:04 but so what?
0:36:06 I just love this concept of like
0:36:08 when I have a bottleneck in my business,
0:36:10 I can just go like prompt an app into existence
0:36:12 that solves that bottleneck.
0:36:16 – Yeah, I think that carried out one step farther
0:36:18 towards this like infinite app store
0:36:21 where like truly everyone is creating
0:36:24 and contributing to this thing and like remixing.
0:36:26 This is the stuff that gets me excited about the future
0:36:28 ’cause like there’s so much cool stuff to be created.
0:36:31 And really, I think that the lens of all of this is like,
0:36:33 how do you democratize access and make it
0:36:35 so that anyone can go and build this stuff?
0:36:36 And as someone who can program,
0:36:39 but also knows how painful it is in a lot of ways,
0:36:41 it’s just like so cool that more folks
0:36:43 are gonna be able to participate in that.
0:36:44 It’s gonna be awesome.
0:36:46 – Yeah, coincidentally, someone today
0:36:48 from my hometown of Alabama like messaged me like,
0:36:49 hey, I have this idea for an app.
0:36:50 I get this kind of stuff all the time.
0:36:52 I have an idea for an app and who can I hire
0:36:53 to build it and all that stuff?
0:36:55 And I’m like, I’m about to send him a link to like replit.
0:36:56 Have you like tried this yet?
0:36:58 You know, it’s like, just go try that.
0:37:01 And instead of paying someone $5,000,
0:37:03 that’s probably like a ton of money for him, right?
0:37:05 Instead, just go try replit and you know,
0:37:07 sign up for one month, cancel it if you don’t like that
0:37:10 to that and then just see what you can get.
0:37:11 And it’s gonna get better and better.
0:37:12 Like I’ve tried replit and all of them
0:37:13 and like they’re pretty good.
0:37:15 It feels like there’s something that’s like slightly missing,
0:37:17 but every time I check it, it’s better than the last time.
0:37:20 And it feels like probably within the next year or two,
0:37:22 you’re just gonna make any kind of software you want
0:37:23 just by talking.
0:37:25 It’s just, that’s gonna be such a magical moment.
0:37:26 Like in the early days of the internet,
0:37:27 the internet I feel like was more fun.
0:37:29 ‘Cause people, there’s all these like different websites
0:37:31 and different kinds of things or like you’d have Winamp
0:37:32 and you put a skin on your Winamp.
0:37:33 But there’s all these different things
0:37:35 in terms of customization that was happening more
0:37:36 than there is now on the internet.
0:37:38 And it feels like this kind of stuff might bring that back
0:37:39 where like, yeah, the internet,
0:37:42 you can kind of customize how you interact with the internet
0:37:44 through creating your own custom software with AI.
0:37:45 – Yeah, I was just thinking about
0:37:47 as you were describing like a fun internet,
0:37:49 I was thinking of my personal website,
0:37:51 which is like a blank HTML page.
0:37:53 And like there’s no styling or anything like that.
0:37:56 But like, if I didn’t have to shoulder the costs like,
0:37:58 and the LLM on someone’s computer,
0:37:59 I could just kind of like talk to and say,
0:38:01 like, you know, remix this site
0:38:02 and do it in all types of crazy ways.
0:38:06 Like that would be so fun of like every time someone shows up,
0:38:08 it’s a different product or it’s a different experience
0:38:09 to see this content.
0:38:11 And I think there’s a lot of interesting
0:38:12 threads to pull on that.
0:38:15 – Is there anything else happening at Google right now?
0:38:17 Any other things that you’re working on
0:38:19 that you’re allowed to talk about?
0:38:21 Is there any avenues we haven’t gone down
0:38:22 that you really wanna talk about
0:38:24 that you’re allowed to talk about, I guess?
0:38:26 – Yeah, I think the only other thread,
0:38:28 and we alluded to it a couple of times
0:38:29 is reasoning model stuff.
0:38:31 It feels like, and I tweeted this the other day,
0:38:33 it feels like the GBT2 era for these models.
0:38:36 Like there’s so much new capability
0:38:39 and so much progress being squeezed out of the models
0:38:40 in such a short time.
0:38:42 And we released our first reasoning model
0:38:42 back in December,
0:38:45 right after the Gemini 2.0 flash moment,
0:38:46 one month later,
0:38:49 like a normal like six months worth of progress,
0:38:51 honestly, on like a bunch of the benchmarks
0:38:53 that matter for this stuff.
0:38:54 We released an updated version
0:38:57 like January 21st, a couple of weeks ago.
0:38:58 And if you look at the chart,
0:39:02 it’s like literally linear progress up into the right
0:39:03 across a bunch of the end.
0:39:06 Like it’s just crazy to think that,
0:39:08 again, like a month ago, the narrative was like,
0:39:09 the models are hitting a wall,
0:39:11 there’s no more progress to be had.
0:39:14 And it’s funny like how much nuance
0:39:15 some of the conversation lacks
0:39:19 because these innovations are like deeply intertwined.
0:39:21 I was having a conversation earlier today
0:39:23 about like long context
0:39:25 and how long context is actually like
0:39:27 a fundamental enabler of the reasoning models
0:39:29 because like by themselves,
0:39:30 the long context innovation,
0:39:33 like the model’s okay at pulling out
0:39:34 certain pieces of information.
0:39:36 Like it can do needle in a haystack well,
0:39:39 it can find a couple of things and a million tokens,
0:39:41 but it’s really hard for the models to attend
0:39:43 to the context of like,
0:39:44 you know, find a hundred things
0:39:47 in this million token context window example.
0:39:49 Reasoning models are the unlock for this
0:39:50 because the reasoning models,
0:39:52 the model can really just like continue
0:39:53 to go through that process
0:39:55 and think through all the content
0:39:57 and like really do the due diligence.
0:40:00 And it’s almost uncanny how similar it is to like,
0:40:01 how would you go about this?
0:40:03 Like I couldn’t watch a two hour movie
0:40:06 and then if you quizzed me on a hundred random little things
0:40:08 as part of like, I’m not going to get those things, right?
0:40:10 Like it’s going to be really hard to do that.
0:40:11 But if you let me go through the movie
0:40:13 and you know, watching an eye movie
0:40:17 and like add little inserts and like clip things
0:40:17 and cut things and do all this.
0:40:19 So like I’d be able to find those things
0:40:21 if you asked me those questions again.
0:40:23 And it feels like that’s kind of what reasoning is doing
0:40:25 is actually being able to do that.
0:40:28 So I think we’re super early in this progress
0:40:29 and it’s going to be a lot of fun
0:40:32 to see both the progress continue for us.
0:40:35 But again, through this narrative of how all this innovation
0:40:38 trickles into the hands of people who are building stuff.
0:40:39 And like there’s going to be a ton of new products
0:40:42 that get built, like maybe text to app
0:40:44 just like it’s 10x better in the next year
0:40:45 because of reasoning models.
0:40:48 Like that’s possible, which is just crazy to think about.
0:40:49 – Yeah, yeah.
0:40:50 – But like the other models, they almost feel like they’re like
0:40:53 double checking triple checking themselves in real time.
0:40:56 It’ll be like sort of starting to give a response
0:40:58 and then be like, let me actually double check
0:40:58 what I just said.
0:41:02 And when it comes to coding, that seems like it’s the ideal
0:41:03 use case almost, right?
0:41:05 Cause it can almost look back at its code and be like,
0:41:07 oh, I think I made a mistake there.
0:41:09 And sort of continually fix its code
0:41:11 before it finally even gives you an output,
0:41:13 which I’ve just found to be really, really cool.
0:41:17 But also from what I understand, that’s where a lot of the cost
0:41:18 and the future is going to come in.
0:41:22 The cost of the inference to do all of this like analysis
0:41:25 in real time as it’s giving its output.
0:41:26 – Yeah, and the other thing to think about
0:41:29 which is interesting is we’re seeing all of this progress
0:41:33 with the reasoning models and they are doing
0:41:36 like the most naive version of thinking.
0:41:38 Like they really are like, if you were to think about
0:41:40 like the human example of this, like you’re sort of sitting
0:41:42 in a box like thinking to yourself.
0:41:44 Like you have no interaction with the outside world.
0:41:46 You’re not able to like test your hypothesis,
0:41:50 use a calculator, search the internet, any of those things.
0:41:52 And like you have to sort of form your thoughts
0:41:54 independent of the outside world.
0:41:55 And you imagine what starts to happen
0:41:57 when you give these things tools.
0:41:59 And like it really does feel like
0:42:01 that’s the agentic future that we’ve been promised
0:42:04 is like all of these tools in a sandbox interacting
0:42:06 with the model letting it sort of have that feedback loop
0:42:08 of trying things and seeing what doesn’t work.
0:42:10 So I couldn’t be more excited about that.
0:42:12 – Yeah, that’s really interesting to think about.
0:42:15 So like right now it’s just sort of thinking through things
0:42:17 and sort of double checking itself, but in the future
0:42:19 it could actually be working with other tools
0:42:21 that can also like assist in the double checking
0:42:25 and things like that and get even smarter in those ways.
0:42:26 – Yeah, and I think put a different way,
0:42:28 like to be more extreme, I think it has to do that.
0:42:30 Like I think the version of the future
0:42:31 that we’re going towards is like,
0:42:33 we’re not going to be able to see the progress
0:42:35 continue to scale unless the models can do that.
0:42:37 And again, this goes back to this thread of like,
0:42:39 there’s lots of hard problems to solve in the world,
0:42:41 like making it so the models can do that efficiently
0:42:45 and like securely and safely have that sort of sandbox
0:42:47 to do that type of thinking and work
0:42:48 is going to have to happen.
0:42:50 And it’s probably a lot of work that hasn’t been solved today,
0:42:53 which is interesting and opportunistic.
0:42:55 – Yeah, it’s crazy to think that like probably soon
0:42:57 like AI is going to be helping create all those tools as well.
0:42:59 So that’s when we’ll see things just go exponential.
0:43:00 – It already is.
0:43:02 – It’s like whether people with AI or AI itself
0:43:05 creating the tools and just move that back into the system.
0:43:06 And it’s going to be wild,
0:43:08 how fast things are going to get better.
0:43:10 – All the engineers being powered by cursor.
0:43:11 It’s crazy, like it’s happening today.
0:43:14 Like so many people are, I feel this way for myself,
0:43:15 like I write more software now than I did
0:43:17 when I was a software engineer
0:43:20 because I have AI tools
0:43:22 and I can do all this crazy stuff.
0:43:22 – Yeah.
0:43:24 – How far do you think we are from like AI
0:43:26 actually being able to update its own weights
0:43:28 based on conversation.
0:43:31 So it actually learns based on new input
0:43:33 that it gets through conversations that it has.
0:43:36 – I think in the small scale example sense,
0:43:39 you could probably already do this to a certain extent.
0:43:43 I think in like the like real frontier use cases,
0:43:44 probably far from that.
0:43:47 Some of the open AI operator stuff was talking about this
0:43:51 around like, you know, the need for having evals
0:43:53 of like basically like creating economic value,
0:43:57 like actually creating money and where we are in that.
0:43:58 And like you probably don’t want the models
0:44:00 to do things that have a high cost today
0:44:02 because if they get it wrong, it costs you a lot of money.
0:44:05 And training frontier models is definitely
0:44:07 on the list of things that would cost you a lot of money.
0:44:09 If you got that wrong, like you don’t want
0:44:12 a bunch of training that are just wasted compute.
0:44:14 Like that’s, you know, millions of dollars
0:44:15 of potential loss money.
0:44:17 So I think there’ll be a human in the driver’s seat
0:44:19 for those things for a while.
0:44:22 But I do think you can sort of accelerate this,
0:44:24 you know, small scale feedback loop.
0:44:26 And I think that’s why small models matter.
0:44:28 Like this like innovation that’s happening
0:44:31 of being able to compress the frontier capabilities
0:44:32 down into small models.
0:44:35 I think it enables that like rapid iteration loop
0:44:38 where maybe AI is more a co-pilot in that example.
0:44:39 – Gotcha.
0:44:42 Well, cool Logan, this has been absolutely amazing.
0:44:45 If people want to follow you, what’s the best platform
0:44:46 to pay attention to what you’re doing
0:44:49 and to keep up with what Google and DeepMind are up to?
0:44:51 – Yeah, yeah, I’m on Twitter, I’m on LinkedIn,
0:44:53 I’m on email.
0:44:57 So whichever one of those three is easiest to get ahold of me
0:45:00 would love to chat with folks about Gemini stuff or the like.
0:45:03 – Yeah, you’re pretty active over on X slash Twitter,
0:45:04 whatever you want to call it.
0:45:07 Whenever there’s a new like Google or DeepMind rollout,
0:45:08 you’re pretty much either tweeting about it
0:45:09 or retweeting about it.
0:45:11 So very, very good resource to keep up with
0:45:14 what’s going on in the world of AI with Google.
0:45:16 And Logan, thank you so much for hanging out again
0:45:17 with us today.
0:45:19 I’m sure we’ll have you back in the future if you want,
0:45:21 but this has been an absolutely fascinating conversation.
0:45:23 So thanks again for hanging out.
0:45:24 – Yeah, this is a ton of fun.
0:45:25 I’ll see you both at IO, I hope.
0:45:27 I think hopefully we’ll get the game back together
0:45:29 and we’ll spend time in person.
0:45:31 Hopefully at IO, it’s gonna be fun.
0:45:31 – We’d love to do it.
0:45:32 Thanks.
0:45:33 – Thank you.
0:45:36 (upbeat music)
0:45:38 (upbeat music)
0:45:41 (upbeat music)
0:45:43 (upbeat music)
0:45:46 you