a16z Podcast
a16z’s Erik Torenberg, Ben Horowitz, and Jennifer Li sit down with Vals founder and CEO Rayan Krishnan to discuss one of AI’s increasingly difficult problems: how do you actually measure whether a model is getting better?
As public benchmarks saturate and models get better at optimizing for the tests themselves, Rayan makes the case for independent, continuously evolving evaluations. They unpack why self-reported model scores can be misleading, how VALS evaluates models in the hours before a release, and why measuring increasingly agentic systems means testing work that can unfold over hours, days, or even weeks.
They also explore why evals are becoming critical for enterprises trying to understand the ROI of AI, what happens if token spend begins to rival employee salaries, and how evaluations could eventually provide a shared language for everything from model routing and recursive self-improvement to AI policy and international coordination.
Resources:
Follow Rayan Krishnan on X: https://x.com/RayanKrishnan
Follow Ben Horowitz on X:https://x.com/bhorowitz
Follow Jennifer Li on X: https://x.com/JenniferHli
Stay Updated:
Find a16z on YouTube: YouTube
Find a16z on X
Find a16z on LinkedIn
Listen to the a16z Show on Spotify
Listen to the a16z Show on Apple Podcasts
Follow our host: https://twitter.com/eriktorenberg
Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.
Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.
-
Autonomy Across Air, Land, and Sea
2024 has been a milestone year for autonomous tech. Waymo’s fully autonomous driver has surpassed 20 million miles, while FAA approvals now allow commercial drones to fly without visual observers, advancing air autonomy in unprecedented…
-
State of Crypto 2024: Builder Energy, U.S. Election, Stablecoins, AI, More
We take you behind the scenes of our newly released, annual State of Crypto Report — a16z crypto’s analysis of the latest data and trends that have defined the industry in 2024. This year’s report…
-
Tesla’s Road Ahead: The Bitter Lesson in Robotics
What does Rich Sutton’s “Bitter Lesson” reveal about the decisions Tesla is making in its pursuit of autonomy? In this episode, we dive into Tesla’s recent “We, Robot” event, where they unveiled bold plans for…
-
The story of Apple Pay with Jennifer Bailey
In 2024, marking 10 years since its launch, Apple Pay now boasts hundreds of millions of consumers in 78 markets, at checkout on millions of websites and apps, in tens of millions of stores worldwide,…
-
Lisa Su on the AI Ecosystem Behind AMD’s 50x Growth
Lisa Su has transformed AMD into a global leader in AI and high-performance computing. In this episode of the AI Revolution (AIR) series , Bob Swan, a16z Operating Partner and former CEO of Intel, sits…
-
The Deepfake Dilemma: The Technology, Policy, and Economy
Deepfakes—AI-generated fake videos and voices—have become a widespread concern across politics, social media, and more. As they become easier to create, the threat grows. But so do the tools to detect them. In this episode,…
-
A Big Week in Tech: NotebookLM, OpenAI’s Speech API, & Custom Audio
Last week was another big week in technology. Google’s NotebookLM introduced its Audio Overview feature, enabling users to create customizable podcasts in over 35 languages. OpenAI followed with their real-time speech-to-speech API, making voice integration…
-
From Swipe to Scale: How Tinder Became #1
In 1995, just 2% of couples met online. Today, that number has surged to over 50%, making online dating the top way couples connect. In this episode, a16z General Partner Andrew Chen chats with Tinder…
-
Human Data is Key to AI: Alex Wang from Scale AI
What if the key to unlocking AI’s full potential lies not just in algorithms or compute, but in data? In this episode, a16z General Partner David George sits down with Alex Wang, founder and CEO…
-
The Frontier of Spatial Intelligence with Fei-Fei Li
Fei-Fei Li and Justin Johnson are pioneers in AI. While the world has only recently witnessed a surge in consumer AI, our guests have long been laying the groundwork for innovations that are transforming industries…