AI transcript
0:00:11 [MUSIC]
0:00:14 >> Hello, and welcome to the NVIDIA AI podcast.
0:00:16 I’m your host, Noah Kravitz.
0:00:20 What’s in NASA’s Mars rovers have in common with the quest to cure cancer?
0:00:21 A lot, as it turns out.
0:00:26 Some of the same technology used in the Mars rover missions is now being used to
0:00:29 help detect, predict, and treat cancerous cells.
0:00:31 Which technology, and how does it work?
0:00:34 Here to talk about it is Thomas Fuchs.
0:00:39 Thomas is the Dean of Artificial Intelligence and Human Health at Mount Sinai in New York City.
0:00:43 And he’s also the co-founder and Chief Scientific Officer at PAGE,
0:00:48 the first company with an FDA-approved AI tool for cancer diagnosis.
0:00:53 Thomas, welcome to the NVIDIA AI podcast, and thanks so much for joining us.
0:00:54 >> Thank you so much for having me, Noah.
0:00:57 Very much looking forward to the conversation.
0:00:59 Likewise, there’s a lot to get into.
0:01:00 Let’s start with PAGE.
0:01:03 What is the company all about, and how did it get started?
0:01:08 >> So PAGE is the leading AI company in pathology.
0:01:10 It was founded 2017.
0:01:14 So we spun it out of Memorial Slang Catering.
0:01:19 And at its core, it does AI in cancer research,
0:01:24 and especially clinical care, and there within pathology.
0:01:27 So we look at these digitized pathology slides.
0:01:33 We find cancer, classify cancer, predict response to treatment, predict outcome,
0:01:37 help pathologists to do their job not just faster and better,
0:01:39 but especially help them to do things they can’t do yet,
0:01:44 and then help oncologists to find better treatments for their patients.
0:01:46 PAGE is actually an acronym.
0:01:50 It stands for Pathology, Artificial Intelligence Guidance Engine.
0:01:51 And that’s what we do.
0:01:57 And these days, PAGE is used across the globe on four continents,
0:02:03 and with thousands of patients treated last year already based on diagnosis
0:02:06 that were rendered with the help of PAGE.
0:02:09 >> So applications of machine learning, deep learning, AI,
0:02:13 to use the umbrella term in medicine and health and wellness
0:02:16 have really been exploding over the past few years in particular.
0:02:23 And for my money are one of the most interesting and useful applications of the technology.
0:02:27 We’ve had some different folks, some folks who are working to try to battle cancer
0:02:33 and some other folks doing other things with machine learning on the podcast before.
0:02:36 Let’s dive into how PAGE works a little bit.
0:02:40 My understanding is there’s sort of a combination of the PAGE tools
0:02:46 and then also third-party applications that are used throughout the diagnosis process.
0:02:48 Maybe you can take us in a little bit deeper.
0:02:50 >> Yeah, of course.
0:02:52 I think it’s always good to start with the patient.
0:02:57 So suppose you have some pain in the chest or somewhere else.
0:03:02 First you get the radiology, and then if the radiologist sees something,
0:03:06 the first thing that’s usually done is to take a biopsy.
0:03:10 So a needle biopsy at the location.
0:03:14 And then what comes out of there is some tissue.
0:03:19 And in the last 150 years in which pathology didn’t change much,
0:03:23 the pathologist looked at the tissue through a microscope
0:03:26 and then decided the diagnosis.
0:03:29 So it’s very subjective, of course, in many ways.
0:03:33 So what PAGE does is really help to digitize the whole process
0:03:37 and then build AI on top of it from its beginning.
0:03:41 The whole thing is part of the field of computational pathology.
0:03:44 We coined the term over 15 years ago,
0:03:49 and since then the field exploded, but of course back then it was very niche.
0:03:54 So after you have the biopsy, the tissue is then put on microscope slides,
0:03:58 and these are digitized and end up being enormously large images.
0:04:03 So 100,000 pixels times 100,000 pixels with millions and millions of cells.
0:04:08 So you could fit all your holiday snaps on one of these single slides.
0:04:11 And large institutions produce a lot of them.
0:04:14 So at Monsignor we produce over a million of these slides per year.
0:04:18 Then pathologists really have to look for the needle in the haystack.
0:04:21 If you have, for example, a mastectomy is a breast cancer,
0:04:23 you could end up with hundreds of these slides
0:04:27 and you’re really looking for a few cells or a larger lesion
0:04:29 that actually is cancer or not cancer.
0:04:32 And that’s a very long and cumbersome process.
0:04:35 And that’s exactly where our AI comes in.
0:04:39 So we trained AI at scale to actually find cancer,
0:04:45 and that also led to the first and only FDA approval in pathology, you mentioned.
0:04:50 So to do that, you trained these very large computer vision models first.
0:04:53 These, of course, transformer models these days.
0:04:57 And to do so, we digitized enormous amount of slides over the years,
0:05:02 linked it to all kinds of clinical data and pathology report data,
0:05:07 to train models directly from the image against these reports.
0:05:13 To do so in 2017, we actually built a dedicated compute cluster with NVIDIA DGX.
0:05:18 So the first ones that came out back in the day, which was, of course, great.
0:05:25 And that allowed us to build a model based on 60,000 of these slides for the FDA approval.
0:05:26 And let’s use clinically.
0:05:29 But these days, of course, that’s not enough.
0:05:35 And we build now a very large foundation models from millions of these slides.
0:05:39 And that’s done in partnership with Microsoft and Microsoft Research.
0:05:44 So we can actually use thousands of GPUs to build these very large computer vision models.
0:05:48 Since you brought it up, let’s talk about that foundation model.
0:05:54 My understanding is it’s one of or perhaps the largest image-based foundation model in the world.
0:05:56 So it’s by far the largest in pathology.
0:06:01 And what we are training now, based on four million slides,
0:06:05 is for sure one of the largest publicly announced computer vision models.
0:06:11 What you have to do is actually have to separate these very large slides into images of normal size,
0:06:14 like you would have an image net or other databases.
0:06:19 And in that space, we have 50 billion images.
0:06:24 So again, image net is only 14 million, so it’s 50 billion images.
0:06:27 I take a lot of photos as a dad, but to your point earlier,
0:06:30 I think that’s plenty of room for my holiday snaps.
0:06:35 Well, it depends how many kits you have, you know.
0:06:37 But that’s remarkable, 50 billion images.
0:06:43 Yeah, so if you count actually the pixels, it’s 10 times more data than all of Netflix.
0:06:46 So think of all the shows you watch, all the movies.
0:06:52 So if you count the pixels throughout, that’s just 10% of the image data we pipe through these models.
0:06:56 And the whole point of that is you end up with a foundation model
0:07:02 that truly understands microscopic morphology of tissue very well,
0:07:04 not only of cancer, but also of normal tissue.
0:07:09 And then like in language models, you get these embeddings of these images,
0:07:11 and you can use them for all kinds of tasks.
0:07:15 Again, cancer detection, classification, segmentation.
0:07:18 We also predict molecular changes, for example.
0:07:23 If you have a specific mutation that can be targeted by some therapy,
0:07:28 or sometimes you predict outcome to a therapy or to a treatment regime.
0:07:31 This is making me think of a few episodes we’ve done in the past.
0:07:39 One was with, to bring up NASA, some astronomers who were taking, looking at images, or using AI.
0:07:43 I shouldn’t say they were looking, they were processing images taken.
0:07:48 I think this was with the James Webb, one of the very powerful telescopes, right?
0:07:54 And another one more recently talking about cardiac care and early detection of cardiac disease,
0:07:58 using AI to power the techniques.
0:08:06 And something that stuck with me was this notion that not only can these models
0:08:10 churn through data with the speed and scale that humans just couldn’t do
0:08:13 and detect things that humans couldn’t with the naked eye.
0:08:18 But that in some cases, and I think this is specific to the cardiac care episode now that I think about it,
0:08:23 they were actually able to detect things that, and forgive my lack of medical knowledge here,
0:08:30 that were sort of in areas and types of things that researchers and medical professionals
0:08:36 hadn’t even thought to look for before, because they were almost sort of hidden in the cells,
0:08:40 in sort of the walls of the heart, or what have you.
0:08:45 Is that the kind of thing that you’re finding with your research, and can you speak to that a little bit?
0:08:49 Yes, so you’re pointing actually at the very, very interesting development.
0:08:54 So some of the models we built, especially also the clinical ones that are FDA approved,
0:08:57 that more or less mimic what the human does.
0:09:03 So the pathologist does visual pattern recognition to, for example, do Gleason grading in prostate,
0:09:06 or sub-classify breast cancer.
0:09:12 And that can be replicated based on the training data, or for example, the pathology reports.
0:09:15 But it gets really interesting when you go beyond that.
0:09:18 I mentioned that example for mutations.
0:09:23 So we have also sequence data of 100,000 patients where we have the slides,
0:09:26 and then the somatic sequencing of the tumor.
0:09:31 And for some of the mutations, you can actually predict them based on the image.
0:09:34 So the mutations lead to a change of pattern in the image.
0:09:38 And most of these are not recognizable by humans.
0:09:42 So we were not trained to do that, and pathologists are not trained to do that.
0:09:46 Some of them are very obvious, also humans can see, but others not.
0:09:50 And there, of course, it also gets more into basic science,
0:09:53 because then it becomes very interesting in reverse.
0:09:55 Can we find out what the model looked like?
0:10:01 So was it, for example, as you said, some subcellulum of our phyrophology at the nucleus,
0:10:06 or was it larger vessel structures or a combination of different kinds of cells?
0:10:10 For example, for the immuno-oncology work we do,
0:10:14 you want to detect these tumor-infiltrating lymphocytes.
0:10:17 Is there inflammation close to the cancer or within the cancer?
0:10:23 And then you have very high-level or complicated or nonlinear interactions
0:10:26 that are actually quite difficult for us humans to understand.
0:10:32 So model introspection becomes something very important to know what’s going on.
0:10:36 But philosophically, if you actually spin it a little bit further into the future,
0:10:43 the time will come when these AI models might be able to perform very, very well for one of these tasks.
0:10:47 For example, predict if the drug will work for you as a patient or not.
0:10:55 And we as humans might not be able to understand the intrinsic combinations these models look at.
0:11:02 And then the question is, how can you then still assure that these AI’s are safe and effective and equitable?
0:11:07 And that’s where all the whole FDA framework and all the efforts into testing these models comes in.
0:11:14 And I do think that I hope actually we’re going to get to that place because I want to go beyond the state of the art.
0:11:21 And there’s so much more, of course, in biology and physics, which are underlying these processes that produce that tissue
0:11:27 than we humans can grasp today, but that can really, really help patients to get better outcomes.
0:11:31 Full disclosure, I come from a liberal arts background, not a science background per se.
0:11:40 But all of the talk about all the scientific progress that these technologies are helping further and talk of new discoveries
0:11:45 is what really gets me excited and interested in where this stuff is headed.
0:11:49 I want to come back to the technical aspects of this in a moment.
0:11:54 But I want to ask you, where is page at now in terms of results?
0:11:56 What are you able to do?
0:12:00 I don’t know if success rate is the proper way to phrase it.
0:12:05 But what kinds of actual on the ground results are you seeing in using the tech?
0:12:10 So I think a good place to study is actually the FDA study we ran.
0:12:16 And there we could show that the error rate was reduced by 70%.
0:12:22 So pathologists find much more cancer than they would have before without increase of false positives.
0:12:24 So that’s very important.
0:12:33 And we also showed across the globe on 14,000 patients out of 800 different institutions out of 45 countries
0:12:37 that the AI really works everywhere across the globe.
0:12:42 That’s because at that scale that we train, it really focuses on the morphology.
0:12:49 It’s really on the growth patterns that it’s not thrown off by all kinds of nuisance variables or bubbles or air
0:12:52 or all kinds of things you see on these slides.
0:13:00 And that has been replicated at DA, at Cornell, in Brazil, in Europe, everywhere across the world.
0:13:01 That’s great.
0:13:05 So what’s of course interesting here is that if you think about FDA approvals of AI,
0:13:09 you have hundreds of AI’s approved in radiology.
0:13:11 I think they’re up to 500 by now.
0:13:14 And in pathology, you still have only one.
0:13:18 There’s just one that is actually safe and effective.
0:13:20 And that’s of course a huge difference.
0:13:25 And one reason is that the FDA looks differently at the two domains,
0:13:28 radiology to a large degree is a screening step.
0:13:33 You can always go back to another x-ray or another mammography.
0:13:37 But in pathology, it analyzed the diagnosis.
0:13:40 And you don’t have cancer until the pathologist says so.
0:13:44 So it’s much more dramatic in the treatment pathway.
0:13:51 But it also shows that there’s that huge chest in between all that height we have in AI,
0:13:57 15,000 biotech AI companies and all that fluff and everything.
0:14:00 And then the reality in the ground.
0:14:04 So if you have physician in the trenches and you want to use something that’s safe and effective,
0:14:05 you have the choice of one.
0:14:07 And that’s of course something we have to change.
0:14:12 So PAGE does that drastically and actually expanding to all kinds of different cancer types.
0:14:15 So we have, for example, a breakthrough designation for the breast system.
0:14:18 We can do bladder lymph nodes and so forth.
0:14:23 And then we use these very large foundation models we discussed, especially the new one.
0:14:28 It’s actually named after the founder of pathology, Rudolf Wirhoff.
0:14:33 So it’s the Wirhoff one model, the V1 model, and V2 will follow soon.
0:14:38 And we can actually use it for pan-cancer applications.
0:14:42 So instead of just doing what we just described for prostate cancer breast,
0:14:45 you can address nearly all cancers.
0:14:49 And 50% of cancers are rare cancers.
0:14:52 So if you inflict cancer, it’s a coin toss.
0:14:55 If you have a big one where there are a lot of treatment options,
0:14:58 or you have one of these rare ones where there’s nothing available.
0:15:04 And the foundation models allow us because they understand morphology in the body of tissue
0:15:09 so well that you can hit the ground running with few-shot learning on rarer cancers
0:15:14 and rarer conditions, and then also produce AI for these.
0:15:16 And that’s very, very promising.
0:15:17 Absolutely.
0:15:20 You sort of touched on this mentioning the few-shot learning,
0:15:23 but are there specific types of cancers,
0:15:27 and please replace types of cancers with a more precise term,
0:15:33 but are there specific variants that are more difficult to detect
0:15:35 because of technical problems?
0:15:37 And how are you trying to get around some of those?
0:15:38 Yes.
0:15:41 So there are usually two reasons.
0:15:43 First off, they might be rare.
0:15:51 So you have the big ones like breast, prostate, lung, derm, and so forth.
0:15:57 And then rarer ones like cholangiocasinoma, so that’s phyldoc, which is terrible.
0:16:02 And there usually it didn’t have the numbers to actually build robust systems that channel us.
0:16:05 And that’s what the foundation model is changing.
0:16:10 And then you also have within the large cancers, you have subtypes
0:16:13 that can be very rare and difficult to diagnose.
0:16:18 We just published results in breast cancer that the foundation model now
0:16:22 can actually really find these super rare forms as well,
0:16:26 which you wouldn’t do with a normal breast AI.
0:16:31 And that actually shows that it’s worthwhile building these very large systems,
0:16:34 although of course they are tremendously expensive.
0:16:38 Just the production of the data that go into these systems is very cumbersome.
0:16:43 But it really leads to the benefits for the patients at the end of the day.
0:16:44 Amazing.
0:16:46 And a question that popped into my mind earlier.
0:16:52 Are your system, your techniques sort of backwards compatible with older,
0:16:57 lower resolution, or not necessarily older, but just kind of lower resolution images?
0:17:02 Or is it all built atop kind of a, I don’t know, proprietary,
0:17:08 but sort of current very high res way of scanning for cancer cells?
0:17:10 So that’s a good question.
0:17:14 So most scanners that are out there, the high throughput scanners,
0:17:20 either have a 200 times magnification, 20x, or 440x.
0:17:23 And we can work with both.
0:17:24 Okay.
0:17:28 On the cancer detection side, so we could show that it’s equivalent.
0:17:33 Sometimes if you have very specific tasks that are really tailored towards,
0:17:37 I don’t know, the membrane of the nucleus, then of course you want to have higher resolution.
0:17:39 So it depends on the task.
0:17:41 I’m speaking with Thomas Fuchs.
0:17:46 Thomas is the Dean of Artificial Intelligence and Human Health at Mount Sinai Hospital in New York City.
0:17:50 And he’s also the co-founder and Chief Scientific Officer at PAGE,
0:17:54 as we’ve been discussing, PAGE is an AI company focused on pathology.
0:18:00 And they are the first company with an FDA approved AI tool for cancer diagnosis.
0:18:06 Thomas, I want to shift gears a little bit, kind of jog your memory back to something we tease in the opener.
0:18:10 You helped out with some of the technologies in the Mars rover.
0:18:13 I think was it perseverance in particular?
0:18:14 Yeah.
0:18:20 Yeah. So the techniques are used today actually for per se and curiosity before.
0:18:24 The images we worked with were on the previous rovers.
0:18:25 Okay.
0:18:31 And it was the Athena class, so spirit and opportunity, and then also from orbit.
0:18:36 So I had the enormous privilege to work for JPL for NASA a few years.
0:18:38 Did JPL, the Jet Propulsion Lab?
0:18:39 Yes, yes.
0:18:44 Athena, the Jet Propulsion Lab, who is of course handling the Mars rovers.
0:18:55 And what we did is we, for example, used the imagery from the navigation cameras to differentiate sand from gravel and from other terrain types so they don’t get stuck.
0:18:56 Right?
0:18:58 So spirit got stuck in this powder.
0:18:59 Oh, right, right.
0:19:00 Yeah, yeah.
0:19:04 And you want to, for example, omit that and it didn’t hit obstacles and so forth.
0:19:10 And we also had imagery from the Mars Reconnaissance Orbiter, so which is the satellite around Mars.
0:19:18 And the images it takes of the ground actually have quite good resolutions, so 30 centimeters per pixel in the best case.
0:19:19 Okay.
0:19:23 And they are nearly exactly the same size as these histology slides.
0:19:25 So they’re really enormously large.
0:19:33 And you also try to classify different terrain type to, for example, look for good landing spots for the navigation and so forth.
0:19:36 And yeah, the beauty of machine learning and the eyes.
0:19:37 Right.
0:19:40 Use it from the microscopic to the microscopic level.
0:19:50 And so with the rovers, how does it work sort of training and updating the model and then running inference from that distance?
0:19:58 Were these, you know, the systems all on the rovers and you would push an update out or I’m getting a little over my head here.
0:20:04 But how did that work kind of, you know, doing all this work to help real time navigation but from so far away?
0:20:05 Right.
0:20:16 So most of the research we did back in the time was, of course, more basic research show that capability or again do it from orbit for all kinds of purposes like looking for landing sites.
0:20:24 One of the big problems, of course, in space exploration is that as soon as you send something out there, it has to be radiation hardened.
0:20:30 And that means the machines are usually 10, 10 years behind what you actually have.
0:20:31 Right.
0:20:33 There’s no GPU out there.
0:20:48 I worked on another project where we did actually computer vision for flyby missions at asteroids or for example, Pluto, where the light gap is so big that you large that you can’t send meaningful comments out there.
0:20:52 And then it really automatically has to, for example, find the asteroid to take pictures.
0:21:00 And there also JPL had worked on building specific FPGAs and ASICs that could do that in the future.
0:21:05 But that’s certainly one limitation that you can’t use the…
0:21:06 Right.
0:21:08 And maybe that’s something for NVIDIA, right?
0:21:11 Have radiation hardened GPUs for that specific…
0:21:12 No, you read my mind.
0:21:17 I was thinking about the, you know, special edition 40 series radiation hardened.
0:21:30 So in addition to the work you’re doing now with PAGE and Mount Sinai and the JPL stuff, you’ve had a hand in at least a couple of other really interesting and sort of large profile, high profile projects.
0:21:34 The Memorial Sloan Kettering AI Research Department.
0:21:38 And then also there’s a supercomputer in New Jersey, I believe.
0:21:39 Yes.
0:21:40 Tell us a little bit about those.
0:21:41 Yeah, sure.
0:21:43 So that was the one I was referring to.
0:21:46 We built in 2017, 18 specifically for PAGE.
0:21:56 So it’s still the largest standalone supercomputer only for pathology, and we extended it over the years, but it’s of course not big enough to train these humongous foundation models.
0:21:57 Okay.
0:22:10 That’s why we needed a relationship with Microsoft, not only for compute, but because of course they’re fabulous engineers and scientists and it’s really a great partnership to drive that forward on modern hardware to build that.
0:22:25 Because of course we are always, always limited with compute, right, be it at PAGE, but also at Mount Sinai, of course, there’s so much fantastic research that could be drastically accelerated if the research community had access to more compute.
0:22:40 But it was, of course, a very interesting exercise. I mean, we wrote papers and were published in Nature Medicine and Well-Cited and so forth. But the secret behind all of that is of course engineering, right? It’s MLOps.
0:22:53 How do you actually build a stack? So usually in that case, the problem are not even the GPUs, but it’s I/O. How can you pipe adabytes of data quickly enough to the GPUs to actually train these very large models.
0:23:07 And so there’s an enormous software stack to build computer vision models at scale and enormous experience that PAGE has in doing these applications that are, of course, broadly applicable in all kinds of computer vision tasks.
0:23:12 Or now, of course, we do multi-modal approaches with HACS and radiology and so forth.
0:23:14 Your own background is originally in engineering?
0:23:22 Yeah, so I had my undergrad was in mathematics in Austria, which you can hear from Arnold Schwarzenegger accent.
0:23:23 Sure.
0:23:27 After that, I did a PhD in machine learning at the time where it wasn’t cool yet.
0:23:28 Yeah, yeah.
0:23:37 Europe’s was 400 people and it was very teach and combining, of course, AI with pathology was even more niche back then.
0:23:40 And I was going to ask, how did you find your way into working in pathology?
0:23:54 Yeah, so that was at the beginning of my PhD one day pathologist of the hospital we worked with came into the lab and just wanted some high level statistics based on the excel sheets, but they showed us a few images.
0:23:55 Right.
0:24:01 And as a naive student, I saw these and said, oh, these are just roundish object cells should be easy to detect.
0:24:04 And now we are 20 years later and we still can’t do it.
0:24:10 But at least I stuck with it and we can do it quite well.
0:24:11 Amazing.
0:24:12 These origin stories are the best.
0:24:14 So many of them are so similar to that.
0:24:18 You know, the student working on this, somebody wandered into the lab.
0:24:21 I was the only person there 20 years later.
0:24:26 I’m working with Microsoft to build the world’s largest foundational image model to fight cancer.
0:24:27 Amazing.
0:24:31 A lot of times we like to end these shows and kind of a forward looking note.
0:24:33 I’m going to give you a two part thing here.
0:24:45 The second part, I want to kind of open it up given your background and, you know, your hands on engineering prowess as well as obviously all the things you’re doing in the real world to ask you about AI more broadly.
0:24:53 But before that, Paige, your work at Mount Sinai, you talked about this a little bit, but to kind of put a point on it.
0:25:03 Where do you see the future of AI and medicine and healthcare and we can narrow it down specifically to, you know, using AI and pathology?
0:25:04 Where do you see things headed?
0:25:15 What are kind of the big obstacles beyond, of course, everybody needs more compute, you know, kind of the big hurdles that you’re looking to get over near term and take it out as far as you want as far as what you see in the future.
0:25:26 Yeah, of course. Let’s start with pathology. So pathology has a very unique challenge and that that for the last 150 years, it really didn’t change.
0:25:30 So first has to go digital before it’s completely AI based.
0:25:39 So part of what AI does, sorry, what Paige does, it also helps large labs and academic institutions to actually go digital.
0:25:46 So get these canners, link it to your LIS, make sure everything flows and then it’s a SaaS service in the clouds.
0:25:51 You can access it from everywhere and you can diagnose from everywhere and so forth.
0:25:55 So that’s something that has to be driven in pathology.
0:26:03 Another part is, of course, it’s important that the FDA also as they want to do now take oversight.
0:26:15 So there are some loopholes, some other companies who don’t validate at that level of scrutiny I mentioned can use and these LUT loopholes should be closed.
0:26:21 So in healthcare much broader and that’s what brings us to Mount Sinai, which is of course a very unique place.
0:26:27 It’s the largest health system in New York City with eight hospitals and 200 locations and so forth.
0:26:32 There we are trying to more or less replicate what we did in pathology for all these different areas.
0:26:46 So really level up in cardiology, in radiology, in psychiatry and AI now plays a role in all these areas from basic research in drug discovery and drug development to the bad side.
0:27:00 So Mount Sinai has more than 20 models that are actually used in the hospital to direct the care teams to the right patients who had risk for seizure or malnutrition or for other things.
0:27:21 And then on the basic research side there’s work in cardiology based on EKGs and ECGs to actually early predict all kinds of cardiology issues or an oftalmology where we look at fundus images to try to even predict new degenerative diseases from that.
0:27:31 Mount Sinai is also building a neuropathology AI center where 10,000 brains are gonna be scanned and that’s of course for work in Parkinson’s and in Alzheimer’s.
0:27:32 Amazing.
0:27:41 And so in all these areas AI is a key to provide capabilities we don’t have yet, mid-care or mid-research.
0:27:48 Besides all these doomsday scares that AI is an existential threat for us and so forth.
0:28:03 I think especially in healthcare we have the moral obligation to develop AI as fast and as safe as we can because patients are dying not because of AI but because of the lack of AI.
0:28:13 In the US alone 800 patients die or are permanently disabled because of diagnostic errors.
0:28:21 And as we just discussed for example in pathology as we proved out these are things that can be addressed to a large degree with AI.
0:28:30 So we have to build the tools for the physicians to give better treatment and a slowdown there is certainly not responsible.
0:28:35 Now I’m gonna ask the old school engineering PhD and you.
0:28:47 Looking at the current looking ahead a little bit when it comes to the field and the technical everything going on technically and it’s been a what’s the word it’s been a whiz bag.
0:28:59 There’s a better word but it’s been a whiz bag couple of years in particular, at least as far as you know public exposure to everything going on but I’ve had the chance to talk to a bunch of engineering leaders who’ve just been saying
0:29:05 I’ve never seen this kind of rate of acceleration and in tech you know in my career anyway.
0:29:10 What are you excited about what kinds of you know things on the technical ground level.
0:29:15 Are you looking forward to you know in the next I don’t know a year or two years five years.
0:29:34 Very mundane side is just to get these tools into the clip right the first thing huge hurdle and usually it’s it’s not capabilities it’s not regulation usually it’s reimbursement very very petty monetary issues that where health systems are hard to change.
0:29:41 So on the AI side, of course a lot of the current excitement is driven by language models.
0:29:47 And that’s only for some reason, of course for good reasons we humans over index language.
0:29:54 Yes, everybody who can who can string five sentences together in a coherent way seems to be intelligent.
0:30:03 And that’s why we certainly assign a lot of of capability or properties to large language models that might or might not be there.
0:30:17 But at the end of the day language is of course a human technology it’s produced by our brain when so our brain our relatively feeble wet brain with it with its few neurons can produce language and reason about language.
0:30:24 But if you go beyond that, for example in biology and cancer research and if you look at the issues there.
0:30:30 Just at a single cell all these processes are produced by physics and by biology.
0:30:41 Just if you think of all the proteins that are at play at a single moment in a single cell that is in complexity so far beyond language.
0:30:49 That there’s a whole universe out there, literally our physical universe and then the biological one that goes far beyond language.
0:30:59 You see it even now in pathology the large models we built we touched upon that we humans are missing the vocabulary to even describe it in language.
0:31:07 And then usually we come up with all kinds of comparisons where these cell look like people in single file or these cells look like this.
0:31:10 But there’s so much more going on.
0:31:18 We start to capture with their eye be image based be genomic based and so forth that goes beyond our capabilities with language.
0:31:27 And I think that space is going to be dramatically exciting because that will deliver new drugs better care better outcome.
0:31:31 It’s an exciting time to be alive to see transformation.
0:31:32 Amazing.
0:31:34 Thomas for listeners.
0:31:39 What while they ponder all of this who want to learn more about what page is doing.
0:31:42 Maybe what’s going on in Mount Sinai and the other things we touched upon.
0:31:47 Are there some resources on the web where you you would direct folks to go.
0:31:48 Of course for page.
0:31:53 So just go to page dot A.I. B A I G E dot A.I.
0:32:02 And at Mount Sinai if you look for Mount Sinai you see our initiatives there and the I centers and if you want to go beyond pathology that’s of course the place to go.
0:32:03 Excellent.
0:32:05 Thomas this has been an absolute pleasure.
0:32:06 Thanks so much for making the time.
0:32:10 There’s a lot to chew on here and a lot to be optimistic about.
0:32:15 So we appreciate all the work you and your teams are doing and for taking a few minutes to tell us about it.
0:32:16 Thank you so much Noah.
0:32:17 It was really a pleasure.
0:32:19 Thank you for the great questions.
0:32:21 [Music]
0:32:23 [Music]
0:32:25 [Music]
0:32:27 [Music]
0:32:29 [Music]
0:32:31 [Music]
0:32:34 [Music]
0:32:39 [Music]
0:32:42 [Music]
0:32:44 [Music]
0:32:47 [Music]
0:32:50 [Music]
0:32:52 [Music]
0:32:54 [Music]
0:32:56 [Music]
0:32:58 [Music]
0:33:00 [Music]
0:33:02 [Music]
0:33:04 [Music]
0:33:06 [Music]
0:33:08 [Music]
0:33:18 [BLANK_AUDIO]
0:00:14 >> Hello, and welcome to the NVIDIA AI podcast.
0:00:16 I’m your host, Noah Kravitz.
0:00:20 What’s in NASA’s Mars rovers have in common with the quest to cure cancer?
0:00:21 A lot, as it turns out.
0:00:26 Some of the same technology used in the Mars rover missions is now being used to
0:00:29 help detect, predict, and treat cancerous cells.
0:00:31 Which technology, and how does it work?
0:00:34 Here to talk about it is Thomas Fuchs.
0:00:39 Thomas is the Dean of Artificial Intelligence and Human Health at Mount Sinai in New York City.
0:00:43 And he’s also the co-founder and Chief Scientific Officer at PAGE,
0:00:48 the first company with an FDA-approved AI tool for cancer diagnosis.
0:00:53 Thomas, welcome to the NVIDIA AI podcast, and thanks so much for joining us.
0:00:54 >> Thank you so much for having me, Noah.
0:00:57 Very much looking forward to the conversation.
0:00:59 Likewise, there’s a lot to get into.
0:01:00 Let’s start with PAGE.
0:01:03 What is the company all about, and how did it get started?
0:01:08 >> So PAGE is the leading AI company in pathology.
0:01:10 It was founded 2017.
0:01:14 So we spun it out of Memorial Slang Catering.
0:01:19 And at its core, it does AI in cancer research,
0:01:24 and especially clinical care, and there within pathology.
0:01:27 So we look at these digitized pathology slides.
0:01:33 We find cancer, classify cancer, predict response to treatment, predict outcome,
0:01:37 help pathologists to do their job not just faster and better,
0:01:39 but especially help them to do things they can’t do yet,
0:01:44 and then help oncologists to find better treatments for their patients.
0:01:46 PAGE is actually an acronym.
0:01:50 It stands for Pathology, Artificial Intelligence Guidance Engine.
0:01:51 And that’s what we do.
0:01:57 And these days, PAGE is used across the globe on four continents,
0:02:03 and with thousands of patients treated last year already based on diagnosis
0:02:06 that were rendered with the help of PAGE.
0:02:09 >> So applications of machine learning, deep learning, AI,
0:02:13 to use the umbrella term in medicine and health and wellness
0:02:16 have really been exploding over the past few years in particular.
0:02:23 And for my money are one of the most interesting and useful applications of the technology.
0:02:27 We’ve had some different folks, some folks who are working to try to battle cancer
0:02:33 and some other folks doing other things with machine learning on the podcast before.
0:02:36 Let’s dive into how PAGE works a little bit.
0:02:40 My understanding is there’s sort of a combination of the PAGE tools
0:02:46 and then also third-party applications that are used throughout the diagnosis process.
0:02:48 Maybe you can take us in a little bit deeper.
0:02:50 >> Yeah, of course.
0:02:52 I think it’s always good to start with the patient.
0:02:57 So suppose you have some pain in the chest or somewhere else.
0:03:02 First you get the radiology, and then if the radiologist sees something,
0:03:06 the first thing that’s usually done is to take a biopsy.
0:03:10 So a needle biopsy at the location.
0:03:14 And then what comes out of there is some tissue.
0:03:19 And in the last 150 years in which pathology didn’t change much,
0:03:23 the pathologist looked at the tissue through a microscope
0:03:26 and then decided the diagnosis.
0:03:29 So it’s very subjective, of course, in many ways.
0:03:33 So what PAGE does is really help to digitize the whole process
0:03:37 and then build AI on top of it from its beginning.
0:03:41 The whole thing is part of the field of computational pathology.
0:03:44 We coined the term over 15 years ago,
0:03:49 and since then the field exploded, but of course back then it was very niche.
0:03:54 So after you have the biopsy, the tissue is then put on microscope slides,
0:03:58 and these are digitized and end up being enormously large images.
0:04:03 So 100,000 pixels times 100,000 pixels with millions and millions of cells.
0:04:08 So you could fit all your holiday snaps on one of these single slides.
0:04:11 And large institutions produce a lot of them.
0:04:14 So at Monsignor we produce over a million of these slides per year.
0:04:18 Then pathologists really have to look for the needle in the haystack.
0:04:21 If you have, for example, a mastectomy is a breast cancer,
0:04:23 you could end up with hundreds of these slides
0:04:27 and you’re really looking for a few cells or a larger lesion
0:04:29 that actually is cancer or not cancer.
0:04:32 And that’s a very long and cumbersome process.
0:04:35 And that’s exactly where our AI comes in.
0:04:39 So we trained AI at scale to actually find cancer,
0:04:45 and that also led to the first and only FDA approval in pathology, you mentioned.
0:04:50 So to do that, you trained these very large computer vision models first.
0:04:53 These, of course, transformer models these days.
0:04:57 And to do so, we digitized enormous amount of slides over the years,
0:05:02 linked it to all kinds of clinical data and pathology report data,
0:05:07 to train models directly from the image against these reports.
0:05:13 To do so in 2017, we actually built a dedicated compute cluster with NVIDIA DGX.
0:05:18 So the first ones that came out back in the day, which was, of course, great.
0:05:25 And that allowed us to build a model based on 60,000 of these slides for the FDA approval.
0:05:26 And let’s use clinically.
0:05:29 But these days, of course, that’s not enough.
0:05:35 And we build now a very large foundation models from millions of these slides.
0:05:39 And that’s done in partnership with Microsoft and Microsoft Research.
0:05:44 So we can actually use thousands of GPUs to build these very large computer vision models.
0:05:48 Since you brought it up, let’s talk about that foundation model.
0:05:54 My understanding is it’s one of or perhaps the largest image-based foundation model in the world.
0:05:56 So it’s by far the largest in pathology.
0:06:01 And what we are training now, based on four million slides,
0:06:05 is for sure one of the largest publicly announced computer vision models.
0:06:11 What you have to do is actually have to separate these very large slides into images of normal size,
0:06:14 like you would have an image net or other databases.
0:06:19 And in that space, we have 50 billion images.
0:06:24 So again, image net is only 14 million, so it’s 50 billion images.
0:06:27 I take a lot of photos as a dad, but to your point earlier,
0:06:30 I think that’s plenty of room for my holiday snaps.
0:06:35 Well, it depends how many kits you have, you know.
0:06:37 But that’s remarkable, 50 billion images.
0:06:43 Yeah, so if you count actually the pixels, it’s 10 times more data than all of Netflix.
0:06:46 So think of all the shows you watch, all the movies.
0:06:52 So if you count the pixels throughout, that’s just 10% of the image data we pipe through these models.
0:06:56 And the whole point of that is you end up with a foundation model
0:07:02 that truly understands microscopic morphology of tissue very well,
0:07:04 not only of cancer, but also of normal tissue.
0:07:09 And then like in language models, you get these embeddings of these images,
0:07:11 and you can use them for all kinds of tasks.
0:07:15 Again, cancer detection, classification, segmentation.
0:07:18 We also predict molecular changes, for example.
0:07:23 If you have a specific mutation that can be targeted by some therapy,
0:07:28 or sometimes you predict outcome to a therapy or to a treatment regime.
0:07:31 This is making me think of a few episodes we’ve done in the past.
0:07:39 One was with, to bring up NASA, some astronomers who were taking, looking at images, or using AI.
0:07:43 I shouldn’t say they were looking, they were processing images taken.
0:07:48 I think this was with the James Webb, one of the very powerful telescopes, right?
0:07:54 And another one more recently talking about cardiac care and early detection of cardiac disease,
0:07:58 using AI to power the techniques.
0:08:06 And something that stuck with me was this notion that not only can these models
0:08:10 churn through data with the speed and scale that humans just couldn’t do
0:08:13 and detect things that humans couldn’t with the naked eye.
0:08:18 But that in some cases, and I think this is specific to the cardiac care episode now that I think about it,
0:08:23 they were actually able to detect things that, and forgive my lack of medical knowledge here,
0:08:30 that were sort of in areas and types of things that researchers and medical professionals
0:08:36 hadn’t even thought to look for before, because they were almost sort of hidden in the cells,
0:08:40 in sort of the walls of the heart, or what have you.
0:08:45 Is that the kind of thing that you’re finding with your research, and can you speak to that a little bit?
0:08:49 Yes, so you’re pointing actually at the very, very interesting development.
0:08:54 So some of the models we built, especially also the clinical ones that are FDA approved,
0:08:57 that more or less mimic what the human does.
0:09:03 So the pathologist does visual pattern recognition to, for example, do Gleason grading in prostate,
0:09:06 or sub-classify breast cancer.
0:09:12 And that can be replicated based on the training data, or for example, the pathology reports.
0:09:15 But it gets really interesting when you go beyond that.
0:09:18 I mentioned that example for mutations.
0:09:23 So we have also sequence data of 100,000 patients where we have the slides,
0:09:26 and then the somatic sequencing of the tumor.
0:09:31 And for some of the mutations, you can actually predict them based on the image.
0:09:34 So the mutations lead to a change of pattern in the image.
0:09:38 And most of these are not recognizable by humans.
0:09:42 So we were not trained to do that, and pathologists are not trained to do that.
0:09:46 Some of them are very obvious, also humans can see, but others not.
0:09:50 And there, of course, it also gets more into basic science,
0:09:53 because then it becomes very interesting in reverse.
0:09:55 Can we find out what the model looked like?
0:10:01 So was it, for example, as you said, some subcellulum of our phyrophology at the nucleus,
0:10:06 or was it larger vessel structures or a combination of different kinds of cells?
0:10:10 For example, for the immuno-oncology work we do,
0:10:14 you want to detect these tumor-infiltrating lymphocytes.
0:10:17 Is there inflammation close to the cancer or within the cancer?
0:10:23 And then you have very high-level or complicated or nonlinear interactions
0:10:26 that are actually quite difficult for us humans to understand.
0:10:32 So model introspection becomes something very important to know what’s going on.
0:10:36 But philosophically, if you actually spin it a little bit further into the future,
0:10:43 the time will come when these AI models might be able to perform very, very well for one of these tasks.
0:10:47 For example, predict if the drug will work for you as a patient or not.
0:10:55 And we as humans might not be able to understand the intrinsic combinations these models look at.
0:11:02 And then the question is, how can you then still assure that these AI’s are safe and effective and equitable?
0:11:07 And that’s where all the whole FDA framework and all the efforts into testing these models comes in.
0:11:14 And I do think that I hope actually we’re going to get to that place because I want to go beyond the state of the art.
0:11:21 And there’s so much more, of course, in biology and physics, which are underlying these processes that produce that tissue
0:11:27 than we humans can grasp today, but that can really, really help patients to get better outcomes.
0:11:31 Full disclosure, I come from a liberal arts background, not a science background per se.
0:11:40 But all of the talk about all the scientific progress that these technologies are helping further and talk of new discoveries
0:11:45 is what really gets me excited and interested in where this stuff is headed.
0:11:49 I want to come back to the technical aspects of this in a moment.
0:11:54 But I want to ask you, where is page at now in terms of results?
0:11:56 What are you able to do?
0:12:00 I don’t know if success rate is the proper way to phrase it.
0:12:05 But what kinds of actual on the ground results are you seeing in using the tech?
0:12:10 So I think a good place to study is actually the FDA study we ran.
0:12:16 And there we could show that the error rate was reduced by 70%.
0:12:22 So pathologists find much more cancer than they would have before without increase of false positives.
0:12:24 So that’s very important.
0:12:33 And we also showed across the globe on 14,000 patients out of 800 different institutions out of 45 countries
0:12:37 that the AI really works everywhere across the globe.
0:12:42 That’s because at that scale that we train, it really focuses on the morphology.
0:12:49 It’s really on the growth patterns that it’s not thrown off by all kinds of nuisance variables or bubbles or air
0:12:52 or all kinds of things you see on these slides.
0:13:00 And that has been replicated at DA, at Cornell, in Brazil, in Europe, everywhere across the world.
0:13:01 That’s great.
0:13:05 So what’s of course interesting here is that if you think about FDA approvals of AI,
0:13:09 you have hundreds of AI’s approved in radiology.
0:13:11 I think they’re up to 500 by now.
0:13:14 And in pathology, you still have only one.
0:13:18 There’s just one that is actually safe and effective.
0:13:20 And that’s of course a huge difference.
0:13:25 And one reason is that the FDA looks differently at the two domains,
0:13:28 radiology to a large degree is a screening step.
0:13:33 You can always go back to another x-ray or another mammography.
0:13:37 But in pathology, it analyzed the diagnosis.
0:13:40 And you don’t have cancer until the pathologist says so.
0:13:44 So it’s much more dramatic in the treatment pathway.
0:13:51 But it also shows that there’s that huge chest in between all that height we have in AI,
0:13:57 15,000 biotech AI companies and all that fluff and everything.
0:14:00 And then the reality in the ground.
0:14:04 So if you have physician in the trenches and you want to use something that’s safe and effective,
0:14:05 you have the choice of one.
0:14:07 And that’s of course something we have to change.
0:14:12 So PAGE does that drastically and actually expanding to all kinds of different cancer types.
0:14:15 So we have, for example, a breakthrough designation for the breast system.
0:14:18 We can do bladder lymph nodes and so forth.
0:14:23 And then we use these very large foundation models we discussed, especially the new one.
0:14:28 It’s actually named after the founder of pathology, Rudolf Wirhoff.
0:14:33 So it’s the Wirhoff one model, the V1 model, and V2 will follow soon.
0:14:38 And we can actually use it for pan-cancer applications.
0:14:42 So instead of just doing what we just described for prostate cancer breast,
0:14:45 you can address nearly all cancers.
0:14:49 And 50% of cancers are rare cancers.
0:14:52 So if you inflict cancer, it’s a coin toss.
0:14:55 If you have a big one where there are a lot of treatment options,
0:14:58 or you have one of these rare ones where there’s nothing available.
0:15:04 And the foundation models allow us because they understand morphology in the body of tissue
0:15:09 so well that you can hit the ground running with few-shot learning on rarer cancers
0:15:14 and rarer conditions, and then also produce AI for these.
0:15:16 And that’s very, very promising.
0:15:17 Absolutely.
0:15:20 You sort of touched on this mentioning the few-shot learning,
0:15:23 but are there specific types of cancers,
0:15:27 and please replace types of cancers with a more precise term,
0:15:33 but are there specific variants that are more difficult to detect
0:15:35 because of technical problems?
0:15:37 And how are you trying to get around some of those?
0:15:38 Yes.
0:15:41 So there are usually two reasons.
0:15:43 First off, they might be rare.
0:15:51 So you have the big ones like breast, prostate, lung, derm, and so forth.
0:15:57 And then rarer ones like cholangiocasinoma, so that’s phyldoc, which is terrible.
0:16:02 And there usually it didn’t have the numbers to actually build robust systems that channel us.
0:16:05 And that’s what the foundation model is changing.
0:16:10 And then you also have within the large cancers, you have subtypes
0:16:13 that can be very rare and difficult to diagnose.
0:16:18 We just published results in breast cancer that the foundation model now
0:16:22 can actually really find these super rare forms as well,
0:16:26 which you wouldn’t do with a normal breast AI.
0:16:31 And that actually shows that it’s worthwhile building these very large systems,
0:16:34 although of course they are tremendously expensive.
0:16:38 Just the production of the data that go into these systems is very cumbersome.
0:16:43 But it really leads to the benefits for the patients at the end of the day.
0:16:44 Amazing.
0:16:46 And a question that popped into my mind earlier.
0:16:52 Are your system, your techniques sort of backwards compatible with older,
0:16:57 lower resolution, or not necessarily older, but just kind of lower resolution images?
0:17:02 Or is it all built atop kind of a, I don’t know, proprietary,
0:17:08 but sort of current very high res way of scanning for cancer cells?
0:17:10 So that’s a good question.
0:17:14 So most scanners that are out there, the high throughput scanners,
0:17:20 either have a 200 times magnification, 20x, or 440x.
0:17:23 And we can work with both.
0:17:24 Okay.
0:17:28 On the cancer detection side, so we could show that it’s equivalent.
0:17:33 Sometimes if you have very specific tasks that are really tailored towards,
0:17:37 I don’t know, the membrane of the nucleus, then of course you want to have higher resolution.
0:17:39 So it depends on the task.
0:17:41 I’m speaking with Thomas Fuchs.
0:17:46 Thomas is the Dean of Artificial Intelligence and Human Health at Mount Sinai Hospital in New York City.
0:17:50 And he’s also the co-founder and Chief Scientific Officer at PAGE,
0:17:54 as we’ve been discussing, PAGE is an AI company focused on pathology.
0:18:00 And they are the first company with an FDA approved AI tool for cancer diagnosis.
0:18:06 Thomas, I want to shift gears a little bit, kind of jog your memory back to something we tease in the opener.
0:18:10 You helped out with some of the technologies in the Mars rover.
0:18:13 I think was it perseverance in particular?
0:18:14 Yeah.
0:18:20 Yeah. So the techniques are used today actually for per se and curiosity before.
0:18:24 The images we worked with were on the previous rovers.
0:18:25 Okay.
0:18:31 And it was the Athena class, so spirit and opportunity, and then also from orbit.
0:18:36 So I had the enormous privilege to work for JPL for NASA a few years.
0:18:38 Did JPL, the Jet Propulsion Lab?
0:18:39 Yes, yes.
0:18:44 Athena, the Jet Propulsion Lab, who is of course handling the Mars rovers.
0:18:55 And what we did is we, for example, used the imagery from the navigation cameras to differentiate sand from gravel and from other terrain types so they don’t get stuck.
0:18:56 Right?
0:18:58 So spirit got stuck in this powder.
0:18:59 Oh, right, right.
0:19:00 Yeah, yeah.
0:19:04 And you want to, for example, omit that and it didn’t hit obstacles and so forth.
0:19:10 And we also had imagery from the Mars Reconnaissance Orbiter, so which is the satellite around Mars.
0:19:18 And the images it takes of the ground actually have quite good resolutions, so 30 centimeters per pixel in the best case.
0:19:19 Okay.
0:19:23 And they are nearly exactly the same size as these histology slides.
0:19:25 So they’re really enormously large.
0:19:33 And you also try to classify different terrain type to, for example, look for good landing spots for the navigation and so forth.
0:19:36 And yeah, the beauty of machine learning and the eyes.
0:19:37 Right.
0:19:40 Use it from the microscopic to the microscopic level.
0:19:50 And so with the rovers, how does it work sort of training and updating the model and then running inference from that distance?
0:19:58 Were these, you know, the systems all on the rovers and you would push an update out or I’m getting a little over my head here.
0:20:04 But how did that work kind of, you know, doing all this work to help real time navigation but from so far away?
0:20:05 Right.
0:20:16 So most of the research we did back in the time was, of course, more basic research show that capability or again do it from orbit for all kinds of purposes like looking for landing sites.
0:20:24 One of the big problems, of course, in space exploration is that as soon as you send something out there, it has to be radiation hardened.
0:20:30 And that means the machines are usually 10, 10 years behind what you actually have.
0:20:31 Right.
0:20:33 There’s no GPU out there.
0:20:48 I worked on another project where we did actually computer vision for flyby missions at asteroids or for example, Pluto, where the light gap is so big that you large that you can’t send meaningful comments out there.
0:20:52 And then it really automatically has to, for example, find the asteroid to take pictures.
0:21:00 And there also JPL had worked on building specific FPGAs and ASICs that could do that in the future.
0:21:05 But that’s certainly one limitation that you can’t use the…
0:21:06 Right.
0:21:08 And maybe that’s something for NVIDIA, right?
0:21:11 Have radiation hardened GPUs for that specific…
0:21:12 No, you read my mind.
0:21:17 I was thinking about the, you know, special edition 40 series radiation hardened.
0:21:30 So in addition to the work you’re doing now with PAGE and Mount Sinai and the JPL stuff, you’ve had a hand in at least a couple of other really interesting and sort of large profile, high profile projects.
0:21:34 The Memorial Sloan Kettering AI Research Department.
0:21:38 And then also there’s a supercomputer in New Jersey, I believe.
0:21:39 Yes.
0:21:40 Tell us a little bit about those.
0:21:41 Yeah, sure.
0:21:43 So that was the one I was referring to.
0:21:46 We built in 2017, 18 specifically for PAGE.
0:21:56 So it’s still the largest standalone supercomputer only for pathology, and we extended it over the years, but it’s of course not big enough to train these humongous foundation models.
0:21:57 Okay.
0:22:10 That’s why we needed a relationship with Microsoft, not only for compute, but because of course they’re fabulous engineers and scientists and it’s really a great partnership to drive that forward on modern hardware to build that.
0:22:25 Because of course we are always, always limited with compute, right, be it at PAGE, but also at Mount Sinai, of course, there’s so much fantastic research that could be drastically accelerated if the research community had access to more compute.
0:22:40 But it was, of course, a very interesting exercise. I mean, we wrote papers and were published in Nature Medicine and Well-Cited and so forth. But the secret behind all of that is of course engineering, right? It’s MLOps.
0:22:53 How do you actually build a stack? So usually in that case, the problem are not even the GPUs, but it’s I/O. How can you pipe adabytes of data quickly enough to the GPUs to actually train these very large models.
0:23:07 And so there’s an enormous software stack to build computer vision models at scale and enormous experience that PAGE has in doing these applications that are, of course, broadly applicable in all kinds of computer vision tasks.
0:23:12 Or now, of course, we do multi-modal approaches with HACS and radiology and so forth.
0:23:14 Your own background is originally in engineering?
0:23:22 Yeah, so I had my undergrad was in mathematics in Austria, which you can hear from Arnold Schwarzenegger accent.
0:23:23 Sure.
0:23:27 After that, I did a PhD in machine learning at the time where it wasn’t cool yet.
0:23:28 Yeah, yeah.
0:23:37 Europe’s was 400 people and it was very teach and combining, of course, AI with pathology was even more niche back then.
0:23:40 And I was going to ask, how did you find your way into working in pathology?
0:23:54 Yeah, so that was at the beginning of my PhD one day pathologist of the hospital we worked with came into the lab and just wanted some high level statistics based on the excel sheets, but they showed us a few images.
0:23:55 Right.
0:24:01 And as a naive student, I saw these and said, oh, these are just roundish object cells should be easy to detect.
0:24:04 And now we are 20 years later and we still can’t do it.
0:24:10 But at least I stuck with it and we can do it quite well.
0:24:11 Amazing.
0:24:12 These origin stories are the best.
0:24:14 So many of them are so similar to that.
0:24:18 You know, the student working on this, somebody wandered into the lab.
0:24:21 I was the only person there 20 years later.
0:24:26 I’m working with Microsoft to build the world’s largest foundational image model to fight cancer.
0:24:27 Amazing.
0:24:31 A lot of times we like to end these shows and kind of a forward looking note.
0:24:33 I’m going to give you a two part thing here.
0:24:45 The second part, I want to kind of open it up given your background and, you know, your hands on engineering prowess as well as obviously all the things you’re doing in the real world to ask you about AI more broadly.
0:24:53 But before that, Paige, your work at Mount Sinai, you talked about this a little bit, but to kind of put a point on it.
0:25:03 Where do you see the future of AI and medicine and healthcare and we can narrow it down specifically to, you know, using AI and pathology?
0:25:04 Where do you see things headed?
0:25:15 What are kind of the big obstacles beyond, of course, everybody needs more compute, you know, kind of the big hurdles that you’re looking to get over near term and take it out as far as you want as far as what you see in the future.
0:25:26 Yeah, of course. Let’s start with pathology. So pathology has a very unique challenge and that that for the last 150 years, it really didn’t change.
0:25:30 So first has to go digital before it’s completely AI based.
0:25:39 So part of what AI does, sorry, what Paige does, it also helps large labs and academic institutions to actually go digital.
0:25:46 So get these canners, link it to your LIS, make sure everything flows and then it’s a SaaS service in the clouds.
0:25:51 You can access it from everywhere and you can diagnose from everywhere and so forth.
0:25:55 So that’s something that has to be driven in pathology.
0:26:03 Another part is, of course, it’s important that the FDA also as they want to do now take oversight.
0:26:15 So there are some loopholes, some other companies who don’t validate at that level of scrutiny I mentioned can use and these LUT loopholes should be closed.
0:26:21 So in healthcare much broader and that’s what brings us to Mount Sinai, which is of course a very unique place.
0:26:27 It’s the largest health system in New York City with eight hospitals and 200 locations and so forth.
0:26:32 There we are trying to more or less replicate what we did in pathology for all these different areas.
0:26:46 So really level up in cardiology, in radiology, in psychiatry and AI now plays a role in all these areas from basic research in drug discovery and drug development to the bad side.
0:27:00 So Mount Sinai has more than 20 models that are actually used in the hospital to direct the care teams to the right patients who had risk for seizure or malnutrition or for other things.
0:27:21 And then on the basic research side there’s work in cardiology based on EKGs and ECGs to actually early predict all kinds of cardiology issues or an oftalmology where we look at fundus images to try to even predict new degenerative diseases from that.
0:27:31 Mount Sinai is also building a neuropathology AI center where 10,000 brains are gonna be scanned and that’s of course for work in Parkinson’s and in Alzheimer’s.
0:27:32 Amazing.
0:27:41 And so in all these areas AI is a key to provide capabilities we don’t have yet, mid-care or mid-research.
0:27:48 Besides all these doomsday scares that AI is an existential threat for us and so forth.
0:28:03 I think especially in healthcare we have the moral obligation to develop AI as fast and as safe as we can because patients are dying not because of AI but because of the lack of AI.
0:28:13 In the US alone 800 patients die or are permanently disabled because of diagnostic errors.
0:28:21 And as we just discussed for example in pathology as we proved out these are things that can be addressed to a large degree with AI.
0:28:30 So we have to build the tools for the physicians to give better treatment and a slowdown there is certainly not responsible.
0:28:35 Now I’m gonna ask the old school engineering PhD and you.
0:28:47 Looking at the current looking ahead a little bit when it comes to the field and the technical everything going on technically and it’s been a what’s the word it’s been a whiz bag.
0:28:59 There’s a better word but it’s been a whiz bag couple of years in particular, at least as far as you know public exposure to everything going on but I’ve had the chance to talk to a bunch of engineering leaders who’ve just been saying
0:29:05 I’ve never seen this kind of rate of acceleration and in tech you know in my career anyway.
0:29:10 What are you excited about what kinds of you know things on the technical ground level.
0:29:15 Are you looking forward to you know in the next I don’t know a year or two years five years.
0:29:34 Very mundane side is just to get these tools into the clip right the first thing huge hurdle and usually it’s it’s not capabilities it’s not regulation usually it’s reimbursement very very petty monetary issues that where health systems are hard to change.
0:29:41 So on the AI side, of course a lot of the current excitement is driven by language models.
0:29:47 And that’s only for some reason, of course for good reasons we humans over index language.
0:29:54 Yes, everybody who can who can string five sentences together in a coherent way seems to be intelligent.
0:30:03 And that’s why we certainly assign a lot of of capability or properties to large language models that might or might not be there.
0:30:17 But at the end of the day language is of course a human technology it’s produced by our brain when so our brain our relatively feeble wet brain with it with its few neurons can produce language and reason about language.
0:30:24 But if you go beyond that, for example in biology and cancer research and if you look at the issues there.
0:30:30 Just at a single cell all these processes are produced by physics and by biology.
0:30:41 Just if you think of all the proteins that are at play at a single moment in a single cell that is in complexity so far beyond language.
0:30:49 That there’s a whole universe out there, literally our physical universe and then the biological one that goes far beyond language.
0:30:59 You see it even now in pathology the large models we built we touched upon that we humans are missing the vocabulary to even describe it in language.
0:31:07 And then usually we come up with all kinds of comparisons where these cell look like people in single file or these cells look like this.
0:31:10 But there’s so much more going on.
0:31:18 We start to capture with their eye be image based be genomic based and so forth that goes beyond our capabilities with language.
0:31:27 And I think that space is going to be dramatically exciting because that will deliver new drugs better care better outcome.
0:31:31 It’s an exciting time to be alive to see transformation.
0:31:32 Amazing.
0:31:34 Thomas for listeners.
0:31:39 What while they ponder all of this who want to learn more about what page is doing.
0:31:42 Maybe what’s going on in Mount Sinai and the other things we touched upon.
0:31:47 Are there some resources on the web where you you would direct folks to go.
0:31:48 Of course for page.
0:31:53 So just go to page dot A.I. B A I G E dot A.I.
0:32:02 And at Mount Sinai if you look for Mount Sinai you see our initiatives there and the I centers and if you want to go beyond pathology that’s of course the place to go.
0:32:03 Excellent.
0:32:05 Thomas this has been an absolute pleasure.
0:32:06 Thanks so much for making the time.
0:32:10 There’s a lot to chew on here and a lot to be optimistic about.
0:32:15 So we appreciate all the work you and your teams are doing and for taking a few minutes to tell us about it.
0:32:16 Thank you so much Noah.
0:32:17 It was really a pleasure.
0:32:19 Thank you for the great questions.
0:32:21 [Music]
0:32:23 [Music]
0:32:25 [Music]
0:32:27 [Music]
0:32:29 [Music]
0:32:31 [Music]
0:32:34 [Music]
0:32:39 [Music]
0:32:42 [Music]
0:32:44 [Music]
0:32:47 [Music]
0:32:50 [Music]
0:32:52 [Music]
0:32:54 [Music]
0:32:56 [Music]
0:32:58 [Music]
0:33:00 [Music]
0:33:02 [Music]
0:33:04 [Music]
0:33:06 [Music]
0:33:08 [Music]
0:33:18 [BLANK_AUDIO]
Improved cancer diagnostics — and improved patient outcomes — could be among the changes generative AI will bring to the healthcare industry, thanks to Paige, the first company with an FDA-approved tool for cancer diagnosis. In this episode of NVIDIA’s AI Podcast, host Noah Kravitz speaks with Paige cofounder and Chief Scientific Officer Thomas Fuchs. He’s also dean of artificial intelligence and human health at the Icahn School of Medicine at Mount Sinai.
Tune in to hear Fuchs on machine learning and AI applications and how technology brings better precision and care to the medical industry.