AI transcript
[MUSIC]
Hello, and welcome to the NVIDIA AI podcast.
I’m your host, Noah Krabitz.
Our guest today is a machine learning and AI leader who’s worked on projects ranging from
sustainability and reforestation efforts to creating a virtual closed swap platform for
environmentally friendly fashion and steps. But for the past year and a half or so, she’s been
serving as director of machine learning at the non-profit Surgical Data Science Collective,
where she leads research focused on utilizing video data from surgeries to develop tools
that can provide surgeons with immediate feedback and insights on their performance.
Just a recently gala TEDx talk titled, “Why You Want AI to Watch Your Surgery?”
which I encourage you all to go check out on YouTube after you listen to our conversation.
Because she’s here right now to talk with us about the potential for AI to help surgeons bring
better health care to everyone. Margot Mason Forsythe, welcome, and thank you so much for joining
the NVIDIA AI podcast. Hi, Noah. Thanks for having me. Margot, maybe we can still
do a little bit about your background. I alluded just a little bit in the intro. You’ve
worked on various projects after studying machine learning and leveraging your skills
and experience for a lot of AI for what we’d call AI for good projects, but really just
in my view, projects that are helping improve quality of life for everyone.
So maybe you can detail that a little bit and then kind of to bring us up to the present
with how you started that involved with the Surgical Data Science Collective.
Yeah, sure. So like you said, I’ve been working in the AI field for quite some time.
My first field of studies was come to science, so I’ve done enough software development,
and then I realized that I wanted to do a bit more scientific projects. So I went back
to finish my masters and specialized in computer vision. Computer vision is something that I really,
really love because I’m a very visual person. And so analyzing images and videos is something
that I’m pretty passionate about, I would say. And from that, I’ve worked on many different
projects with very big video and image files. So like you said, I’ve worked on several projects
that go from analyzing lumber scanning, for example, to satellite imagery, to detect a
deforestation, or now surgical videos. So it’s been quite a ride for sure. And I’ve learned
a lot mostly on how to producturize AI and how we can use AI to make an impact in the world and
have a focus on is it actually going to be useful, you know, and make a difference at some point.
So that’s kind of how I see my career so far.
Have you found that the projects you’ve been drawn to, has it been kind of being drawn to the
next sort of technical or scientific challenge, kind of pursuing the craft and kind of pushing
the boundaries of what computer vision and image and video analysis can do? Or have you been driven
more by the mission of these different projects or have you just kind of worked out that you sort
of been able to follow both of your blessings, so to speak?
I would say both actually, yes. Slightly luckily, always.
I mean, I’ve always been passionate about different sciences. So I actually have a
hard time focusing on only one thing or one project. And when I learned about climate tech,
for example, I really wanted to see how I could help and how AI could help process all of this
giant xactolite and imagery, you know, remote sensing is really hard to process.
And then I learned about surgical videos and I learned that there were thousands of terabytes
of surgical videos that were not used. And it’s a pretty big challenge because surgical videos are
really heavy, really long, you can imagine eight hours procedure. No one really wants to watch
those videos. So that’s when I was thinking that AI is indeed the perfect tool for this kind of data
that is really being really long, but also has a temporal element to it, which is quite difficult.
And when I learned about that, it was a really interesting impactful project, but also in terms
of technical challenges, I thought it was really interesting. And that’s actually what made me
join SDSC in the first place. So how did you find out what drew you into the
troubles of surgical videos? I started to work at the surgical data science collective SDSC
when I met the founder, who is a pediatric neurosurgeon, Dr. Denoho. And he introduced me
to this issue, you know, he was telling me I have all these videos saved in drives and I don’t do
anything with them. And I know any of my coworkers and friends and other surgeons have drives with
hours and hours of surgical videos, and they are just sitting on the desk, not really doing anything
with them. Right. Just a context set for the audience, for me too. Is it standard procedure that
surgeries are just all video to these cameras that are inside of people, sort of, you know,
guiding the surgeons? Or are they sort of operating room overhead cameras? Or how does this, how does
it work surgical video? That’s a great question. It’s pretty diverse. We have a lot of endoscopic
videos and microscopic videos. So endoscopic will go, for example, through the nose or any
other part of the body where you need to see inside. And actually, the endoscopic videos are
a really good data point for us because we see the computer vision algorithm sees what the surgeon
sees. Right. And that is, you know, golden because there is a lot of information in these videos.
For the microscopic videos, it is also used by the surgeons sometimes when they do the surgery to
really magnify what they’re looking at. For example, if they’re operating on very small arteries,
they need to have this big intense zone. That’s what they’re going to be using. Often it seems 3D
for them, which we have some of those 3D videos as well. So yeah, it’s a mix of microscopic surgical
videos and doscopic videos. And we don’t have yet the video, you know, kind of like camera security
from the OR, but some of our collaborators use this kind of videos to get a sense of what is
happening in the operating room. Right. And so you met the founder. So the surgical data science
collective was already in existence and you met the founder and got involved? Yes. It was pretty
early on. So the surgical data science collective is a non-profit organization that was started by
Dr. Daniel Ho and the main mission and the main idea was to create and analyze a repository of
surgical videos in order to improve surgical techniques and patient outcomes. Because still
today, there are 5 billion people who lack access to safe surgery and there are at least 4.2 million
people around the world who die within 30 days of surgery. So if we consider surgery as a disease,
it would be the third leading cause of death. That’s why, you know, the goal of SDSC is to
utilize this annual surgical videos to identify best practices, support medical education,
or even predict potential outcomes and complications in advance of surgery.
Right. So we’ve had other healthcare practitioners and people from the health industry on the podcast
talking about ripping the cums to mind is talking about analyzing still images, you know, x-rays
and scams and using AI to discover things often kind of, as you said, at that super zoomed in level
that, you know, cancer prediction and that kind of thing. Tell us about some of the opportunities
and challenges involved with analyzing all of the surgical video. I would assume, you know,
the first challenge is just gathering the data and then processing all of it. But
kind of take us through it. What is it that, you know, you said improving best practices and real
time feedback. So maybe you can speak to that as well. Yes. So the first challenge, like you said,
is actually to gather all of this data. And like I said earlier, a lot of these surgical
videos are stored on drives and it’s really difficult to get access to these drives. Sometimes,
you know, you kind of have to go and fly somewhere and meet with the surgeons to be able to get the
videos. Most of the time, the videos are not even recorded because people don’t know what
they can be used for. So why would they record them? So actually, one of the biggest challenge
is asking people to press the record button. Right. I mean, I’m laughing, but I’m imagining,
you know, if I was a surgeon, that’d probably be the last thing on my mind, right? So yeah.
Exactly. Yeah. I mean, that is definitely not the priority. And then if they think about recording,
so pressing this button, they have to export the video from the device, walk around with a USB key,
upload the videos on a laptop, upload to the cloud. So there are so many steps here for these people
who are extremely busy, who have so many other important things to do of their day. It’s definitely
not a priority. So that is our first challenge, and that has been one of the biggest challenges
that we’ve had. But we’ve been pretty successful in gathering at least a good first base of a
surgical video library. By now, we are about 40 terabytes of surgical videos. Okay. And we expect
to get more, you know, and the other challenge here is to get diverse surgical videos. We don’t want
obviously for an AI model, we don’t want videos from one surgeon in one hospital doing the same
procedure. Hundreds of hours of tonsillectomies is only going to get you so far, I’d imagine.
Yes, exactly. So that is the other challenge is how do we get these videos from diverse sources
and diverse fields, which is also a lot of networking, because you have to go and talk
to the people and ask them to record and then do they want to work with us so that we can start
gathering these videos. So this is the other second part of this challenge of data collection.
But in terms of the other challenges, obviously, surgical videos are quite long, but they are also
temporal. So videos, right? So it is a different type of models that you would use for still images.
We have the kind of the same architecture that you would use for other models,
but we always have to think about the temporality of what is happening in the video and that is
actually how we implement most of our models. Let’s say if you’re trained to track surgical
tools, you know, you have to think about all of the challenge that comes with that in surgical
videos, which are going to be obstructions and can sometimes you have, you know, explosion of blood
or something like that. And you want to be able to subtract the tools without losing them or
with dealing with these problems, which are pretty similar to other computer vision problems.
But it is slightly more challenging because of how messy these environments are.
Sure, I can only imagine. And so from a technical perspective, you know, obviously there are,
we’re hearing all the time these days about video models in the news kind of being opened up for
you know, consumer use, that kind of thing. You’ve been working with the Data Science Collective
for going on two years now. Is that right? Yes. So are you using, are you are you building tools
yourself? Are you using off the shelf to kind of modifying them to suit? Do you have partnerships
with other AI labs? How are you kind of fine tuning the tools to get what you need out of them?
A mix of all of what you just said, we usually will. So, you know, we’re a pretty small team and
a nonprofit organization. So we will try to use the most efficient methods for us. A lot of the
times that we’ll be reusing some architectures that are existing and then fine tuning them to our
needs. Combining some architectures is something that we’ve done a lot, especially with the temporal
model. So having, you know, a mix of a CNN and a temporal architecture or we’ve been playing that
with vision transformers and more recently with vision text transformers, which are the big models
you’re talking about here. And a lot of the time we will always be careful about new technologies.
So we want to try them and we want to make sure that we stay on, you know, on top of the innovation
that is happening to see if we can apply it to the surgical data science field. Something that is
quite interesting and challenging with this kind of data is that it requires a lot of expertise
that as computer scientists, engineers, we don’t have. So we need to work very closely
with clinicians and surgical experts. And that’s where the most important part of the work is
happening, actually, not even the model architecture or the new cool AI tools. For us, it’s really
understanding what the expertise is and then what model should apply to bring that information that
will be useful to the surgeons. Can you maybe walk us through an example of, and correct me if I’m
wrong here, but I imagine you have partnerships with surgeons and other medical professionals
and institutions and are you sending them images or videos to just kind of analyze and let you know
what they see or kind of what’s the, I guess I’m wondering what the process is or what it’s like
getting from footage and people revealing the footage to then an outcome that other practitioners
can benefit from, whether it’s, you know, a new technique or refining a best practice or something
like that. So for most of our collaborations, we will work with clinicians and surgeons who have
videos, but they don’t have the computer science knowledge. So they will come to us to do all of
the computer vision and analysis. So when they start working with us on these projects, maybe I
can go through a concrete example. We’ve been working with several NGOs who are focused on
surgical education. One of them is called All Safe and they focus on teaching surgical procedures to
several students all across low income countries and they do it through a digital platform. So
it’s online courses and then the review is done through videos. And so what we’re trying to see
here is can we analyze these videos and give feedback to the students with computer vision
and that is useful. So that is the important point is that is useful. So then we will work on
in developing the computer vision models to extract the features that we need to be
extracted to do the analysis and then collaborate with the clinicians and the surgeons on what
exactly do they need to have in the feedback or what do they believe is something we should focus
on because most of the time, you know, we’re going to look at something and I’m going to think with
my engineer mind. Oh, I’m going to look at this feature and I’m going to make this graph and
it’s going to be amazing. And then I say the surgeons and they’re like, what? So that’s why
it’s called the social data science collective because it’s the first step is creating a community
with the clinicians and the computer science experts. And we also have some collaboration
with computer scientists groups. So where we will work with them to analyze some of the videos we
have. So that is almost a connection between we have surgeons who want to do something very specific
and we have computer scientists collaborators who can help us do that specific task that maybe we
don’t have bandwidth for. So we are trying to expand that part of our community as well to really,
you know, have a really impact and scale that because it’s not only going to be a whole community
effort. I’m speaking with Marder Mason Forsyth. Marder is the director of machine learning at
the Surgical Data Science Collective, a nonprofit that is using AI machine learning tools to analyze
video data from surgeries to develop tools and feedback loops and other mechanisms that can
help surgeons with insights and feedback on their procedures and techniques and really just
bring better health care to more people across the globe. As Margot was just talking about,
you mentioned, you know, being a nonprofit doing AI research is a little bit unusual right now.
What is that like? Are there big things that are either, you know, well, I mean, I’ll be
saying I would imagine something. Resources is an issue as it is for almost all nonprofits. But
there are things, are there things specific to being a nonprofit AI kind of research group
that stick out to you? So there are many interesting aspect that come from being a nonprofit. Like you
said, resources are indeed limited. So we have to be creative in the way we train computer vision
models. We will always start simple, which is actually something I’ve always done and advocated
for is if you want to start a computer vision project, maybe you don’t need to start with the
biggest model that exists. You know, start simple with a small data set, do a proof of concept,
and then iterate. So that is what we have as a development pipeline and research pipeline
process. We will always start simple and small and then scale. And that is limited because of
our resources, obviously. But to me, it’s something that is actually good that I would even, I would
do the same if I had, you know, 10x budget. I would probably do the same, but it helps in that way.
And then it brings a lot of different projects. Being a nonprofit, we’re able to work on a project
that maybe we wouldn’t be able to work on if we were a for-profit. For sure it would be, it would
actually be completely different. And that’s why I really wanted to give SDSC a shot when I first
met Dr. De Noho because I was curious about it. I was like, how are we gonna do that? You know,
I’ve never done AI. I’ve never seen AI done in a nonprofit. There are some others, but it’s
really research focused and community focused, which you wouldn’t really be able to do as well
in a for-profit, I believe. We’re gonna ask this an answer, please, based on what’s happened so far
and/or, you know, what you see coming in the near future. What are some of the big benefits
for clinicians, for patients that you’ve seen or expect to see from, you know, not just the work
that you’re doing at the collective, but more broadly leveraging AI to help with the surgical
process? The AI field will bring a lot of new and good things to the medical field, I believe.
In the surgical space, which is what I’ve been exposed to mostly, it will bring a lot of standardization.
I believe something I’ve discovered working in that field is that every surgeon and every hospital
will perform surgeries and procedures in different ways, and no one really knows
ABCD of how you’re supposed to do a specific procedure. So by having a tool here, AI, to first
encourage people to collect the data. So first, we’re gonna get the surgical videos, we’re gonna
finally start looking at these videos that are not being looked at, and then share them between
surgeons all across the globe that will bring a lot of standardization, or at least they will
start to talk to each other, which I think is kind of beautiful, because right now, you don’t really
have a good way to talk to each other. And through the surgical videos, the hope is that they will
start to talk to each other. And when you can imagine so many applications, for example, the one
that always comes back is education. Instead of using a medical textbook with drawings, the students
can watch a tutorial on how to do the specific procedure. So that’s a big difference that I
think will change a lot of things. And then being able to find best practices through this
analysis of surgical videos is gonna be pretty interesting, because who knows what is in these
videos. And there’s so much that has to be discovered, and there is a big need to be creative
when we think about this data, because no one has ever looked at this data, and no one has ever
really thought about what can we do with all of that, and what is my question that I want to be
answered. And that’s one of our challenges, actually, is sometimes we ask surgeons, “Oh,
what do you want to answer through all of these videos that you have?” And they don’t really know,
because they haven’t had this option before. Right. That’s interesting. It makes me think of
that. I mentioned there were some of the examples of MRI and scan analysis and cardiac care and that
kind of thing. And I’m thinking about the AI tools being able to help practitioners find differences
in cells on a very, very sort of nano basis, right? But even with that, I’m thinking, “Oh,
well, they know what they’re looking for.” Or even if it’s they’re looking for an anomaly,
it’s still kind of we know what we’re looking for. But yeah, with surgery, my very kind of naive,
not knowing much about the field coming in this conversation thinking, “Oh, well, video footage
is being used to train AI systems. Are we moving towards better education for humans or even training
robotic surgical algorithms or that kind of thing?” But it’s fascinating to hear people say that it
makes sense to me as a non-surgeon that what would they be looking for? It’s not the same as looking
for an anomaly in a cell that might stick out. I think at the beginning probably of the when
they first started to analyze MRI with AI, they also had to be creative because someone had to be
asking for these questions for it. And for surgical videos, one of the first steps would be to look
at anomalies, which actually was trying to do now is what are the outliers? Who is using this tool
and no one else is using it for the same procedure? So we are kind of starting with the low hanging
foods, I guess, but the deeper existential questions are not there yet. And I’m really excited to work
with the clinicians to help them come up with these questions by showing them the data because
no one else is going to come up with these questions. It has to be the people who are working
every day in the OR. And actually, the videos are a really great source of data, but there is so much
more going on. Obviously, there is the patient data, there’s the patient outcomes, there is
everything that is going on in the operating room. And all our engineers have actually been
in the operating room so that they understand what is happening behind that camera. And I’ve
been in the operating room to a couple of times now. And it’s really helped me understand better
what is happening. And sometimes when we have a new procedure type that we’re exposed to,
I go to the OR because I want to understand better like, oh, some random questions sometimes
like, where are you? Where is it in the body? Or how many people are operating? Because sometimes
you have more than one surgeon. It’s just so many things that you don’t capture in the video,
but there’s still obviously a lot of information in the videos.
Fantastic. I go for listeners who would like to learn more or hopefully perhaps there’s even
some surgeons, some clinicians listening who are thinking, oh, I have surgical video that, you know,
in a shelf on a drive somewhere, maybe I can send it and how about the cause? Where can listeners go
to find out more about the work that the Surgical Data Science Collective was doing,
the work that you were doing perhaps to get involved as a partner? Who knows? Where can listeners go
to learn more? So we have our website is thesurgicalvideo.io and we can also find us on social media
at Surgical Data Science Collective. And I would also encourage if anyone is a computer engineer,
computer scientist who wants to work on a different project that they’ve been working on
and are interested in surgical AI to also reach out to us because we are working with quite a lot
of different parts in this. So anyone who is interested should reach out to us.
Fantastic. Well, Margot, thank you so much for taking the time to stop by, join the podcast
and talk a lot about the work you’re doing. It’s, I don’t know, stories like this with the technical
aspects kind of match up with the societal impact. I think they’re just fantastic stories.
There’s sort of something for everybody, right? And it sounds like you’re finding a really interesting
path to fuse your technical interests with making an impact in your own work. So congratulations
and all the best developed to you and all of your partners and cohorts at the collective.
Well, thank you, Noah. Thanks for having me on the podcast. It was really enjoyed the conversation.
Me too, our pleasure.
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[BLANK_AUDIO]
Hello, and welcome to the NVIDIA AI podcast.
I’m your host, Noah Krabitz.
Our guest today is a machine learning and AI leader who’s worked on projects ranging from
sustainability and reforestation efforts to creating a virtual closed swap platform for
environmentally friendly fashion and steps. But for the past year and a half or so, she’s been
serving as director of machine learning at the non-profit Surgical Data Science Collective,
where she leads research focused on utilizing video data from surgeries to develop tools
that can provide surgeons with immediate feedback and insights on their performance.
Just a recently gala TEDx talk titled, “Why You Want AI to Watch Your Surgery?”
which I encourage you all to go check out on YouTube after you listen to our conversation.
Because she’s here right now to talk with us about the potential for AI to help surgeons bring
better health care to everyone. Margot Mason Forsythe, welcome, and thank you so much for joining
the NVIDIA AI podcast. Hi, Noah. Thanks for having me. Margot, maybe we can still
do a little bit about your background. I alluded just a little bit in the intro. You’ve
worked on various projects after studying machine learning and leveraging your skills
and experience for a lot of AI for what we’d call AI for good projects, but really just
in my view, projects that are helping improve quality of life for everyone.
So maybe you can detail that a little bit and then kind of to bring us up to the present
with how you started that involved with the Surgical Data Science Collective.
Yeah, sure. So like you said, I’ve been working in the AI field for quite some time.
My first field of studies was come to science, so I’ve done enough software development,
and then I realized that I wanted to do a bit more scientific projects. So I went back
to finish my masters and specialized in computer vision. Computer vision is something that I really,
really love because I’m a very visual person. And so analyzing images and videos is something
that I’m pretty passionate about, I would say. And from that, I’ve worked on many different
projects with very big video and image files. So like you said, I’ve worked on several projects
that go from analyzing lumber scanning, for example, to satellite imagery, to detect a
deforestation, or now surgical videos. So it’s been quite a ride for sure. And I’ve learned
a lot mostly on how to producturize AI and how we can use AI to make an impact in the world and
have a focus on is it actually going to be useful, you know, and make a difference at some point.
So that’s kind of how I see my career so far.
Have you found that the projects you’ve been drawn to, has it been kind of being drawn to the
next sort of technical or scientific challenge, kind of pursuing the craft and kind of pushing
the boundaries of what computer vision and image and video analysis can do? Or have you been driven
more by the mission of these different projects or have you just kind of worked out that you sort
of been able to follow both of your blessings, so to speak?
I would say both actually, yes. Slightly luckily, always.
I mean, I’ve always been passionate about different sciences. So I actually have a
hard time focusing on only one thing or one project. And when I learned about climate tech,
for example, I really wanted to see how I could help and how AI could help process all of this
giant xactolite and imagery, you know, remote sensing is really hard to process.
And then I learned about surgical videos and I learned that there were thousands of terabytes
of surgical videos that were not used. And it’s a pretty big challenge because surgical videos are
really heavy, really long, you can imagine eight hours procedure. No one really wants to watch
those videos. So that’s when I was thinking that AI is indeed the perfect tool for this kind of data
that is really being really long, but also has a temporal element to it, which is quite difficult.
And when I learned about that, it was a really interesting impactful project, but also in terms
of technical challenges, I thought it was really interesting. And that’s actually what made me
join SDSC in the first place. So how did you find out what drew you into the
troubles of surgical videos? I started to work at the surgical data science collective SDSC
when I met the founder, who is a pediatric neurosurgeon, Dr. Denoho. And he introduced me
to this issue, you know, he was telling me I have all these videos saved in drives and I don’t do
anything with them. And I know any of my coworkers and friends and other surgeons have drives with
hours and hours of surgical videos, and they are just sitting on the desk, not really doing anything
with them. Right. Just a context set for the audience, for me too. Is it standard procedure that
surgeries are just all video to these cameras that are inside of people, sort of, you know,
guiding the surgeons? Or are they sort of operating room overhead cameras? Or how does this, how does
it work surgical video? That’s a great question. It’s pretty diverse. We have a lot of endoscopic
videos and microscopic videos. So endoscopic will go, for example, through the nose or any
other part of the body where you need to see inside. And actually, the endoscopic videos are
a really good data point for us because we see the computer vision algorithm sees what the surgeon
sees. Right. And that is, you know, golden because there is a lot of information in these videos.
For the microscopic videos, it is also used by the surgeons sometimes when they do the surgery to
really magnify what they’re looking at. For example, if they’re operating on very small arteries,
they need to have this big intense zone. That’s what they’re going to be using. Often it seems 3D
for them, which we have some of those 3D videos as well. So yeah, it’s a mix of microscopic surgical
videos and doscopic videos. And we don’t have yet the video, you know, kind of like camera security
from the OR, but some of our collaborators use this kind of videos to get a sense of what is
happening in the operating room. Right. And so you met the founder. So the surgical data science
collective was already in existence and you met the founder and got involved? Yes. It was pretty
early on. So the surgical data science collective is a non-profit organization that was started by
Dr. Daniel Ho and the main mission and the main idea was to create and analyze a repository of
surgical videos in order to improve surgical techniques and patient outcomes. Because still
today, there are 5 billion people who lack access to safe surgery and there are at least 4.2 million
people around the world who die within 30 days of surgery. So if we consider surgery as a disease,
it would be the third leading cause of death. That’s why, you know, the goal of SDSC is to
utilize this annual surgical videos to identify best practices, support medical education,
or even predict potential outcomes and complications in advance of surgery.
Right. So we’ve had other healthcare practitioners and people from the health industry on the podcast
talking about ripping the cums to mind is talking about analyzing still images, you know, x-rays
and scams and using AI to discover things often kind of, as you said, at that super zoomed in level
that, you know, cancer prediction and that kind of thing. Tell us about some of the opportunities
and challenges involved with analyzing all of the surgical video. I would assume, you know,
the first challenge is just gathering the data and then processing all of it. But
kind of take us through it. What is it that, you know, you said improving best practices and real
time feedback. So maybe you can speak to that as well. Yes. So the first challenge, like you said,
is actually to gather all of this data. And like I said earlier, a lot of these surgical
videos are stored on drives and it’s really difficult to get access to these drives. Sometimes,
you know, you kind of have to go and fly somewhere and meet with the surgeons to be able to get the
videos. Most of the time, the videos are not even recorded because people don’t know what
they can be used for. So why would they record them? So actually, one of the biggest challenge
is asking people to press the record button. Right. I mean, I’m laughing, but I’m imagining,
you know, if I was a surgeon, that’d probably be the last thing on my mind, right? So yeah.
Exactly. Yeah. I mean, that is definitely not the priority. And then if they think about recording,
so pressing this button, they have to export the video from the device, walk around with a USB key,
upload the videos on a laptop, upload to the cloud. So there are so many steps here for these people
who are extremely busy, who have so many other important things to do of their day. It’s definitely
not a priority. So that is our first challenge, and that has been one of the biggest challenges
that we’ve had. But we’ve been pretty successful in gathering at least a good first base of a
surgical video library. By now, we are about 40 terabytes of surgical videos. Okay. And we expect
to get more, you know, and the other challenge here is to get diverse surgical videos. We don’t want
obviously for an AI model, we don’t want videos from one surgeon in one hospital doing the same
procedure. Hundreds of hours of tonsillectomies is only going to get you so far, I’d imagine.
Yes, exactly. So that is the other challenge is how do we get these videos from diverse sources
and diverse fields, which is also a lot of networking, because you have to go and talk
to the people and ask them to record and then do they want to work with us so that we can start
gathering these videos. So this is the other second part of this challenge of data collection.
But in terms of the other challenges, obviously, surgical videos are quite long, but they are also
temporal. So videos, right? So it is a different type of models that you would use for still images.
We have the kind of the same architecture that you would use for other models,
but we always have to think about the temporality of what is happening in the video and that is
actually how we implement most of our models. Let’s say if you’re trained to track surgical
tools, you know, you have to think about all of the challenge that comes with that in surgical
videos, which are going to be obstructions and can sometimes you have, you know, explosion of blood
or something like that. And you want to be able to subtract the tools without losing them or
with dealing with these problems, which are pretty similar to other computer vision problems.
But it is slightly more challenging because of how messy these environments are.
Sure, I can only imagine. And so from a technical perspective, you know, obviously there are,
we’re hearing all the time these days about video models in the news kind of being opened up for
you know, consumer use, that kind of thing. You’ve been working with the Data Science Collective
for going on two years now. Is that right? Yes. So are you using, are you are you building tools
yourself? Are you using off the shelf to kind of modifying them to suit? Do you have partnerships
with other AI labs? How are you kind of fine tuning the tools to get what you need out of them?
A mix of all of what you just said, we usually will. So, you know, we’re a pretty small team and
a nonprofit organization. So we will try to use the most efficient methods for us. A lot of the
times that we’ll be reusing some architectures that are existing and then fine tuning them to our
needs. Combining some architectures is something that we’ve done a lot, especially with the temporal
model. So having, you know, a mix of a CNN and a temporal architecture or we’ve been playing that
with vision transformers and more recently with vision text transformers, which are the big models
you’re talking about here. And a lot of the time we will always be careful about new technologies.
So we want to try them and we want to make sure that we stay on, you know, on top of the innovation
that is happening to see if we can apply it to the surgical data science field. Something that is
quite interesting and challenging with this kind of data is that it requires a lot of expertise
that as computer scientists, engineers, we don’t have. So we need to work very closely
with clinicians and surgical experts. And that’s where the most important part of the work is
happening, actually, not even the model architecture or the new cool AI tools. For us, it’s really
understanding what the expertise is and then what model should apply to bring that information that
will be useful to the surgeons. Can you maybe walk us through an example of, and correct me if I’m
wrong here, but I imagine you have partnerships with surgeons and other medical professionals
and institutions and are you sending them images or videos to just kind of analyze and let you know
what they see or kind of what’s the, I guess I’m wondering what the process is or what it’s like
getting from footage and people revealing the footage to then an outcome that other practitioners
can benefit from, whether it’s, you know, a new technique or refining a best practice or something
like that. So for most of our collaborations, we will work with clinicians and surgeons who have
videos, but they don’t have the computer science knowledge. So they will come to us to do all of
the computer vision and analysis. So when they start working with us on these projects, maybe I
can go through a concrete example. We’ve been working with several NGOs who are focused on
surgical education. One of them is called All Safe and they focus on teaching surgical procedures to
several students all across low income countries and they do it through a digital platform. So
it’s online courses and then the review is done through videos. And so what we’re trying to see
here is can we analyze these videos and give feedback to the students with computer vision
and that is useful. So that is the important point is that is useful. So then we will work on
in developing the computer vision models to extract the features that we need to be
extracted to do the analysis and then collaborate with the clinicians and the surgeons on what
exactly do they need to have in the feedback or what do they believe is something we should focus
on because most of the time, you know, we’re going to look at something and I’m going to think with
my engineer mind. Oh, I’m going to look at this feature and I’m going to make this graph and
it’s going to be amazing. And then I say the surgeons and they’re like, what? So that’s why
it’s called the social data science collective because it’s the first step is creating a community
with the clinicians and the computer science experts. And we also have some collaboration
with computer scientists groups. So where we will work with them to analyze some of the videos we
have. So that is almost a connection between we have surgeons who want to do something very specific
and we have computer scientists collaborators who can help us do that specific task that maybe we
don’t have bandwidth for. So we are trying to expand that part of our community as well to really,
you know, have a really impact and scale that because it’s not only going to be a whole community
effort. I’m speaking with Marder Mason Forsyth. Marder is the director of machine learning at
the Surgical Data Science Collective, a nonprofit that is using AI machine learning tools to analyze
video data from surgeries to develop tools and feedback loops and other mechanisms that can
help surgeons with insights and feedback on their procedures and techniques and really just
bring better health care to more people across the globe. As Margot was just talking about,
you mentioned, you know, being a nonprofit doing AI research is a little bit unusual right now.
What is that like? Are there big things that are either, you know, well, I mean, I’ll be
saying I would imagine something. Resources is an issue as it is for almost all nonprofits. But
there are things, are there things specific to being a nonprofit AI kind of research group
that stick out to you? So there are many interesting aspect that come from being a nonprofit. Like you
said, resources are indeed limited. So we have to be creative in the way we train computer vision
models. We will always start simple, which is actually something I’ve always done and advocated
for is if you want to start a computer vision project, maybe you don’t need to start with the
biggest model that exists. You know, start simple with a small data set, do a proof of concept,
and then iterate. So that is what we have as a development pipeline and research pipeline
process. We will always start simple and small and then scale. And that is limited because of
our resources, obviously. But to me, it’s something that is actually good that I would even, I would
do the same if I had, you know, 10x budget. I would probably do the same, but it helps in that way.
And then it brings a lot of different projects. Being a nonprofit, we’re able to work on a project
that maybe we wouldn’t be able to work on if we were a for-profit. For sure it would be, it would
actually be completely different. And that’s why I really wanted to give SDSC a shot when I first
met Dr. De Noho because I was curious about it. I was like, how are we gonna do that? You know,
I’ve never done AI. I’ve never seen AI done in a nonprofit. There are some others, but it’s
really research focused and community focused, which you wouldn’t really be able to do as well
in a for-profit, I believe. We’re gonna ask this an answer, please, based on what’s happened so far
and/or, you know, what you see coming in the near future. What are some of the big benefits
for clinicians, for patients that you’ve seen or expect to see from, you know, not just the work
that you’re doing at the collective, but more broadly leveraging AI to help with the surgical
process? The AI field will bring a lot of new and good things to the medical field, I believe.
In the surgical space, which is what I’ve been exposed to mostly, it will bring a lot of standardization.
I believe something I’ve discovered working in that field is that every surgeon and every hospital
will perform surgeries and procedures in different ways, and no one really knows
ABCD of how you’re supposed to do a specific procedure. So by having a tool here, AI, to first
encourage people to collect the data. So first, we’re gonna get the surgical videos, we’re gonna
finally start looking at these videos that are not being looked at, and then share them between
surgeons all across the globe that will bring a lot of standardization, or at least they will
start to talk to each other, which I think is kind of beautiful, because right now, you don’t really
have a good way to talk to each other. And through the surgical videos, the hope is that they will
start to talk to each other. And when you can imagine so many applications, for example, the one
that always comes back is education. Instead of using a medical textbook with drawings, the students
can watch a tutorial on how to do the specific procedure. So that’s a big difference that I
think will change a lot of things. And then being able to find best practices through this
analysis of surgical videos is gonna be pretty interesting, because who knows what is in these
videos. And there’s so much that has to be discovered, and there is a big need to be creative
when we think about this data, because no one has ever looked at this data, and no one has ever
really thought about what can we do with all of that, and what is my question that I want to be
answered. And that’s one of our challenges, actually, is sometimes we ask surgeons, “Oh,
what do you want to answer through all of these videos that you have?” And they don’t really know,
because they haven’t had this option before. Right. That’s interesting. It makes me think of
that. I mentioned there were some of the examples of MRI and scan analysis and cardiac care and that
kind of thing. And I’m thinking about the AI tools being able to help practitioners find differences
in cells on a very, very sort of nano basis, right? But even with that, I’m thinking, “Oh,
well, they know what they’re looking for.” Or even if it’s they’re looking for an anomaly,
it’s still kind of we know what we’re looking for. But yeah, with surgery, my very kind of naive,
not knowing much about the field coming in this conversation thinking, “Oh, well, video footage
is being used to train AI systems. Are we moving towards better education for humans or even training
robotic surgical algorithms or that kind of thing?” But it’s fascinating to hear people say that it
makes sense to me as a non-surgeon that what would they be looking for? It’s not the same as looking
for an anomaly in a cell that might stick out. I think at the beginning probably of the when
they first started to analyze MRI with AI, they also had to be creative because someone had to be
asking for these questions for it. And for surgical videos, one of the first steps would be to look
at anomalies, which actually was trying to do now is what are the outliers? Who is using this tool
and no one else is using it for the same procedure? So we are kind of starting with the low hanging
foods, I guess, but the deeper existential questions are not there yet. And I’m really excited to work
with the clinicians to help them come up with these questions by showing them the data because
no one else is going to come up with these questions. It has to be the people who are working
every day in the OR. And actually, the videos are a really great source of data, but there is so much
more going on. Obviously, there is the patient data, there’s the patient outcomes, there is
everything that is going on in the operating room. And all our engineers have actually been
in the operating room so that they understand what is happening behind that camera. And I’ve
been in the operating room to a couple of times now. And it’s really helped me understand better
what is happening. And sometimes when we have a new procedure type that we’re exposed to,
I go to the OR because I want to understand better like, oh, some random questions sometimes
like, where are you? Where is it in the body? Or how many people are operating? Because sometimes
you have more than one surgeon. It’s just so many things that you don’t capture in the video,
but there’s still obviously a lot of information in the videos.
Fantastic. I go for listeners who would like to learn more or hopefully perhaps there’s even
some surgeons, some clinicians listening who are thinking, oh, I have surgical video that, you know,
in a shelf on a drive somewhere, maybe I can send it and how about the cause? Where can listeners go
to find out more about the work that the Surgical Data Science Collective was doing,
the work that you were doing perhaps to get involved as a partner? Who knows? Where can listeners go
to learn more? So we have our website is thesurgicalvideo.io and we can also find us on social media
at Surgical Data Science Collective. And I would also encourage if anyone is a computer engineer,
computer scientist who wants to work on a different project that they’ve been working on
and are interested in surgical AI to also reach out to us because we are working with quite a lot
of different parts in this. So anyone who is interested should reach out to us.
Fantastic. Well, Margot, thank you so much for taking the time to stop by, join the podcast
and talk a lot about the work you’re doing. It’s, I don’t know, stories like this with the technical
aspects kind of match up with the societal impact. I think they’re just fantastic stories.
There’s sort of something for everybody, right? And it sounds like you’re finding a really interesting
path to fuse your technical interests with making an impact in your own work. So congratulations
and all the best developed to you and all of your partners and cohorts at the collective.
Well, thank you, Noah. Thanks for having me on the podcast. It was really enjoyed the conversation.
Me too, our pleasure.
[Music]
[Music]
[Music]
[Music]
[Music]
[Music]
[BLANK_AUDIO]
Margaux Masson-Forsythe, director of machine learning at the Surgical Data Science Collective (SDSC), discusses how AI-driven video analysis is transforming surgical training and practice, making surgery safer and more accessible to billions of people worldwide.
