Ryan James, Ph.D., center, and physicians Stephen Seslar, left, and Wayne Monsky, cofounders of Pyrus Medical, are integrating virtual reality technology with a specialized robot to make remote surgery available in rural areas and beyond.
Tom Alberg is seated at a boardroom table, relaxed and dressed casually in grey slacks, a pale blue shirt and a knit sweater vest. His face is expressionless, like that of a master poker player. Alberg is a soft-spoken man and deep thinker who has a legacy of picking big winners in the technology space. When he talks tech, it’s best to listen and learn.
Alberg, co-founder and managing director of Seattle-based venture capital company Madrona Venture Group, recounts the day some eight years ago when he and one of his Madrona partners, Matt McIlwain, entertained a wager about the future impact of virtual reality and autonomous vehicles. He says they met with a number of companies focused on autonomous vehicles and 20 or so working in the virtual reality space, and Madrona ultimately made a few investments.
“‘OK,’ we said, ‘let’s make a bet on which technology is going to have the biggest impact on people 20 years from now,’” Alberg recalls suggesting to McIlwain, adding that the duo never implemented the bet because, “partly, we couldn’t decide how to test it.” Still, Alberg says both he and McIlwain thought virtual reality would be booming within 20 years, “with all kinds of companies and money made, and people using it.”
“But, you know, it [virtual reality] hasn’t taken off yet,” Alberg adds. “Sometimes technologies take longer than you expect, and I think part of it is that there are technical challenges. I think virtual reality will create great new experiences, but the equipment hasn’t come along fast enough.”
Alberg is right about the fact that virtual reality technology has not yet achieved broad adoption, particularly in the consumer market, according to industry observers. Still, in the eight years since Alberg entertained the bet over the future of virtual reality technology, important progress has been made in conquering the technical challenges, multiple experts in the field say. In fact, that progress has moved fast enough to lead even major companies like Seattle-based Getty Images, which creates, licenses and distributes multimedia products globally, to explore adding the technology to its product mix.
“We try to see, OK, what if the future goes toward a world where there are these more visually rich, more engaging experiences through potentially new devices, then how do we make sure that our content is still going to be relevant to that,” says Getty Chief Executive Officer Craig Peters. “So, we’re clearly thinking about augmented reality, virtual reality and things along those lines.”
Woody Floyd, managing director of the Seattle office of Vectorform, a digital-product consulting and development-services firm, says broad consumer and even workplace adoption of virtual reality, augmented reality and related technology — referred to as immersive technology — is not yet on the near horizon. He does predict, however, that “within the next 10 years, we’re going to see a few key industries that go very deep into it, and within five to 10 years, there will clearly be some early adopters.”
Nowhere is that more the case than in the health care field, particularly in Seattle, where a number of startup companies are applying enhanced-reality technology to address some of the most pressing issues in health care delivery and quality. These new immersive-technology tools, including virtual reality, when combined with robotics, machine-learning/artificial intelligence and cloud technology, are opening up a brave new world of possibilities for the health care profession. They are making it possible for cutting-edge surgery to be performed remotely by the best surgeons in the world, so that patients needing lifesaving procedures no longer face access problems simply because they are located far away from urban centers or in developing nations.
Floyd adds that Seattle is playing a critical role in propelling virtual reality and related technology. He points to the enhanced-reality platforms being developed in the Seattle area by product teams employed by major tech players such as Taiwan-based HTC Corp., Facebook and Microsoft — referring to their Vive, Oculus and HoloLens virtual reality and “mixed reality” headsets, respectively.
As evidence of Floyd’s assessment, a 2018 Harvard Business Review study involving a survey of 394 large-company executives found that 87% were currently exploring, piloting or deploying virtual or related immersive technology, and 68% believed the technology will be “important to achieving their company’s strategic goals.” Marketintelligence firm CBInsights reports that disclosed investments in the virtual and augmented reality market totaled more than $2.3 billion in 2017, up 21% year over year. A Goldman Sachs assessment projects that within five years, the market for virtual reality (VR) and augmented reality (AR) software alone could reach $35 billion, including more than $5 billion focused on health care.
Virtual reality is part of a spectrum of enhanced-reality technology. It is best described as a completely digitized world experienced through total immersion in that landscape — where the laws of physics can be completely upended or suspended. On the other end of the spectrum is augmented reality, which allows you to see and interact with semitransparent, digitized objects that are placed on the “canvas” of the real world.
Finally, there is mixed reality, which is in the middle of virtual and augmented reality, and that involves digitizing the real world in near real time — normally through the use of cameras — and then infusing that digitized real world with additional objects or data. Each of these cyber-driven enhanced states of reality, accessed via tech hardware like Microsoft’s HoloLens or Facebook’s Oculus headsets, among others, are sometimes generically referred to as extended reality (XR).
Based on Goldman Sachs’ projections, the health care industry is clearly one of the early adopters of XR technology. In fact, two of the largest players in the tech arena, Microsoft and Facebook, are each involved in partnerships that advance the use of their respective XR platforms in the health care space.
Facebook is currently in a partnership with the Johnson & Johnson Institute to deploy its Oculus Quest headset around the world in an effort to provide “more frequent, higherquality training to improve patient outcomes and to help save lives,” a Facebook spokesperson says. Microsoft is working with Philips Healthcare to integrate its HoloLens 2 headset into Philips’ Azurion image-guided therapy system, which allows physicians to work in a holographic augmented-reality environment that they control while performing minimally invasive surgical procedures.
“Among the things we learned with the original Microsoft HoloLens is the commercial value this device delivers to customers in a variety of industries and use cases,” a Microsoft spokesperson says. “HoloLens 2 enables a new set of solutions for customers in a range of industries — including health care, manufacturing, retail, architecture, engineering and construction, education and the public sector — to achieve more.” Professor Ira Kemelmacher-Shlizerman is the director of the University of Washington’s Reality Lab, which is an academic research center focused on XR technology and backed by major tech players such as
Google, Facebook and Futurewei — the United States research arm of China’s Huawei Technologies. Kemelmacher-Shlizerman says the future of the XR field is bright, though she concurs that there are still some big challenges to overcome. Those challenges include creating more and better high-resolution 3D content that can be rendered well in head-mounted XR devices as well as building new applications that can leverage the medium well.
Another major challenge is the hardware. “Most applications require high-resolution, high-comfort, wide-field of view, [and need to be] light-weight and low-cost,” Kemelmacher-Shlizerman says. “All those together are hard to achieve today. This is where there is significant investment from companies.”
One area that holds out great promise for the use of XR technology, Kemelmacher-Shlizerman adds, is the health care market. “We envision lots of potential in areas such as training doctors, human-body visualization, surgical planning and others,” she says.
GPS sensors are embedded in a long, flexible catheter that looks like a slender snake. The GPS sensors send signals to computer processors capable of rendering a three-dimensional image of the interior of a patient’s femoral vein, allowing the surgeon to guide the catheter with a wand-like controller through the patient’s circulatory system and into the heart.
Through the lenses of an Oculus Quest virtual reality headset, the interior contours of a digitized human heart appear in a bright red field of color. As the surgeon peers into that three-dimensional image of the heart, seemingly suspended within one of the heart’s chambers, the serpent-like catheter appears. It begins probing one of the heart’s four chambers, with its movements orchestrated by a sophisticated surgical robot being operated remotely by the controller device in the surgeon’s hand.
“They push the catheter through the vein until it reaches your heart and then they move that catheter around inside of your heart,” explains Ryan James, co-founder and chief technology officer of Pyrus Medical, a Seattle startup company that applies XR technology to enable remote, robotically controlled surgical procedures — specifically a procedure called cardiac ablation, which involves burning, or destroying, targeted heart tissue that is promoting abnormal heart rhythms. “When they find the part of your heart that’s generating bad electrical signals, they burn it by turning on energy on the tip of the catheter,” James explains.
That’s right. This technology already exists and is being eyed for use in remote areas where the surgical expertise and expensive operating-room equipment required to perform cardiac ablation procedures are not readily available, such as in rural areas of the United States or in developing nations. James is developing the virtual reality interface (or operating interface) for the surgical system as well as the cloud service that will transfer the digitized information between the remote surgical site housing the robot and the local site where the surgeon is plugged into the virtual reality headset.
The Pyrus team also includes co-founders Dr. Stephen Seslar, a cardiologist and electrophysiologist who is a professor in the Department of Cardiology at the University of Washington; and Seslar’s colleague, Dr. Wayne Monsky, an expert in interventional radiology and professor of radiology at UW. Pyrus is employing an existing surgical robot already approved for use by federal regulators and designed to perform cardiac ablation procedures with minimal human assistance — and developing virtual reality technology for use with that robot.
“The whole world can basically be in there [inside the virtual reality headset] and it’s put in the palm of your hand,” Seslar says. “From a portability standpoint, and from an access-to-care standpoint, it means these procedures can be done in much simpler environments, including smaller community hospitals.”
Seslar says Pyrus is already meeting with officials from the Food and Drug Administration (FDA), which regulates the safety of all medical devices and products. He adds that the company hopes to begin using the virtual reality-powered surgical technology in the first human cases as early as next year — depending on the speed of the FDA approval process.
“There are not enough patients at every hospital to support the skills needed to do these kind of [complicated surgical] procedures,” Seslar says. “And so I think we need to find another way of getting the care there. And remote robotics [enabled via virtual reality technology], I think provides that opportunity, if we do it safely and correctly.”
Seslar adds that Pyrus will eventually need to seek investment funding, but for now the founders are bootstrapping the startup and exploring various grant options to “convert what is essentially a prototype development into a commercializable product.” Pyrus is currently a participant in the University of Washington’s CoMotion Labs, which is an incubator/accelerator program for promising startups, including those seeking to commercialize XR technology— in many cases being developed by UW researchers and students.
Xuny Haley, a CoMotion Labs manager, says since the program began assisting startups focused on XR technology in 2016, it has helped more than 40 companies, “some of which were with us for a few months and others have been with us since we opened. I’d estimate just over 20 of those companies are still active.”
Exonicus Chairman Sandis Kondrats says his company is now field testing its traumasimulation virtual-reality training software, which is designed for use by the military. Courtesy of Exonicus
Besides Pyrus, another of the CoMotion Labs companies still going strong and focused on the health care field is Exonicus, which is developing trauma-training virtual reality software for military applications to improve decision-making and readiness in the field of combat — with plans to expand its use to civilian applications in the future. The company, according to co-founder and Chairman Sandis Kondrats, has raised some $700,000 to date via contracts to pilot its virtual reality trauma-simulation software, which can be used with or without an internet connection and is compatible with off-the-shelf headset viewers, like Facebook’s Oculus and Microsoft’s HoloLens.
Kondrats says Exonicus is currently working with the Madigan Army Medical Center, located on Joint Base Lewis-McChord near Tacoma, as well as with the Latvian Defense Ministry. He adds that the company also is in discussions with “European militaries” in Germany, France, Estonia, Finland and Spain.
“Our self-training system has already been deployed to Iraq and in Africa in a few cases,” Kondrats says. “With our system, an avatar is built into the [virtual reality] simulation, and basically it pulls the user into a scenario where they have to save a patient’s life, and that [teaches them] how to react in a similar situation [in real life].”
Exonicus’ virtual-reality training software has already been deployed for pilot tests in Iraq and parts of Africa and is focused on preparing military personnel for handling battlefield trauma cases. Courtesy of Exonicus
Seattle is well-positioned to become an epicenter globally for developing and commercializing XR technology, particularly in the health care field, given its booming tech economy and its world-class health care and research institutions, according to Gabriel Jones, co-founder and chief executive officer of Proprio Vision. The Seattle-based med-tech company is applying XR technology to the operating room in a way that Jones says provides surgeons with “superpowers” in the form of “X-ray vision without the X-rays.”
“We’ve got a unique opportunity to build one of the global centers for excellence in immersive technology in Seattle. We need everybody on board,” says Jones, who previously served as a technology and investment adviser to Microsoft founder Bill Gates. “We need investors. We need corporate partners. We need technologists. We need undergraduate and grad students to get on board with this.
“It will take all of us pushing together,” he adds. “We have the raw materials and talent here that should allow Seattle to occupy one of the top positions in the world in augmented and virtual reality technology.”
Jones keeps the details of the technology his company is currently developing in the XR field close to the vest, but he provides enough of a glimpse of the med-tech frontiers the company is exploring to make it clear that, if successful to any degree, it has the potential to revolutionize the way surgeries are performed in the future. Proprio, founded in 2016, has raised some $7 million in funding to date and has a developed a deep bench of potential future investors. It is backed by major companies and investors like Intel, HTC, Bold Capital Partners, The Venture Reality Fund and health care investor Alan Frazier of Frazier Healthcare Partners.
“We’ve invented a proprietary technology that is the next-generation imaging system that enables immersive visualization and real-time robotic navigation in the operating room,” Jones says. “The initial-use cases are in orthopedic and brain surgery, but it’s actually going to be impactful for all of surgery.”
Jones explains that the company’s technology employs a system of multiple sensors and cameras of various types that make it possible for “professionals in the operating room to effectively capture all of the data in the operating field, and once you have done that, once you have this immersive video that you can actually fly through, you can have an infinite number of users looking from an infinite number of perspectives.”
Proprio Vision co-founder and CEO Gabriel Jones says his company is developing a next-generation imaging system that integrates extended reality technology, pre-operative data, machine learning and real-time imaging capabilities to essentially give surgeons “superpowers” in the operating room. Photograph by Hayley Young
Jones says Proprio is already in conversations with FDA regulators, adding that because the company’s technology is primarily a visualization tool, the “path for [FDA] approval is actually fairly straight-forward.” He adds, however, that normally time to market is about 10 years for a complicated technology like Proprio’s XR-enabled surgical-imaging system, which is capable of integrating pre-operative diagnostic imaging and data with real-time imaging capabilities and harnessing machine learning to augment surgical navigation. “We started about three years ago, and we’re well ahead of that [10-year] time frame,” Jones says. “The software, the algorithms, the calculations, the sensors, the CPU and the GPU computing, all the pieces — there’s a lot to it — that are required [for Proprio’s system to work] wouldn’t have been possible even five years ago.”
Jones adds that a majority of the hospitals in the world can’t afford the millions of dollars’ worth of surgical navigation equipment currently needed to perform the most complex surgeries, and it’s a particularly vexing issue for rural areas and developing nations. “So, either those surgeries just don’t get done and the patient suffers or dies, or they try to make do with what they have, and you have that outcome,” Jones says.
“Our vision is that any functioning hospital in the world should be able to connect to a global network of surgical expertise and use one of our systems to effectively perform at near the same level that a very sophisticated hospital would be able to,” he adds. “That’s a multiyear vision, but just think about the impact.”
The long journey to where we are today, on the cusp of a global revolution in remote surgery and telemedicine enabled by XR technology, actually started with another bold dream being pursued by NASA in the 1970s, according to Pyrus’ Seslar. He says scientist with NASA, as part of the quest to land the first human being on the moon, were “contemplating the idea of long space missions” and wondering what they would do if an astronaut “got appendicitis as they were flying past the moon.” It was not possible to do remote surgery across such vast distances then, and it’s still not feasible today, but “what came out of that then was a real interest” in developing technology to accomplish that goal, Seslar notes.
“NASA started out with the idea that they were going to operate on people in space,” and now, robotics and XR technology have advanced greatly, the internet is here, “and so,” Seslar says, “we’re trying to bring [that dream] back to life again. We’re part of a larger telehealth movement that is breaking down these barriers ahead of us.” What the future will look like as XR technology continues to advance is still anybody’s guess. But solutions now appear to be within sight for conquering some of the biggest technical challenges facing XR technology, which Madrona’s Alberg points out have been impeding the technology’s broader adoption.
“If it was just one company, I would be very skeptical of their ability to execute on all of that is required,” Jones says. “But the fact is there are at least 10 massive technology companies, including the largest ones in the world — Google, Amazon, Apple Facebook and Microsoft — who are all very much continuing to invest in this space, despite what they say in public at times.
“And they are doing so because they know [XR technology] is going to be the most dominant computing and visualization-interaction paradigm in at least a generation.”