When Ivan Sutherland introduced his “ultimate display” in 1965, he thought about a world existing of real and virtual objects that is presented to the observer through his natural perspective, his eyes. Most of todays Augmented Reality solutions still use hardware interfaces that do not follow this most natural form of immersion, e.g. augmented camera views of tablet PCs and smartphones. Anyhow, in the last few years more and more head worn AR interfaces have been created and released. SDKs related to such head mounted displays have inspired thousands of developers to create AR worlds for instance to enhance industrial tasks or make cultural experiences more interactive and appealing.

However, it is extremely difficult to introduce these devices to the medical world, in particular to intra-operative tasks that require high quality standards of medical products. One could go all the way to certify a device, such as Hololens and comparable hardware, as a medical product. This is certainly a commendable and important approach, which will happen once the benefit for patient treatment has been proven and usability and robustness has reached a level to become accepted in a high performance working environment such as operating theaters.

Another approach is the usage of existing medical products as a platform, which is extended by Augmented Reality functionality, e.g. endoscopes or microscopes. Endoscopic cameras are highly used for many types of minimally invasive surgeries to minimize tissue damages and reach anatomical areas, which are difficult to be accessed in open procedures. Furthermore, reality is captured as digital video data. For this reason, the Augmented Reality scene benefits from all advantages of a video see-through (vs. optical see-through) approach, having been discussed widely in the literature. This includes for instances the synchronization of real and virtual objects ensuring a geometrically correct overlay in any situation. There has been an interview with the company SCOPIS, which has been published some time ago on this blog. SCOPIS is a good example of augmenting the endoscopic video data with 3D planning information, registered with the 3D intraoperative anatomy of the patient.

Beside endoscope cameras also operating microscopes are well established in today’s operating theatres e.g. in the field of neurosurgery. Microscopes are getting even closer to Ivan Sutherland’s vision of an “ultimate display”, presenting the Augmented Reality scene from the user’s natural perspective.Wolfgang Birkfellner and Eddie Edwards have introcudes groundbreaking research in this application field some years ago. However, microscopes are today still pure analog, optical devices, which means that surgeons see a magnified situs through a set of optical lenses. In fact, medical device companies such as Brainlab have started integrating Augmented Reality supported navigation systems into these optical see-through devices.

In order to take full advantage of augmented microscopic views a video see-through device would be a much better platform for augmented reality based navigation software (e.g. registration, synchronization, image composition).
Recently, the company ARRI has released the first video see-through operating microscope. The digitalization of the real world with high quality stereo cameras and optics in combination with surgical applications being characterized by complex anatomical structures, availability of 3D imaging data (CT and MRI), preoperative planning procedures can be a perfect match to develop Augmented Reality software solutions that bring a real benefit for patient treatment.