Didactics in augmented reality at the University of Padua
In just two weeks, we created an application that medical students at University of Padua can use to observe and analyse a 3D model of a heart through augmented reality.
The challenge was not only to create models faithful to reality, it was also fundamental to develop an app infrastructure that could hold all the information needed through hierarchy and a User Interface able to manage great complexity
Augmented reality training
The cognitive process, i.e. the process of acquiring knowledge and understanding, in human beings occurs through thought, senses and experience.
Up to 60% of this process depends on the sense of sight and hearing, the remaining 40% is attributed to the practical or abstract application of what is stored.
A study carried out by the National Laboratory for Education in the US states that rates of mnestic retention (i.e. the ability to remember what you have experienced) increase to 75% when referring to the use of augmented reality within the educational process.
If, in addition to the above, we add that specialised training in the medical field, it involves added risks for the safety of students and teachers. When they find organic material and places to study it represents a considerable cost as well. So we can immediately see the potential of augmented reality in promoting safe and sustainable learning.
From requirements to information hierarchy
With the collaboration of Prof. Vladimiro Vida, Director of the UOC of Paediatric Cardiac Surgery and Congenital Heart Diseases and with the student of Medicine Francesco Bertelli, we analysed the usual learning path and the tools commonly used. The solution adopted so far has been the 3D printing of organs of the human body made by a software that uses scans obtained by means of CT, ECO and magnetic resonance imaging. These 3D models allow students to touch, see and dissect organs in the same way as real ones.
The need was therefore to obtain models as close as possible to real organs that could be fully analysed and dissected as if they were in the hands of a student. It was therefore vital not only to have the 3D model but also to conceive ways of interacting with it, all within a complex range of hearts with different shapes and different pathologies.
What can augmented reality offer more than a 3D printed model or even a real organ?
The positive impact, as explained above, has reduced risks and costs, but this technology also allows you to have a huge amount of material, models and information within a simple smartphone. It is not necessary to make, transport and store physical objects, moreover during the analysis more information can be requested of the system that, through a careful study of the UI, makes them appear on virtual models.
Designing the right technology
An accurate analysis of the most suitable technology has been carried out to reach, first of all, a mobile app able to host 3D models with related information and, subsequently, the hearts in augmented reality.
An initial MVP allowed us to submit the results of our study to the University of Padua. The findings on interfaces, navigation flow and digital asset yield were then endorsed and formed the basis for the development of a complete and structured app.
Therefore, all the screens were designed to allow users to access the multiple functionalities through a menu able to manage a large amount of information in a very intuitive way.
Augmented reality and 3D models: the heart of the project
At the same time, 3D models of organ cavities have been optimised for use in real-time. Starting from assets already in possession of the University of Padua and giving them a photorealistic appearance as the result of applying high quality textures from the photogrammetry technique.
Augmented reality hearts can then be dissected and enriched with information to provide a complete experience for students and without compromising the quality of teaching. In addition, to maintain the flow of normal teaching, the virtual dissection of the models follows the process carried out during echocardiographic analysis: Long axis parasternal (PLAX), Short axis parasternal (PSSAX) and 4 chamber apical (A4C).
These models allow a precise morphological and functional analysis of the organ and of each of its parts.
To highlight areas of the heart with defects using colour, we had to create the UV Maps, which were not present in the scans. To do this, we painted directly in 3D, making it intuitive and easy to perform. The UV Maps also made it possible to effectively and correctly distribute the textures on the 3D models.
The development team had to manage many models in a short time, consequently the latest generation UV unwrapper was used, which reduces manual work and shortens development time.
In addition, we decided to select only one set of textures for all hearts, so that the mobile application would remain light and efficient (less than 60MB) while maintaining a 4K resolution.
A further challenge was to enrich the 3D models, both used through a 3D viewer and displayed in augmented reality, with the information in written form. This text had to appear in the least invasive way possible but, at the same time, it had to be easily readable.
The result of all this is a non-linear user experience. Each student can interact in a different way. They can use different models with related sections, without a set path for each user but rather fostering different experiences adapted to the needs of all users. A training that is in some ways personalised and, therefore, more effective while managing and not increasing costs.
The output is therefore a mobile app for iOS and Android operating systems that uses ARKit and ARCore for real-time 3D visualization of the environment around us. Simple 3D visualization is also provided for older devices that do not support augmented reality.
The next developments include the possibility to use WebAR technology to enjoy the models without the need to download an app, minimising additional effort.
“Real-time rendering technologies are the perfect solution for this project, making it all really exciting. Learn using a software that is alive and interactive directly from your smartphone. It demonstrates how the technologies created for video games are now fundamental even in completely different areas. Access to such a large amount of information and the intuitive representation of its complexity becomes irreplaceable.”
Giacomo Garoffolo, XR Technical Lead
New technologies in support of didactics
In the training sector this kind of content allows a greater learning experience thanks to the process being comparable to the real one and lived through the normal didactic tools.
This effectiveness then meets efficiency. Virtual assets that can be analysed and manipulated anywhere and at any time drastically reduces the cost of finding and disposing of real organs, of using a specific space and equipment prepared for each student.
We are therefore facing an important evolution in the world of specialist training, which comes from the potential of new technology like augmented reality, already widely used and easy to interact with. A simple smartphone is all that is required to place virtual objects in the environment created around us, and to interact with them simply takes taps on the screen.
The University of Padua, close to 800 years of history, has embraced this new technology to improve the teaching of its PhD students in medicine, testifying to how the technology has been really useful; it offers a real benefit and fits seamlessly into processes developed over years and years of work.
“We are proud to have been the first at University of Padova to develop this new way of teaching. Thanks to our app, every student has a library of anatomical models of hearts affected by simple and complex congenital heart disease. The possibility of three-dimensional reconstruction of heart models starting from common radiological techniques and the aid of Augmented Reality allows anyone with a tablet or smartphone to study the anatomy of congenital heart disease, to navigate inside the heart, making learning interactive, personalized, stimulating and very effective”.
Prof. Vladimiro Vida, Full Professor of Cardiac Surgery at the University of Padua