Empowering navigation for the visually impaired through Augmented Reality

Vision loss and visual impairment have long been a significant concern for human well-being amid an increasingly aging population. In a collaborative effort by researchers from The Hong Kong Polytechnic University (PolyU) and The University of Waterloo, they have invented a groundbreaking device that utilizes Augmented Reality (AR) technology, revolutionizing navigation for visually impaired individuals on their world. This device offers them a newfound sense of independence and freedom.

The research project “Augmented Reality Obstacle Detection” (ObstAR), is led by Prof. Allen Cheong, Associate Head (National and International Engagement) and Professor of the School of Optometry of PolyU, and Deputy Director of Centre for Eye and Vision Research (CEVR), in collaboration with Prof. Ben Thompson, University Research Chair and Professor, School of Optometry and Vision Science, the University of Waterloo, and Chief Executive Officer and Scientific Director of CEVR. The research aims to develop an AR-based navigation device that allows visually impaired individuals to minimize their dependence on conventional assistive tools, like walking canes or assistance from others. CEVR is a partnership between PolyU and the University of Waterloo, operating under the Health@InnoHK cluster.

Prof. Cheong said, “Individuals with visual impairments may encounter various forms of vision loss, which can be attributed to neurological or ocular disorders or even the natural aging process. Tailored route navigation solutions are required to meet the needs of people.” Prof. Cheong specializes in geriatric and vision rehabilitation, leading the Vision Rehabilitation Clinic of PolyU Optometry Clinic.

Clinical research for practical applications

The research combines a clinical study that examines behavior of visually impaired patients and healthy participants when navigating familiar and unfamiliar obstacles, with the practical implementation of a navigational aid built using AR glasses and an artificial intelligence recognition algorithm.

To enhance the ability to recognize the environment and avoid obstacles, the device integrates a suite of advanced algorithms, including obstacle avoidance navigation, object recognition and segmentation, scene recognition, text recognition, and gesture recognition. This comprehensive approach aims to meet the diverse navigation needs of patients, ensuring safe navigation and heightened environmental awareness.

One key research focus is identifying specific areas of interest (AOIs), such as traffic lights, zebra crossings, sharp turns, and large banners. This personalized guidance can greatly benefit users who frequently traverse the same routes, as the system can offer customized support based on their familiarity with the environment.

Navigating a new frontier

The distinguishing design of ObstAR lies in the development of an innovative algorithm for image segmentation and information fusion, using RGB (Red, Green, Blue) and depth cameras to enable real-time obstacle avoidance navigation. This advancement allows the identification of more distant navigable paths within the camera’s capture area, while also enabling more accurate recognition of obstacles that are difficult to identify using traditional image segmentation techniques. Also, the team aims to incorporate real-time text-to-speech instructions to supplement areas not covered by the AR, ensuring comprehensive support for users.

Notably, ObstAR stands at the forefront of assistive technology, offering a transformative solution for visually impaired individuals. It was awarded the prestigious “Gold Medal with Congratulations of the Jury” at the 49th Geneva Inventions Expo.

Prof. Cheong said, “The advancements in AR and its growing acceptance provide an ideal platform to introduce this new form of assistive technology. This project fully demonstrates the immense potential of technology to enhance the quality of life for the visually impaired. It promises to open up new possibilities for the mobility freedom and social inclusion of the visually impaired.”

Prof. Cheong’s research interests focus on the psychophysical, behavioral, and clinical aspects of aging and low vision research. Her primary goal is to use different interventions to improve patients’ functional performance in daily activities, such as reading, mobility and navigation. The research also aims to establish cost effective vision rehabilitation models to enhance patients’ quality of life.

Prof. Cheong believes that ObstAR’s has a profound potential impact. Users could gain confidence in tackling daily challenges, thereby enhancing their functional performance and overall well-being. “We are on a mission to redefine independence for those living with vision loss. It is not just about creating an innovative product, but about bringing change and improvement to their lives,” she said.