U-M assistant professor of computer science and engineering Anhong Guo led the development of the two apps for visually impaired users with a team based at the University of Michigan. The research has been funded by both U-M and Google.
One of these apps, VizLens, is listed as an, “appliance reader app for users with visual impairment.” VizLens allows users to scan and access an existing image of an inaccessible interface– for example, a flat microwave panel –and hear the button labels read aloud or, alternatively, use the camera to have labels read out in real-time in correspondence to where the user places their finger.
The other app, ImageExplorer, is listed as an, “image exploration app for users with visual impairment.” The alt text or caption may describe an image as, “A boy on a skateboard.” ImageExplorer can tell you that the man is wearing a black shirt, blue shorts, and blue and black shoes. It can also detect people and objects in the background and identify these objects in more detail with a quick double-tap.
The level of detail ImageExplorer provides, with tags recognizing objects from benches to bags, and its ability to identify where each object and person rests in an image make it more unique and more comprehensive.
As of publication, ImageExplorer will occasionally provide conflicting information in its descriptions, and the app often simplifies certain details, such as identifying most tops as “shirts.”
“In spite of these errors, the results we presented in 2022 show that ImageExplorer enables users to make more informed judgements of the accuracy of the AI-generated captions,” Guo said.
The two apps were tested by hundreds of visually impaired, user-testing participants who provided feedback to the team, which is continuing to work to make improvements.
“The accuracy relies on the models we use, and as they improve, ImageExplorer will improve,” Guo said. He hopes that the public release of the apps will bring forth feedback for the team.
According to Guo’s website, both apps were released and made publicly available on the iOS App Store in May of 2023. Both are available to download for free today.
Article summarized from Michigan Engineering: New apps for visually impaired users provide virtual labels for controls and a way to explore images.
