When he was in school, Michael Hingson created a Braille computer terminal so he could study like all the other students. Fresh out of college, he worked on the development of the Kurzweil Reading Machine for the Blind, the first commercial text-to-speech machine for the visually impaired. He's used white canes and guide dogs, voice controls on his smartphone and virtual assistants like Alexa, all in the name of doing things on his own despite being blind since birth. But something as simple as reading a comic book, or finding the split pea soup among all the cans in the pantry? Until recently, that just seemed impossible.
So when Hingson talks about the time he assembled a piece of furniture with Ikea-style pictorial directions, it's as if he's scaled a mountain. He did it wearing Aira, a camera-enabled set of glasses that beams his field of view to someone who can see, as if to momentarily borrow their eyesight. “I absolutely could not have done that on my own, period, any other way,” Hingson says.
Aira launched six months ago and counts about 400 blind or vision-impaired subscribers. They use the service mostly for help with ordinary tasks—reading a handwritten note, navigating the grocery store, checking when the milk in the fridge expires—and sometimes extraordinary ones, like when this April, a blind man tapped into Aira to run the Boston Marathon. One woman regularly calls in to read comic books out loud to her son; others use the service to take photos good enough to share with friends. “It’s a way to make any visual information available that was never available before," Hingson says.
Right now, Aira’s customers share their video streams with people—Aira calls them "agents"—who work on a model like Uber, with the ability to log on, pick up a user’s call, and get paid for the hours they work. But in the future, Aira’s founder Suman Kanuganti hopes to offload most of that work to artificial intelligence. The same technology that powers computer vision projects at Google and Facebook and Pinterest could one day tell Hingson where he left his house keys, or read the street signs at an intersection, or recognize which of his friends are in the room. One day, computer vision could give the blind more information about their environment than any sighted person could see with the naked eye. And it's all coming, thanks to a new class of wearable devices.
The Aira headset looks sort of like a tricked-out Google Glass, because essentially, it is. Kanuganti experimented with the device as part of the Glass Explorers program in 2013, and he couldn't help but think of his friend Matt Brock, who lost his vision from retinitis pigmentosa in 2006. Glass came with a 5-megapixel camera built-in, capable of 720p video. If Kanuganti could use the camera to snap photos and stream video, could someone like Brock use it to see?
Kanuganti mailed his set to Brock, invited him to a Google Hangouts call, and started a video chat. When Brock's camera came into focus, Kanuganti felt like he was looking directly through his friend’s eyes.
It was a “game-changing moment” for Brock, who rarely left the house without his wife. He had a guide dog, but a dog can’t tell you how to get from your house to the grocery store, or which aisle stocks the bread, or make sure you get the right amount of money back in change. With Kanuganti there for support, Brock strolled down the street to a neighborhood store and bought a bouquet of flowers for his wife. He'd never done anything like that before.
“There were a lot of apps that exist for identifying objects, color, text, and so on, but there wasn’t an application for blind and visually-impaired people to just get up and go,” Kanuganti says. This, it seemed, offered Brock more independence than he had experienced in years.
Together with co-founder Yuja Chang, Kanuganti put together a prototype of what would later become Aira. The device, worn like glasses, pairs to the user’s smartphone to dial an Aira agent. The service runs on a subscription model, like a prepaid mobile phone plan: For $129 a month, a blind customer gets 200 minutes with an agent; $199 gets them 400 minutes and a guarantee that an agent will pick up within 10 seconds.
Right now, Aira isn't a replacement for accessibility tools like white canes. But Chris Danielsen, the Director of Public Relations for the National Federation of the Blind, says it can give blind users more autonomy and help solve the "last 50-foot problem"—that blind people can use GPS to find their way around, but then get stuck trying to find the doorway to a building. Whether the agent is human or AI, he says, that kind of knowledge could be huge. (As a sign of its confidence, the National Federation of the Blind signed on as an Aira investor.) "We're cautious about saying [technology] will solve all of our problems," says Danielsen, "but I do expect that this type of technology will become more and more of an integral part of blind people's lives."
The first modern technology for the blind, Braille, was invented in 1819. The system of raised dots allowed blind people to read and write for the first time, and became standard by the early 20th century. Soon after, white canes and guide dogs offered new ways for blind people to navigate the physical world; auditory tools, like the Kurzweil Reading Machine and text-to-speech programs, would soon follow.
But newer technologies are exploring how blind people might receive visual information in other ways, skirting the sense of hearing and touch. In 2007, a group of California-based researchers introduced a prototype of a "bionic eye" called Argus II. The system translates visual information from a small camera mounted on sunglasses onto a surgically implanted retinal device, which creates electrical pulses inside the eye. It doesn’t replicate vision per se, but those who use Argus II can recognize flashes of light that allow them to identify objects, people, and even large text. "Without the glasses you can't see a thing, put the glasses on you can suddenly see," one Argus II user told Wired UK. "You get that wow factor every time you do it."
The Argus II was approved by the FDA in 2013, but it's hardly become mainstream. The system costs $150,000, before the fees from surgery and training. And while the effects can be remarkable, it doesn't work on everyone—during clinical trials, 30 percent of people experienced adverse effects from the implant, including retinal detachment.
But the concept of using a camera like a surrogate eye? That paved the way for a new class of wearables, which offer some degree of computer vision magic without the cost or commitment of an implanted device.
One such device, MyEye by Orcam, translates visual information from a small camera into an audio earpiece. Unlike Aira, MyEye runs entirely on AI software. Press a button and the device can dictate text, or identify an object in view. The software can also memorize the faces of 100 people and 150 things—so when you’re looking for a familiar face in a crowd or trying to remember where you put down your wallet, MyEye searches for you. Eliminating the human element also returns autonomy and privacy to users. No need to call an agent to read aloud a personal document, or to just see what’s on a restaurant menu. The computer does all that on its own.
Other wearables use light and magnification to assist people with limited vision. A device called eSight, which users wear like a VR headset, uses a high-resolution camera to magnify images and project them onto the OLED screen in front of the user’s eyes. Brian Mech, eSight’s CEO, says that only about 15 percent of the visually impaired population is totally blind; for everyone else, devices like eSight help enhance some of the vision that's left. And because it's a wearable, Mech says, you eliminate the cost and the risk of surgery. “All you have to do it put it on,” Mech says, "and you know within seconds whether or not it works for you."
Even more remarkable, a group of neuroscientists in Wisconsin designed a system by which blind people could receive optical sensations through their tongue. The device, called BrainPort, picks up light signals from the camera mounted on a set of sunglasses and translates it into electrical pulses on a tiny electric "lollipop." With a bit of training, those pulses—which some users have described as feeling like the tiny explosions of Pop Rocks on the tongue—can be used to navigate or understand the layout of a room.
At $10,000, the BrainPort is not cheap. Other devices are similarly priced in the thousands: eSight retails for $10,000; MyEye for $3,500. (More basic tools, like a finger-mounted device that scans text and reads it aloud, cost less; and some apps, like Seeing AI, provide smartphone computer vision technology to the blind for free.) Expense aside, the design of these devices may need to improve before people sign on to walk around every day wearing something that looks like an Oculus Rift. But the results can be profound: Erik Weihenmayer, who became the first and only blind person to scale Mount Everest in 2001, has used the BrainPort to navigate climbs; Marc Muszynski, a man who suffers from macular degeneration, used eSight to fly a plane.
The potential for these devices only grows as companies like Google, Pinterest, Uber, and many others improve computer vision for commercial purposes. The software teaching self-driving cars how to navigate the rules of the road could help a blind person pass through a busy intersection without the need for a guide dog. The AI that Pinterest uses to recognize your breakfast and send you related pins could also be used to recognize that those waffles have strawberries on them, which you shouldn’t eat because you’re allergic. And Google’s computer vision program, which can discern if someone is genuinely emoting or just faking it, could tell you whether people are smiling or grimacing at your terrible joke.
For now, this corner of the wearables industry is still young. Those who use Aira find that calls sometimes drop and the built-in camera doesn’t capture images with perfect fidelity. There are ethical and legal limitations too: Agents toe a delicate line between telling Aira customers what they’re seeing and telling them what to do. An agent can’t, for example, tell someone it’s safe to cross the street; the liability is too high. Instead, they’re instructed to say things like, “There’s a crosswalk a few feet to your left” and “The traffic light is green.” And, just like Google Glass, the device is hardly in danger of becoming fashionable.
But that’s right now. Within three years, Kanuganti says not only will the device’s hardware improve, but more than half of Aira calls will be automated.
“There’s a lot of computer vision stuff that exists already: platforms like Google Cloud Vision,” says Kanuganti, referring to Google's computer vision platform. “And we have the data coming to us from the glasses.”
Aira will still staff humans, he says, which sets his product apart from AI-only tools like MyEye. Some requests amount to more than simply understanding what something is or what it says, and Kanuganti sees the ability to handle those tasks as a reason to choose Aira over similar devices. Once, a customer asked an Aira agent to provide a running visual commentary on a trip to Disneyland. That’s the kind of stuff you just can’t offload to a computer.
But for the rest of it, artificial intelligence could pave a new path forward. Imagine a world in which Aira scans the faces of people in a room and tells you when one of your friends has arrived, based on the contacts in your phone or your Facebook friend list. Or a world in which Aira connects to devices beyond your phone, syncing with the virtual assistant that already lives in your phone and your smart speaker. Hingson, like many in the blind community, already relies on his Amazon Echo for audio dictation tasks like making grocery lists. Wouldn’t it be cool, he tells me, if he could share one of those lists with Aira and then use the glasses—whether powered by AI or a human—to guide him through the grocery store, telling him where to find each item in each aisle?
It’s a future within view. For now, though, Hingson talks about all the things he can already do—like scanning the information on a business card, or finding his way around a crowded conference hall—thanks to assistive technology. If, in the coming years, artificial intelligence makes it even easier for him to navigate the world around him, then all the better.
