Brighton Vino Jegarajan

Project Team
My Role
Interaction Design | Interface Design | Google Glass App Development | Android App Development | User Evaluation | Field Testing

The Goal

The overarching goal of the project is to assist the hearing-impaired to communicate with the assistance of Google Glass. Approximately 17 percent (36 million) of American adults report some degree of hearing loss. Late-deaf and hard-of-hearing individuals need accommodations that transform aural information to visual information.

Captions that are generated in real-time can enhance accessibility to audio information in conversations, lectures and other events. We use the visual display capabilities of Google Glass to assist the deaf by displaying real-time captioning of the conversation.

The Motivation

With this project, we targeted two main user groups: the hard-of-hearing and the late deaf. We realize that several of the hard-of-hearing members of the Deaf community will be wearing a hearing aid and have a heightened ability to read lips; however, these individuals cannot understand speech without seeing the speaker’s face or without their hearing aids. Instant captioning would make their lives a lot easier in many aspects.

Research and Application

This project will be an extension of the Captioning on Glass project conducted by Jay Zuerndorfer and Dr. Thad Starner. The original prototype consisted of a base app for Google Glass with a companion Android phone app for voice input.

The system requires the hearing impaired user to wear Google Glass and give his Android phone to the person that he wants to have a conversation with. Using the Android phone’s microphone and speech processing capabilities, an Android application would send the text to the Glass over a secured Bluetooth connection. Google’s Speech to Text API is used to help with that transcription. The text would be sent to the Glass.

Heuristic Evaluation and User Study

Before we developed this system further and proceeded to test it with our target user group, we first conducted a heuristic evaluation with peer experts on this existing prototype and conducted semi-structure interviews with target users. Over the course of this phase, we realized the limitations of the system better and understand the requirements of the target population and changes necessary to accommodate typical characteristics of the population.

Key Areas

We identified the following key areas to focus our development efforts

  • Natural Communication: The process of communication should be more fluid, with an effortless connection and close to-real-time captioning.
  • Accessibility: With majority of the target population not greatly techy-savy and usually have below average vision, we focused on enhancements for such users to use the system with ease.


The system has been specifically designed to ease the use for the older-adult population. The following are the features that were added to the initial prototype system.

  • Seamless Connection: System remembers the last device that it was paired to and connects to the device if it is in its vicinity.
  • Real-Time Transcription: Transcription of the conversation as close to real-time as possible sending transcriptions on the fly.
  • Transcription Accuracy: User can select an alternate transcriptions from the five best context based possibilities or manually edit the text.
  • Environment Aware: Adapts to the environment by altering hue, saturation and brightness of the background and text based on the ambient light.
  • Accessibility: Size of the font can be altered to account for Glass users with vision issues, Audio and haptic feedback on the application allows to maintain eye contact in face-to-face conversations.

User Evaluation and Diary Study

15 user evaluation sessions and a 3-day diary study in collaboration with HomeLab at Georgia Tech Research Institute (GTRI) was completed as a part of this project to evaluate the completed system.

We learned that the system would be most useful for those who find it difficult in hearing other people during face-to-face conversations, but have a good speech ability to carry on the conversation. The system seemed overwhelming, largely due to the lack of familiarity with the user interface paradigms of smartphones and Glass. Reducing the delay in transcription that was perceived by the user improved the experience dramatically.

We also envisioned using the capabilities of the system to caption television audio and/or audio books. But the quality of Speech-to-text varied significantly for practical uses in other cases.

Winners - Convergence Innovation Competition - Spring '14

  • One for the album - Winners - Connected Home & People's Choice AwardCIC Spring '14

  • Featured on Georgia Tech Daily Digest and others after the double-win at CIC Spring'14.Read Article

  • Amateur 'Oscar Selfie' in front of the poster at CIC Spring '14 Finals - 15th April


Winners' Interview