TIPS: Tabletop Interactive Play System
More than 90% percent of Deaf and Hard of Hearing (DHH) infants in the US are born to hearing parents. They are at severe risk of language deprivation, which may lead to life-long impact on linguistic, cognitive and socio-emotional development. In this project, we design and develop AR/VR and AI-mediated communication technologies that aims to support context-aware, non-intrusive and culturally-relevant parent-child interaction and collaborative learning of American Sign Language (ASL). The proposed prototype enables empirical studies to collect in-depth design critiques and usability evaluation from domain experts, novice ASL learners, and hearing parents with DHH children.
Participate in a Study
Are you a hearing parent of a deaf or hard-of-hearing child (6 months – 5 years)? Join our fun, interactive, IRB-approved research study!
Research Journey
TIPS and pTIPS: ASL Support on-the-fly
Tabletop Interactive Play System (TIPS) observes parent–child interaction through a camera and microphone, then uses a projector to display relevant ASL signs retrieved from multiple sign language libraries. This line of work investigates how AI can offer context-responsive ASL recommendations and balance usefulness with autonomy in parent-child ASL interactions [ASSETS’22, ASSETS’23, IDC’25]. In addition to the tabletop version, we are also developing portable forms for tablets, smart watches, and smart glasses to make the system more accessible [IDC’22].
CoSignPlay: collaborative ASL learning
CoSignPlay uses an avatar-based collaborative signing approach that splits control between two learners, one managing manual signs and the other non-manual signals like facial expressions [CHI’25, ASSETS’25]. This approach supports family-centered learning, making non-manual signing more accessible and connecting practice to deaf culture.
RhymASL: interactive rhyming ASL storytelling
RhymASL introduces a playful storytelling interactive tool that highlights phonological features of ASL signs. The system generates interactive rhyming ASL stories that families can watch and shape together, encouraging ASL interactions and building phonological awareness [ASSETS’25].
Project Members
