The Original Goal
The goal of the DREAM project is to analyze video data of stories being signed in American Sign Language, focusing specifically on the movement of the mouth, to discover any characteristics governing its movement. Upon analysis, we hope to discover defining characteristics of mouthing and mouth gestures in ASL. The movement of the mouth can be very controversial in Deaf communities. However, researchers have generally classified the movement into two categories: mouthing and mouth gestures. Mouthing refers to the articulation of visual syllables that resemble a spoken word on the mouth. Mouth gestures refer to movements on the mouth that have nothing to do with the spoken language. If we are able to find specific rules governing the mouth, we can program those rules into our avatar to automate the appropriate facial grammar in our translations. Though we have a lot of annotation data for the face and body, there are little to no annotations for the mouth movement. With fellow researchers, Deannia Lucas and Maria Saenz, we plan to use Open Pose to see if we can extract more data.