Sign language recognition with transformer networks

(18-06-2020) Sign language recognition with transformer networks

The Flemish Sign Language corpus is a database of videos in Flemish Sign Language (VGT). In each of these videos, two deaf persons have a conversation about varying subjects. The corpus is used for research into VGT. For this purpose, it is being annotated: annotations are a form of subtitles in written form. The annotation process requires expertise and is time consuming.

This research presents a tool to aid the annotation process. When one wishes to annotate a sign, the proposed system present several possible meanings of that sign. Behind the scenes, this is powered by neural networks. The same kind of neural networks is used as for computer vision and machine translation.

First results are promising. Further research aims to further expedite the annotation process. The annotated videos from the corpus can then be used to train other neural networks to translate conversations in VGT to Dutch, as part of a smartphone app. Such applications would be especially useful in, e.g., education.

This is research of Mathieu De Coster  Mieke Van Herreweghe and Joni Dambre, a cooperation between IDLab and The Department of Linguistics.