In response to the inability of deaf and hard-of-hearing persons to use most smart devices, a software developer has come up with a solution based on deep learning and thanks to which the deaf community can now use Amazon Echo.

Though smart devices have become an integral part of the lives of millions of people, they remain inaccessible for the deaf and hard of hearing, as reported by Numerama. Microsoft, Google and even Pornhub have been working on the problem for some time, though it seems that a software developer by the name of Abhishek Singh has beaten them all in the race for a solution.

In a video posted on Twitter, he showed how Echo, Amazon’s smart device, can understand simple sign language commands thanks to a device that makes use of deep learning. “If these devices are to become a central way in which in interact with our homes or perform tasks then some thought needs to be given towards those who cannot hear or speak” said Singh, who used TensorFlow.js software in making his discovery.

Camera-assisted communication

An artificial intelligence system made available by Google in 2015 and based on deep learning, TensorFlow was used by Singh to make the Alexa-controlled Echo speaker respond to sign language requests.

Thanks to a camera connected to a computer, the signs made by the user can be recorded and interpreted, translated into text and sounds, and then transmitted to Amazon Echo in the same way as an ordinary command. 

Alexa’s answers transcribed in text form

Singh’s device then transmits Alexa’s answers in legible text form to the user’s computer screen. In the demo video, the designer asks the device a question in sign language about the weather, and receives the answer “29 degrees in New York” in writing. He then asks Alexa to tell him a joke.

So will his discovery inspire the tech industry’s leaders? As Numerama reported, Singh has at the very least highlighted the significant amount of progress that still needs to be made in terms of the accessibility of connected devices.

Allianz Partners