We all love a good meme, we all love to spread the word around and enjoy a good laugh. However, people with visual impairments often face difficulty to comprehend these memes. There is good news for them, as researchers have developed a method to automatically identify memes and then apply pre-written templates to add descriptive alt text. These then make them intelligible via existing assistive technologies.
“Memes are images that are copied and then overlaid with slight variations of text. They are often humorous and convey a shared experience, but “if you’re blind, you miss that part of the conversation,” said study researcher Cole Gleason from Carnegie Mellon University in US.
One of the biggest impediments with memes is that they largely live within social media platforms which do not have the bandwidth for text to be added. For instance Twitter allows people to add alt text to their images, but then, even this feature is not entirely easy to locate.
Basic computer vision techniques make it easy to describe the images
According to the researchers, basic computer vision techniques make it easy to describe the images underlying each meme. What essentially happens is that, optical character recognition techniques help decipher the overlaid text, which can change with each iteration of the meme.
Hence, for each meme type, it becomes important to make one specific template which describes the image. Following this, the overlaid text can be tweaked around for each iteration of that meme.
“It depended on the meme if the humor translated. Some of the visuals are more nuanced, and sometimes it’s explicit and you can just describe it,” Gleason said.
The team has also made a platform which translates memes into sound rather than text. So, the users will be able to search through a sound library and then proceed to drag and drop elements into a template.
“One of the reasons we tried the audio memes was because we thought alt text would kill the joke, but people still preferred the text because they’re so used to it,” Gleason said.