Published on Sat Nov 03 2018

SimplerVoice: A Key Message & Visual Description Generator System for Illiteracy

Minh N. B. Nguyen, Samuel Thomas, Anne E. Gattiker, Sujatha Kashyap, Kush R. Varshney

SimplerVoice can automatically generate sensible sentences describing an unknown object. It can also extract semantic meanings of the object in the form of a query string. The system can represent the string as multiple types of visual guidance.

0
0
0
Abstract

We introduce SimplerVoice: a key message and visual description generator system to help low-literate adults navigate the information-dense world with confidence, on their own. SimplerVoice can automatically generate sensible sentences describing an unknown object, extract semantic meanings of the object usage in the form of a query string, then, represent the string as multiple types of visual guidance (pictures, pictographs, etc.). We demonstrate SimplerVoice system in a case study of generating grocery products' manuals through a mobile application. To evaluate, we conducted a user study on SimplerVoice's generated description in comparison to the information interpreted by users from other methods: the original product package and search engines' top result, in which SimplerVoice achieved the highest performance score: 4.82 on 5-point mean opinion score scale. Our result shows that SimplerVoice is able to provide low-literate end-users with simple yet informative components to help them understand how to use the grocery products, and that the system may potentially provide benefits in other real-world use cases

Mon Apr 26 2021
Computer Vision
InfographicVQA
InfographicVQA is a new dataset that comprises a diverse collection of infographics along with natural language questions and answers annotations. We curate the dataset with emphasis on elementary reasoning and basic arithmetic skills.
4
1
1
Tue Apr 07 2020
Artificial Intelligence
e-SNLI-VE-2.0: Corrected Visual-Textual Entailment with Natural Language Explanations
SNLI-VE is a large, real-world dataset for fine-grained multimodal reasoning. However, the automatic way in which it has been assembled gives rise to a large number of errors. We present a data collection effort to correct the class with the highest error
1
0
1
Sat Aug 07 2021our pick
Artificial Intelligence
HelpViz: Automatic Generation of Contextual Visual MobileTutorials from Text-Based Instructions
HelpViz transforms text instructions to graphical tutorials in batch. It extracts a sequence of actions from each text instruction through an instruction parsing model, and executes the extracted actions on an Android emulators. The automatic execution of each instruction produces a set of graphical and structural assets.
2
0
1
Sat Feb 21 2015
Computer Vision
Don't Just Listen, Use Your Imagination: Leveraging Visual Common Sense for Non-Visual Tasks
In this paper we leverage semantic common sense knowledge learned from images in two textual tasks: fill-in-the-blank and visual paraphrasing. We propose to "imagine" the scene behind the text, and leverage visual cues from the "imagined" scenes in addition to
0
0
0
Sun Jul 03 2016
Artificial Intelligence
Visualizing Natural Language Descriptions: A Survey
A natural language interface exploits the conceptual simplicity and naturalness of the language to create a high-level user-friendly communication channel between humans and machines. One of the promising applications of such interfaces is generating visual interpretations of semantic content of a given natural language.
0
0
0
Tue Oct 06 2020
Artificial Intelligence
Converting the Point of View of Messages Spoken to Virtual Assistants
We developed a system to allow virtual assistants to convert a voice message from one user, convert the point of view of the message, and then deliver the result to its target user. We also investigated Neural Machine Translation (NMT) approaches, including LSTMs, CopyNet, and T5.
0
0
0