Benchmark Visual Question Answer Models by using Focus Map

Abstract

Inferring and Executing Programs for Visual Reasoning proposes a model for visual reasoning that consists of a program generator and an execution engine to avoid end-to-end models. To show that the model actually learns which objects to focus on to answer the questions, the authors give a visualization of the norm of the gradient of the sum of the predicted answer scores with respect to the final feature map. However, the authors do not evaluate the efficiency of focus map. This paper purposed a method for evaluating it. We generate several kinds of questions to test different keywords. We infer focus maps from the model by asking these questions and evaluate them by comparing with the segmentation graph. Furthermore, this method can be applied to any model if focus maps can be inferred from it. By evaluating focus map of different models on the CLEVR dataset, we will show that CLEVR-iep model has learned where to focus more than end-to-end models.

图片1

Authors

Wenda Qiu
Yueyang Xianzang
Zhekai Zhang

Introduction

Code

Our source code is available on Github, including:

  • A demo to demonstrate a sample benchmark.
  • A generator for focus map.

Paper

You can download the paper here.

Bibtex

@article{vqabenchmark,
  title={Benchmark Visual Question Answer Models by using Focus Map},
  author={Qiu, Wenda and Xianzang, Yueyang and Zhang, Zhekai},
  journal={arXiv preprint arXiv:1801.05302},
  year={2018}
}

Examples

图片2

 

Advertisements