Counting 3,384 Big Data & Machine Learning Frameworks, Toolsets, and Examples...
Suggestion? Feedback? Tweet @stkim1


Relation Networks and Sort-of-CLEVR in Tensorflow

As part of the implementation series of Joseph Lim's group at USC, our motivation is to accelerate (or sometimes delay) research in the AI community by promoting open-source projects. To this end, we implement state-of-the-art research papers, and publicly share them with concise reports. Please visit our group github site for other projects.

This project is implemented by Shao-Hua Sun and reviewed by Youngwoon Lee.


This project includes a Tensorflow implementation of Relation Networks and a dataset generator which generates a synthetic VQA dataset named Sort-of-CLEVR proposed in the paper A Simple Neural Network Module for Relational Reasoning.

Relation Networks

Relational reasoning is an essential component of intelligent systems. To this end, Relation Networks (RNs) are proposed to solve problems hinging on inherently relational concepts. To be more specific, RN is a composite function:


where o represents inidividual object while f and g are functions dealing with relational reasoning which are implemented as MLPs. Note that objects mentioned here are not necessary to be real objects; instead, they could consist of the background, particular physical objects, textures, conjunctions of physical objects, etc. In the implementation, objects are defined by convoluted features. The model architecture proposed to solve Visual Question Answering (VQA) problems is as follows.

In addition to the RN model, a baseline model which consists of convolutional layers followed by MLPs is also provided in this implementation.


To verify the effectiveness of RNs, a synthesized VQA dataset is proposed in the paper named Sort-of-CLEVR. The dataset consists of paired questions and answers as well as images containing colorful shapes.

Each image has a number of shapes (rectangle or circle) which have different colors (red, blue, green, yellow, cyan, or magenta). Here are some examples of images.

Questions are separated into relational and non-relational questions which are encoded as binary strings to prevent the effect of language parsing and embedding; while answers are represented as one-hot vectors. Examples of images, questions and answers are as follow.

Given a queried color, all the possible questions are as follows.

Non-relational questions

  • Is it a circle or a rectangle?
  • Is it closer to the bottom of the image?
  • Is it on the left of the image?

Relational questions

  • The color of the nearest object?
  • The color of the farthest object?

And the possible answer is a fixed length one-hot vector whose elements represent

[red, blue, green, yellow, cyan, magenta, circle, rectangle, yes, no]

File format

Generated files use HDF5 file format. Each data point contains an image, an one-hot vector q encoding a question, and an one-hot vector a encoding the corresponding answer.

Note that this implementation only follows the main idea of the original paper while differing a lot in implementation details such as model architectures, hyperparameters, applied optimizer, etc. Also, the design of Sort-of-CLEVR only follows the high-level ideas of the one proposed in the orginal paper.

*This code is still being developed and subject to change.




Generate a default Sort-of-CLEVR dataset:

$ python

Or generate your own Sort-of-CLEVR dataset by specifying args:

$ python --dataset_size 12345 --img_size 256

Or you can even change the number of shape presented in the images, the number of possible colors, types of questions and answers by configuring the file


Train a RN model with a default Sort-of-CLEVR dataset:

$ python

Or specify your own settings:

$ python --model baseline --dataset_path Sort-of-CLEVR_xyz --batch_size 64 --learning_rate 1e-4 --lr_weight_decay 

During the training phase, the samples including images, questions, ground truth answers, and predicted answers can be monitored on Tensorboard if the library Tensorflow Plot is installed. Here is an example.


Test a trained model by specifying the dataset and the model used for training and a checkpoint:

$ python --dataset_path Sort-of-CLEVR_default --model rn --checkpoint_path ckpt_dir

Please note that ckpt_dir should be like: train_dir/baseline-Sort-of-CLEVR_default_lr_0.001-20170619-040301/model-10001


Both the baseline model and the RN model were tested on three Sort-of-CLEVR datasets which have 2, 4, or 6 shapes in each image, respectively.


RN model accuracy

RN model loss

Baseline model accuracy

Baseline model loss


Each image has 6 shapes

RN model Baseline model
Non-relational question 93.24% 68.37%
Relational question 70.93% 32.93%
Overall 83.51% 54.19%

Each image has 4 shapes

RN model Baseline model
Non-relational question 97.64% 79.86%
Relational question 75.78% 45.56%
Overall 93.10% 66.10%

Each image has 2 shapes

RN model Baseline model
Non-relational question 99.76% 98.22%
Relational question 100.00% 100.00%
Overall 99.85% 98.93%

Can learned knowledge be transferred?

Models trained on 4-shape dataset and tested on 6-shape dataset

RN model Baseline model
Non-relational question 96.51% 80.57%
Relational question 55.07% 29.63%
Overall 79.95% 60.19%

Models trained on 4-shape dataset and tested on 2-shape dataset

RN model Baseline model
Non-relational question 94.71% 91.47%
Relational question 49.88% 57.58%
Overall 76.73% 77.95%

Related works


Shao-Hua Sun / @shaohua0116 @ Joseph Lim's research lab @ USC