## Counting 3,567 Big Data & Machine Learning Frameworks, Toolsets, and Examples...Suggestion? Feedback? Tweet @stkim1

Author
Contributors
Last Commit
Feb. 13, 2019
Created
Dec. 16, 2018

You can find our report for the reproducibility challenge here.

## Setup Dependencies

The recommended version for running the experiments is Python3.

These experiments have been written in tensorflow's eager mode so installing the dependencies is a must to run the code:

1. Follow the installation guide on Tensorflow Homepage for installing Tensorflow-GPU or Tensorflow-CPU.
2. Follow instructions outlined on Keras Homepage for installing Keras.

Run a vanilla experiment using the following command at the directory root folder.

python vgg16-net/run.py

## Project Structure

The skeletal overview of the project is as follows:

.
├── vgg16-net/
│   ├── run.py  # A script to run the experiments over VGG Net architechture
│   └── model.py     # VGGNet model
├── resnet18/
│   ├── run.py # A script to run the experiments over ResNet architechture
│   └── model.py     # Resnet 18 model
├── wide-resnet/
│   ├── run.py        #A script to run the experiments over ResNet architechture
│   ├── model.py    # Wide Resnet 18 model
.
folders and files below will be generated after you run the experiment in each model directory
.
├── model_{optimizer}_{dataset}.csv                 # Stores logs for the experiment
└── model_{optimizer}_{dataset}.h5              # Stores the weights of the final model trained 

## Defining Experiment Configuration

You can set the experiment configuration by changing the dictionary in the run.py files. These dictionary contains all the hyperparameter for the each optimizers ie. Adam, Amsgrad, SGD Momentum and Padam.

## Experiments

We carry out the experiments to compare the performance of four optimizers - Adam, Amsgrad, SGD Momentum and the proposed algorithm Padam, on 3 modern deep learning architectures ResNet18, WideResNet18 and VGGNet16, over CIFAR-10 and CIFAR-100 datasets. All the experiments have been run for 200 epochs, using categorical cross entropy loss function.

## Results

We were sucessful in reproducing the results as predicted in the paper for Cifar-10 and Cifar-100. It is observed that Padam indeed generalizes better than other adaptive gradient method, although it does have a few shortcomings as mentioned in our report. Here, we show the results for VGGNet16, rest of the results have been included in the report.

Results on the CIFAR-10 dataset for VGGNet.

## References

Original work by authors:

@article{Padam,
title={Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks},
author={Jinghui Chen, Quanquan Gu},
journal={arXiv preprint arXiv:1806.06763},
year={2018}
}

If you find our work useful, please consider citing it:

@article{Padam-ICLR,
title={ICLR Reproducibility Challenge Report (Padam : Closing The Generalization Gap Of Adaptive Gradient Methods in Training Deep Neural Networks)},
author={Harshal Mittal, Kartikey Pandey, Yash Kant},
journal={arXiv preprint arXiv:1901.09878},
year={2019}
}