Counting 2,412 Big Data & Machine Learning Frameworks, Toolsets, and Examples...
Suggestion? Feedback? Tweet @stkim1

Last Commit
Feb. 10, 2018
Nov. 14, 2017

This project generates animations of pytorch optimizers solving toy problems. Examples Below.

Some nice animations were posted a few years ago by Alex Radford but didn't include the Adam optimizer or landscapes with noise. Louis Tiao blogged about how to make the visualizations. The pytorch unit tests show how to run the optimizers on test functions. I pulled these together and shared the result at


Please note that each optimizer has a differen't learning rate, so they are not directly comparable. This is because simpler optimizers perform better on low dimensional problems. So, with a constant learning rate, the simpler SGD optimizer races to the finish while Adam crawls along. In this case we SGD is to fast to see and Adam is too bording to watch. So I used differen't learning rates for each optimizer so I could show them on the same video.

Beales function

Beales function (cyclic annealing)

Beales function with noise

Six humped camel function


  • git clone
  • jupyter notebook
  • open main.ipynb
  • install any missing dependencies with pip