Counting 3,146 Big Data & Machine Learning Frameworks, Toolsets, and Examples...
Suggestion? Feedback? Tweet @stkim1

Last Commit
Sep. 25, 2018
Sep. 8, 2017


OpenSeq2Seq: toolkit for distributed and mixed precision training of sequence-to-sequence models

This is a research project, not an official NVIDIA product.


OpenSeq2Seq main goal is to allow researchers to most effectively explore various sequence-to-sequence models. The efficiency is achieved by fully supporting distributed and mixed-precision training. OpenSeq2Seq is built using TensorFlow and provides all the necessary building blocks for training encoder-decoder models for neural machine translation and automatic speech recognition. We plan to extend it with other modalities in the future.


  1. Sequence to sequence learning. Currently implemented:
    1. Neural Machine Translation (text2text)
    2. Automatic Speech Recognition (speech2text)
    3. Speech Synthesis (text2speech)
  2. Data-parallel distributed training
    1. Multi-GPU
    2. Multi-node
  3. Mixed precision training for NVIDIA Volta GPUs

Software Requirements

  1. TensorFlow >= 1.9
  2. Horovod >= 0.12.0 (using Horovod is not required, but is highly recommended for multi-GPU setup)


Speech-to-text workflow uses some parts of Mozilla DeepSpeech project.

Text-to-text workflow uses some functions from Tensor2Tensor and Neural Machine Translation (seq2seq) Tutorial.

Related resources


If you use OpenSeq2Seq, please cite this paper

OpenSeq2Seq: extensible toolkit for distributed and mixed precision training of sequence-to-sequence models},
  author={Kuchaiev, Oleksii and Ginsburg, Boris and Gitman, Igor and Lavrukhin, Vitaly and  Case, Carl and Micikevicius, Paulius},
  journal={arXiv preprint arXiv:1805.10387},

Latest Releases
OpenSeq2Seq v18.10
 Sep. 24 2018
OpenSeq2Seq v18.09
 Aug. 29 2018
OpenSeq2Seq v0.5
 Jul. 26 2018
OpenSeq2Seq v0.4
 Jun. 29 2018
OpenSeq2Seq v0.3
 May. 31 2018