ESPnet: end-to-end speech processing toolkit
ESPnet is an end-to-end speech processing toolkit, mainly focuses on end-to-end speech recognition. ESPnet uses chainer and pytorch as a main deep learning engine, and also follows Kaldi style data processing, feature extraction/format, and recipes to provide a complete setup for speech recognition and other speech processing experiments.
- Hybrid CTC/attention based end-to-end ASR
- Fast/accurate training with CTC/attention multitask training
- CTC/attention joint decoding to boost monotonic alignment decoding
- Encoder: VGG-like CNN + BLSTM or pyramid BLSTM
- Attention: Dot product, location-aware attention, variants of multihead (pytorch only)
- Incorporate RNNLM/LSTMLM trained only with text data
- Flexible network architecture thanks to chainer and pytorch
- Kaldi style complete recipe
- Support numbers of ASR benchmarks (WSJ, Switchboard, CHiME-4, Librispeech, TED, CSJ, AMI, HKUST, Voxforge, etc.)
- State-of-the-art performance in Japanese/Chinese benchmarks (comparable/superior to hybrid DNN/HMM and CTC)
- Moderate performance in standard English benchmarks
Cuda 8.0 (for the use of GPU)
Cudnn 6 (for the use of GPU)
NCCL 2.0+ (for the use of multi-GPUs)
PyTorch 0.3.x (no support for PyTorch 0.4.x)
Install Kaldi, Python libraries and other required tools using system python and virtualenv
$ cd tools $ make -j
or using local miniconda
$ cd tools $ make -f conda.mk -j
For higher version (>4.9) of gcc and cuda 9.1 use following command:
$ cd tools $ make -j -f Makefile.cuda91.gcc6
You can compare Makefile and Makefile.cuda91.gcc6 to change makefile accordingly for other version of gcc/cuda.
To use cuda (and cudnn), make sure to set paths in your
CUDAROOT=/path/to/cuda export PATH=$CUDAROOT/bin:$PATH export LD_LIBRARY_PATH=$CUDAROOT/lib64:$LD_LIBRARY_PATH export CUDA_HOME=$CUDAROOT export CUDA_PATH=$CUDAROOT
If you want to use multiple GPUs, you should install nccl
and set paths in your
.bash_profile appropriately, for example:
CUDAROOT=/path/to/cuda NCCL_ROOT=/path/to/nccl export CPATH=$NCCL_ROOT/include:$CPATH export LD_LIBRARY_PATH=$NCCL_ROOT/lib/:$CUDAROOT/lib64:$LD_LIBRARY_PATH export LIBRARY_PATH=$NCCL_ROOT/lib/:$LIBRARY_PATH export CUDA_HOME=$CUDAROOT export CUDA_PATH=$CUDAROOT
Execution of example scripts
Move to an example directory under the
We prepare several major ASR benchmarks including WSJ, CHiME-4, and TED.
The following directory is an example of performing ASR experiment with the VoxForge Italian Corpus.
$ cd egs/voxforge/asr1
Once move to the directory, then, execute the following main script with a chainer backend:
or execute the following main script with a pytorch backend (currently the pytorch backend does not support VGG-like layers):
$ ./run.sh --backend pytorch --etype blstmp
With this main script, you can perform a full procedure of ASR experiments including
- Data download
- Data preparation (Kaldi style, see http://kaldi-asr.org/doc/data_prep.html)
- Feature extraction (Kaldi style, see http://kaldi-asr.org/doc/feat.html)
- Dictionary and JSON format data preparation
- Training based on chainer or pytorch.
- Recognition and scoring
Use of GPU
If you use GPU in your experiment, set
--ngpu option in
run.sh appropriately, e.g.,
# use single gpu $ ./run.sh --ngpu 1 # use multi-gpu $ ./run.sh --ngpu 3 # if you want to specify gpus, set CUDA_VISIBLE_DEVICES as follows # (Note that if you use slurm, this specification is not needed) $ CUDA_VISIBLE_DEVICES=0,1,2 ./run.sh --ngpu 3 # use cpu $ ./run.sh --ngpu 0
Default setup uses CPU (
Note that if you want to use multi-gpu, the installation of nccl is required before setup.
Error due to ACS (Multiple GPUs)
When using multiple GPUs, if the training freezes or lower performance than expected is observed, verify that PCI Express Access Control Services (ACS) are disabled. Larger discussions can be found at: link1 link2 link3. To disable the PCI Express ACS follow instructions written here. You need to have a ROOT user access or request to your administrator for it.
To work inside a docker container, execute
run.sh located inside the docker directory.
It will build a container and execute the main program specified by the following GPU, ASR example, and outside directory information, as follows:
$ cd docker $ ./run.sh --docker_gpu 0 --docker_egs chime4/asr1 --docker_folders /export/corpora4/CHiME4/CHiME3 --dlayers 1 --ngpu 1
The docker container is built based on the CUDA and CUDNN version installed in your computer.
The arguments required for the docker configuration have a prefix "--docker" (e.g.,
run.sh accept all normal ESPnet arguments, which must be followed by these docker arguments.
Multiple GPUs should be specified with the following options:
$ cd docker $ ./run.sh --docker_gpu 0,1,2 --docker_egs chime5/asr1 --docker_folders /export/corpora4/CHiME5 --ngpu 3
Note that all experimental files and results are created under the normal example directories (
Setup in your cluster
cmd.sh according to your cluster setup.
If you run experiments with your local machine, please use default
For more information about
cmd.sh see http://kaldi-asr.org/doc/queue.html.
It supports Grid Engine (
queue.pl), SLURM (
Error due to matplotlib
If you have the following error (or other numpy related errors),
RuntimeError: module compiled against API version 0xc but this version of numpy is 0xb Exception in main training loop: numpy.core.multiarray failed to import Traceback (most recent call last): ; : from . import _path, rcParams ImportError: numpy.core.multiarray failed to import
Then, please reinstall matplotlib with the following command:
$ cd egs/voxforge/asr1 $ . ./path.sh $ pip install pip --upgrade; pip uninstall matplotlib; pip --no-cache-dir install matplotlib
CTC, attention, and hybrid CTC/attention
ESPnet can completely switch the mode from CTC, attention, and hybrid CTC/attention
# hybrid CTC/attention (default) # --mtlalpha 0.5 and --ctc_weight 0.3 in most cases $ ./run.sh # CTC mode $ ./run.sh --mtlalpha 1.0 --ctc_weight 1.0 --recog_model loss.best # attention mode $ ./run.sh --mtlalpha 0.0 --ctc_weight 0.0
The CTC training mode does not output the validation accuracy, and the optimum model is selected with its loss value
About the effectiveness of the hybrid CTC/attention during training and recognition, see  and .
We list the character error rate (CER) and word error rate (WER) of major ASR tasks.
|CER (%)||WER (%)|
Chainer and Pytorch backends
|VGG-like encoder||supported||no support|
|#Attention types||3 (no attention, dot, location)||12 including variants of multihead|
References (Please cite the following articles)
 Suyoun Kim, Takaaki Hori, and Shinji Watanabe, "Joint CTC-attention based end-to-end speech recognition using multi-task learning," Proc. ICASSP'17, pp. 4835--4839 (2017)
 Shinji Watanabe, Takaaki Hori, Suyoun Kim, John R. Hershey and Tomoki Hayashi, "Hybrid CTC/Attention Architecture for End-to-End Speech Recognition," IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1240-1253, Dec. 2017