Word Ordering: Can Neural Networks Put a Scramble of Words in Correct Order?
When learning a second language, one of the hardest challenges is likely to be familiar with the word order. Word order can be also important in machine translation because translation is, roughly speaking, a process that one arranges words of target language which are equivalent to source language in order. Probably you've done a word scramble game where you are to put shuffled words or letters in the original order. I think it's quite fun to see if neural networks can do it. Okay. Can you order the following words correctly?
can translation machine order also important be word in
- NumPy >= 1.11.1
- TensorFlow == 1.2 (Probably 1.3 should work, too, though I didn't test it)
I employ the Transformer which was introduced in the paper Attention Is All You Need. It's known to the state-of-the-art model in the machine translation task as of 2017. However, I don't know if it fits in this task the best. Actually, I think a simpler architecture may work. The figure below is borrowed from the paper.
hyperparams.pyincludes all hyper parameters that are needed.
data_load.pycontains functions regarding loading and batching data.
modules.pyhas all building blocks for encoder/decoder networks.
train.pycontains the model and training code.
eval.pyis for evaluation and inference.
- STEP 1. Download and extract Leipzig English News 2015 1M Corpus.
- STEP 2. Adjust hyper parameters in
- STEP 3. Run
train.pyor download the pretrained files.
Training Loss and Accuracy
- Training Loss
- Training Accuracy
We take WER (Word Error Rate) as the metric. WER is computed as follows:
WER = Edit distance / Number of words
Total WER : 10731/23541=0.46
results folder for details.