I'm very busy these months. I'll return to this repo in about 10 days.
An implementation of QANet with PyTorch.
Any contributions are welcome!
- Install pytorch 0.4 for Python 3.6+
pip install -r requirements.txtto install python dependencies.
download.shto download the dataset.
python main.py --mode datato build tensors from the raw dataset.
python main.py --mode trainto train the model. After training,
log/model.ptwill be generated.
python main.py --mode testto test an pretrained model. Default model file is
preproc.py: downloads dataset and builds input tensors.
main.py: program entry; functions about training and testing.
models.py: QANet structure.
Differences from the paper
- The paper doesn't mention which activation function they used. I use relu.
- I don't set the embedding of
- The connector between embedding layers and embedding encoders may be different from the implementation of Google, since the description in the paper is inconsistent (residual block can't be used because the dimensions of input and output are different) and they don't say how they implemented it.
- Reduce memory usage
- Improve converging speed (to reach 60 F1 scores in 1000 iterations)
- Reach state-of-art scroes of the original paper
- Performance analysis
- Test on SQuAD 2.0
- InitialBug: found two bugs: (1) positional encodings require gradients; (2) wrong weight sharing among encoders.
- linthieda: fixed one issue about dependencies and offered computing resources.
- BangLiu: tested the model.
- wlhgtc: (1) improved the calculation of Context-Question Attention; (2) fixed a bug that is compacting embeddings before highway nets.