Counting 3,463 Big Data & Machine Learning Frameworks, Toolsets, and Examples...
Suggestion? Feedback? Tweet @stkim1

Handwritten Text Recognition (OCR) with MXNet Gluon

These notebooks have been created by Jonathan Chung, as part of his internship as Applied Scientist @ Amazon AI, in collaboration with Thomas Delteil who built the original prototype.

The pipeline is composed of 3 steps:

The entire inference pipeline can be found in this notebook. See the pretrained models section for the pretrained models.

A recorded talk detailing the approach is available on youtube. [video]

The corresponding slides are available on slideshare. [slides]

Pretrained models:

You can get the models by running python get_models.py:

  • {deletion, insertion, substitute}_costs.txt: text files containing matrixes of weights used by the weighted edit distance (in class OcrDistanceMeasure). Files were generated with this notebook.

  • paragraph_segmentation2.params generates a region of handwritten text. It was generated with this file with the following commands:

    • python -m ocr.paragraph_segmentation_dcnn -g 0 -r 0.001 -e 181 -n cnn_mse.params -y 0.15
    • python -m ocr.paragraph_segmentation_dcnn -g 0 -r 0.0001 -l iou -e 150 -n cnn_iou.params -f cnn_mse.params
  • word_segmentation2.params generates word crops. It was generated with this file. With the following command: python -m ocr.word_and_line_segmentation--min_c 0.01 --overlap_thres 0.10 --topk 150 --epoch 401

  • handwriting_line_sl_160_a_512_o_2.params: pre-trained models for CNN-biLSTM for handwriting detection. Model was generated with this file. With the following command: python -m ocr.handwriting_line_recognition --epochs 501 -n handwriting_line.params -g 1 -l 0.0001 -x 0.1 -y 0.1 -j 0.15 -k 0.15 -p 0.75 -o 2 -a 512 -sl 160 -g 1

Sample results

The greedy, lexicon search, and beam search outputs present similar and reasonable predictions for the selected examples. In Figure 6, interesting examples are presented. The first line of Figure 6 show cases where the lexicon search algorithm provided fixes that corrected the words. In the top example, “tovely” (as it was written) was corrected “lovely” and “woved” was corrected to “waved”. In addition, the beam search output corrected “a” into “all”, however it missed a space between “lovely” and “things”. In the second example, “selt” was converted to “salt” with the lexicon search output. However, “selt” was erroneously converted to “self” in the beam search output. Therefore, in this example, beam search performed worse. In the third example, none of the three methods significantly provided comprehensible results. Finally, in the forth example, the lexicon search algorithm incorrectly converted “forhim” into “forum”, however the beam search algorithm correctly identified “for him”.

Dataset:

Appendix

1) Handwritten area

Model architecture

Results

2) Line Detection

Model architecture

Results

3) Handwritten text recognition

Model architecture

Results

SClite installation

  1. Download sctk-2.4.10 ftp://jaguar.ncsl.nist.gov/pub/sctk-2.4.10-20151007-1312Z.tar.bz2
  2. Put it into the utils folder
  3. Untar sctk-2.4.10
  4. Install sctk-2.4.10 by following sctk-2.4.10/INSTALL
  5. Check sctk-2.4.10/bin contains built programs