Handwritten Multi-Digit Number Recognition

Introduction

In this project, I built a model to perform handwritten digit string recognition using synthetic data generated by concatenating digits from the MNIST dataset. Different overlapping rates and paddings were used to increase the generalizability of the model. Data augmentation (e.g., random scaling, rotation, shear) was also used during training to increase the divesity of the samples.

Here are some examples of the synthetic data:

I replicated the CRNN model proposed in Shi, Bai & Yao (2015), which uses a CNN as encoder and a bidirecitonal LSTM as decoder, along with a CTC loss. I was able to achieve a character error rate of 0.0114 and an accuracy of 0.9865 in the test set (see the best run here).

Model architecture, from Shi, Bai & Yao (2015)

For deployment, I built a Docker image, pushed it to Amazon Elastic Container Registry (ECR), created a AWS Lambda function from the container image, and finally, put a REST API gateway in front of the Lambda function.

Setup

Create a virtual environment.

make venv

Install dependencies.

make install-dev

Usage

Run an experiement

Example command to run an experiment:

python scripts/run_experiment.py \
    trainer.gpus=1 trainer.max_epochs=100 trainer.auto_lr_find=True \
    data.batch_size=128 data.num_workers=2 data.pin_memory=True \
    data.num_train=20000 data.num_val=2000 data.num_test=2000

Running scripts/run_experiment.py will download the MNIST dataset, generate a synthetic dataset and train a CRNN model. The best model checkpoint will be uploaded to Weights & Biases (W&B) automatically (you will be asked to register or login to W&B before the training starts).

Configurations can be modified in config.yaml or in command line. See Hydra's documentation to learn more.

Example command to download a trained model checkpoint from W&B

python scripts/download_checkpoint.py RUN_PATH

Replace RUN_PATH with the path of your run. The run path can be found in the W&B dashboard and should look something like kingyiusuen/handwritten-multi-digit-number-recognition/3r47q0f5

Docker Image for AWS Lambda

Build a Docker image for AWS Lambda:

docker build -t handwritten-multi-digit-number-recognition -f aws_lambda/Dockerfile .

Run the Docker image:

docker run -p 9000:8080 -it --rm handwritten-multi-digit-number-recognition

Example command to test the API locally:

curl -X POST "http://localhost:9000/2015-03-31/functions/function/invocations" -H 'Content-Type: application/json' -d '{ "image": "data:image/png;base64,'$(base64 -i images/test_image.png)'" }'

Acknowledgements

Full Stack Deep Learning Spring 2021 Labs