TensorFlow as a Service (TFaaS)
A general purpose framework (written in Go) to serve TensorFlow models. It provides reach and flexible set of APIs to efficiently access your favorite TF models via HTTP interface. The TFaaS supports JSON and ProtoBuffer data-formats.
The following set of APIs is provided:
/uploadto push your favorite TF model to TFaaS server
/deleteto delete your TF model from TFaaS server
/modelsto view existing TF models on TFaaS server
/jsonto serve TF model predictions in JSON data-format
/prototo serve TF model predictions in ProtoBuffer data-format
From deployment to production
➀ install docker image (TFaaS port is 8083)
docker run --rm -h `hostname -f` -p 8083:8083 -i -t veknet/tfaas
➁ upload your TF model to TFaaS server
curl -X POST http://localhost:8083/upload -F 'name=ImageModel' -F '[email protected]/path/params.json' -F '[email protected]/path/tf_model.pb' -F '[email protected]/path/labels.txt'
➂ get your predictions
curl https://localhost:8083/image -F '[email protected]/path/file.png' -F 'model=ImageModel'
Fore more information please visit curl client page.
Benchmark results on CentOS, 24 cores, 32GB of RAM serving DL NN with 42x128x128x128x64x64x1x1 architecture (JSON and ProtoBuffer formats show similar performance):
- 400 req/sec for 100 concurrent clients, 1000 requests in total
- 480 req/sec for 200 concurrent clients, 5000 requests in total
For more information please visit bencmarks page.
- Install instructions to build TFaaS from source code
- End-to-end example of serving TF model in Go-server
- CMS experiment use-case