Building standalone model binaries for audio inference
For a recent problem I had to crawl and categorize a large number of audio files from the internet. As the bottleneck was deemed to be downstream link I decided to use a pre-trained event classification model and run inference on the CPU.
Another requirement was that the crawler were running on multiple server and consistent for several weeks. My choice of language for such infrastructure heavy task is Rust and I looked into ways how to deploy an inference model.
It turned out that the process for creating a standalone binary in Rust is pretty simple. Even better, wrapping a model pre-trained on a large event classification dataset is possible. This means that in the end I could just copy the binary to any x86 machines and run them, without the need to setup virtual environments or install any packages.
In this post I will demonstrate how to convert a pre-trained model to ONNX and put it into a binary. Our goal is to create a standalone executable taking samples from stdin and classifying them into 527 known audio classes.