Armednn - an efficient neural network inference engine

Table of Contents

Armednn: an efficient neural network inference engine

Armednn is an efficient neural network inference engine. It is written in C++11 and uses Eigen for linear algebra computation. It is designed to be modular and extensible. It is easy to add new operators and new features.

When I was working as a co-founder in a startup company, we were developing a online singing voice system, which runs inference on cpu-based servers. I tried to use existing inference engines, such as Tensorflow-Lite, Caffe, CNTK, etc. However, I found that they are not suitable for our application. They are either too slow or too difficult to use. So I decided to write my own inference engine. After several months of development, I have a working version of the engine. It is fast and easy to use. I hope it can be useful for other people.

The design

The core disign is modular, extensible and most importantly efficient.

• An operator is a function that takes hyperparameters, params and inputs and produces outputs.

• An armed operator is an operator that has hyperparameters and params.

• A node contains an armed operator, inputs and outputs.

• A graph is a collection of linked nodes.

• The compuatation part of the operators are implemented using Eigen, an efficient C++ template library for linear algebra.

Currently, the following operators are implemented:

• Dense
• Split
• Concat
• Conv1D
• LSTM

https://github.com/lmaxwell/Armednn/

User defined operators

For user defined operators, the user only needs to implement the operator function and register it.

   REGISTER_OP(Dense).add_config<uint32_t>("dim0","dimension 0")
                    .add_config<uint32_t>("dim1","dimension 1")
                    .add_config<std::string>("activation","activation function type")
                    .add_param("weight",{"dim0","dim1"})
                    .add_param("bias",{"1","dim1"})
                    .set_num_input("1")
                    .set_num_output("1");

After registering, the operator can be used by calling helper functions, e.g. make_op, make_node.

Build a Graph

For flexibility, I do not implement a graph class. Instead, a graph can be build step by step.

    int L=2000;
    int C=256;

    auto input_node=make_input("input");


    ConfigMap config;
    config.insert({"activation",{(std::string)"tanh"}});
    config.insert({"dim0",{(uint32_t)C}});
    config.insert({"dim1",{(uint32_t)C}});
    ParamMap param;
    param.insert({"weight",{Matrix::Identity(C,C)}});
    param.insert({"bias",{Matrix::Ones(1,C)}});

    Arm arm(config,param);

    auto dense_0=make_node("Dense",arm,input_node->output(),"dense-0");
    auto dense_1=make_node("Dense",arm,dense_0->output(),"dense-1");
    auto dense_2=make_node("Dense",arm,dense_1->output(),"dense-2");
    auto dense_3=make_node("Dense",arm,dense_2->output(),"dense-3");


    Matrix temp=Matrix::Identity(L,C);
    input_node->feed(temp);

    dense_0->run();
    dense_1->run();
    dense_2->run();
    dense_3->run();

This example builds graph by calling make_node. The graph is a chain of dense operators. The input is a matrix of size LC. The output is a matrix of size LC.

A node can be alternatively made by calling make_op and make_armed. The following code is equivalent to make_node("Dense",arm,input_node->output(),"dense-0");

    auto dense_1=make_op("Dense","dense-1");
    armed=make_armed(arm,std::move(dense_1));
    auto node1=make_node(input_node->output(),std::move(armed));

Other features

Stateful operators and memory sharing are also supported. See more details in the code https://github.com/lmaxwell/Armednn/