Description
The Neural Processing Unit (NPU) improves the inference performance of neural networks. The NPU targets 8-bit and 16-bit integer quantized Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). The NPU supports 8-bit weights.
Arm delivers the hardware Register Transfer Level (RTL) of the NPU with an open-source driver and compiler. A neural network must be compiled offline using the open-source compiler to produce a command stream. The application invokes the driver, which communicates with the NPU to tell it where the command stream is and initiates the network traversal. The command stream describes the steps necessary for the NPU to execute the operators compiled into the command stream autonomously. When complete, the NPU raises an IRQ to the driver.
The driver programs the memory location of the command stream and other payloads into registers in the NPU. The Central Control (CC) processes the command stream.
The NPU includes a Direct Memory Access (DMA) controller that can read and write to external memory. When the NPU performs inferences, the DMA controller reads the neural network description. This description contains:
- The command stream
- Network weights
- Bias information
- Scale information
The DMA controller also transfers the Input Feature Maps (IFMs) and Output Feature Maps (OFMs) and NPU-private intermediate data that is also held in system memory.
During runtime, TensorFlow Lite (TFL) loads the the flatbuf file, in which the Offline Compiler has created an Ethos™–U55 command stream for each custom operator. The driver gives a pointer to this command stream so that the NPU hardware can execute it. This means that the entire network can be a single operator that is run fully on the Ethos™–U55. The NPU reads the data (weights, commands, IFMs, OFMs, bias and scale) autonomously using the DMA.
The NPU uses a working buffer in SRAM for IFMs and OFMs in flight. The Offline Compiler decides the scheduling of this buffer and codes it into the command stream. The NPU uses the DMA to read and write autonomously to this work buffer. The location of the buffer is set at runtime through registers, meaning the coding in the command stream is relative, not absolute.
The external interfaces that the NPU implements are:
- Two Arm® AMBA® 5 AXI Manager interfaces that provide the DMA controller with access to external memory. One read/write manager, M0, and one read-only manager, M1. This means the NPU can present two sets of transactions at the same time. The command, weight, bias, and scale channels can be mapped to either AXI Manager.
