Running artificial intelligence (AI) algorithms, such as neural networks, directly on embedded devices has many advantages compared to running them in the cloud: One can save significant amounts of cloud storage, reduce power consumption and enable real-time applications. In addition, privacy is increased and required bandwidth reduced because only the AI algorithms results are forwarded to the cloud, not the full data. However, setting up the environments for custom neural networks on embedded devices can be difficult. Thats why the HPMM team provides a fully “dockerized” reference workflow for the Nvidia Jetson Nano. It includes:
Container to convert Tensorflow and Pytorch models to .onnx models
Container to cross-compile a C++ TensorRT applications for a Jetson Nano including opencv
Container to run TensorRT networks on the Jetson Nano with the C++ and Python API
Please find the link to the reference workflow here.
If you are interested in running AI algorithms on microcontrollers, such as the Cortex M4, we provide the reference workflows for several frameworks such as TensorFlow lite for microcontrollers, CMSIS-NN and ST CUBE AI here.
Feel free to contact us regarding your custom application or project regarding embedded artificial intelligence!
The Institute of Embedded Systems (InES)High Performance Multimedia group generated a low latency version of the classic Jamulus music rehearsal application for the Raspberry Pi 4
The classic Jamulus is an open source application for music groups who want to rehearse over the internet. Especially with the global pandemic, Jamulus is a great solution for bands and choirs to rehearse from home. With the Jamulus-Direct solution, the Institute of Embedded Systems reduces the latency of the audio connection compared to the classic Jamulus. The low audio latency is achieved through multiple peer-to-peer connections. This means that each participant is connected to the other members of the group via a dedicated connection. The audio no longer needs to be sent via an audio server, which reduces the latency.
Peer-to-Peer communication in Jamulus Direct reduces latency
The figure below shows the audio transmission in a classical server based session with three clients. Three computers run a client software. Computer 0 starts a client and a server. Each client sends their own audio to the server. The audio is mixed together at the server and sent back to the clients. However, the server topology shown below introduces latency because all data has to make a detour over the server.
Therefore, in Jamulus direct, a peer-to-peer topology eliminates the detour via the server (see figure below). Each participant in a peer-to-peer system exchanges its data with each other participant. Peer-to-peer is therefore the preferred structure to achieve the lowest possible latency.
This setup ensures the lowest possible latency between each device. To achieve low latency, a few challenges had to be mastered to get a well functioning system. For instance, that peer to peer connections usually are blocked by the firewall of the network router and therefore need a mechanism to open these ports on the firewall. A further issue is the management of the session. In server based systems, the server usually manages the session. It is the contact point for new clients to register to join the session and also to unregister when leaving. The peer-to-peer audio transmission showed audio latencies under 30 milliseconds for two locations, with a ping time of 13 milliseconds. A detailed paper of Jamulus Direct can be downloaded here.
Jamulus-Direct is written for the Raspberry Pi 4 and achieves high quality audio utilizing a USB audio card connected via the RPI4 USB interface. Every audio card with available Linux drivers is suitable (Tested with Focusrite Scarlet 2i2 and Behringer U-Phoria UM2)
The Institute of Embedded Systems (InES) at ZHAW, with great experience in hardware development for NVIDIA computing modules, is now shipping its modular vision system based on high-performance NVIDIA Xavier AGX.
To shorten the time to market, the prototyping system consists of a greatly reduced motherboard that can be equipped with different types of M2-footprint modules to add functions like HDMI in and out, FPD-Link III in and out, USB-c etc.
Due to the modular architecture, the user may configure a personalized system by adding from a choice of different modules. Due to the low complexity and flexibility of the provided interfaces, more custom-made modules can easily be developed.
The system consists of a minimal mother board, which only includes the necessary circuitry to startup and program the Xavier.
To keep size and costs low, the main board includes USB 2.0, HDMI and one 16-lane PCIe standard card slot. All other interfaces are provided by nine PCIe M2-footprint sockets which share not only PCIe lines, but also dedicated Xavier-AGX interfaces like MIPI- CSI, USB and I2C. An overview of possible configurations and currently available devices is given in the table below.
The Video-Out socket provides two HDMI or DisplayPort lines, as well as the required I2C lines. It is also possible to configure one DisplayPort and one HDMI output. This slot also carries USB2.0, if the user wants to implement HD-BaseT.
The USB socket allows two USB interfaces with USB connectors of choice (USB-A-B-C, USB on the go) I2C, CAN-Bus and serial UART are also supported.
Video-In Module 1 and 2 each support 2 CSI ports with 4 lanes or 4 CSI ports with 2 lanes which allows connecting 4k cameras or HDMI to CSI converters
The Network Module allows the connection of 1GBIT Ethernet over RGMII or another PCIe lane for PCIe high speed PHYs
For more information, availability and pricing, please see contacts on the right.
The ZHAW Institute of Embedded Systems (InES), High Performance Multimedia Group, developed a 4k video for Linux driver for the Lontium LT6911UXC HDMI to MIPI CSI-2 converter IC. The driver was written for NVIDIA Jetson Processors and enables the following features of the LT6911UXC
Supports 4k HDMI 2.0 to MIPI CSI-2, requiring only one CSI port
Up to 4k resolution
Only 4 CSI lanes (one port) are required to receive 4k@30fps
Converts 4:2:2 YcBCr to CSI-2 YUV streams 1)
Converts RGB to RGB CSI-2 streams 1)
A driver for the advanced LT6911GX with HDMI 2.1 support and 4k@60fps on a single CSI-2 port is in the pipeline at InES-HPMM group.