Author: gelk (page 1 of 2)

Running Artificial Intelligence Algorithms Directly on Jetson Nano and Microcontrollers

Running artificial intelligence (AI) algorithms, such as neural networks, directly on embedded devices has many advantages compared to running them in the cloud: One can save significant amounts of cloud storage, reduce power consumption and enable real-time applications. In addition, privacy is increased and required bandwidth reduced because only the AI algorithms results are forwarded to the cloud, not the full data. However, setting up the environments for custom neural networks on embedded devices can be difficult. Thats why the HPMM team provides a fully “dockerized” reference workflow for the Nvidia Jetson Nano. It includes:

  • Container to convert Tensorflow and Pytorch models to .onnx models
  • Container to cross-compile a C++ TensorRT applications for a Jetson Nano including opencv
  • Container to run TensorRT networks on the Jetson Nano with the C++ and Python API

Please find the link to the reference workflow here.

If you are interested in running AI algorithms on microcontrollers, such as the Cortex M4, we provide the reference workflows for several frameworks such as TensorFlow lite for microcontrollers, CMSIS-NN and ST CUBE AI here.

Feel free to contact us regarding your custom application or project regarding embedded artificial intelligence!

USBC2: USB-C Two Port Card

USBC2: USB-C Two Port Card

Technical Data:

  • USB Type-C V1.0
  • Uses Xavier AGX USB MACs

Availability: In stock, request quote

NWG1: 1Gbit Ethernet Card

Technical Data:

  • GbE transceiver
  • 1000BASE-T, 100BASE-TX and 10BASE-T
  • Based on Marvel 88E1512

Availability: In Stock, request a quote

FPDI4: FPD-Link III four Input Card

FPDI4: FPD-Link III four Input Card

Technical Data:

  • CSI-2 data rate scalable to 400/800 Mbps/1.5 Gbps/1.6 Gbps
  • Allows connection of four active sensors
  • Power over coax
  • Supports single ended coaxial or shielded twisted-pair (STP) cable
  • Adaptive Receive Equalization
  • Based on TI DS90UB954-Q1

Availability: In stock, request a quote

HDMI4KO2: 4k HDMI two Output Card

HDMI4KO2: 4k HDMI two Output Card

Technical Data:

  • 2 x HDMI Type A HDMI2.0
  • 4k@60fps

Availability: In stock, request a quote

AGXC512: Modular Carrier Board for NVIDIA Jetson AGX

AGXC512:Modular Carrier Board for NVIDIA Jetson AGX
(Configuration example only, Jetson AGX and carrier boards not included)

Technical Data:

  • Power 12V
  • 1 x HDMI2.0 Output
  • 1 x USB 3.0
  • 1 x USB 2.0
  • 1 x USB-C
  • 1 x PCIe E-Key slot
  • 2 x PCIe B-Key slot
  • 4 x PCIe M-Key slot
  • 8 x 4-Lane MIPI-CSI

Availability: In stock, request a quote

HDMI4KI2: 4k HDMI two Input Card

Technical Data:

  • HDMI2.0b, HDMI1.4
  • compliant with HDCP2.2
  • 4k@30fps
  • 2k@ 60fps
  • YcBCr4:2:2, 16/20/24-bit
  • RGB, 16/20/24-bit
  • Based on Lontium LT6911UXC
  • NVIDIA Jetson driver available

Availability: In Stock, request a quote

Jamulus-Direct low latency music performance application for RaspberryPi 4

The Institute of Embedded Systems (InES) High Performance Multimedia group generated a low latency version of the classic Jamulus music rehearsal application for the Raspberry Pi 4

The classic Jamulus is an open source application for music groups who want to rehearse over the internet. Especially with the global pandemic, Jamulus is a great solution for bands and choirs to rehearse from home.
With the Jamulus-Direct solution, the Institute of Embedded Systems reduces the latency of the audio connection compared to the classic Jamulus. The low audio latency is achieved through multiple peer-to-peer connections. This means that each participant is connected to the other members of the group via a dedicated connection. The audio no longer needs to be sent via an audio server, which reduces the latency.

Peer-to-Peer communication in Jamulus Direct reduces latency

The figure below shows the audio transmission in a classical server based session with three clients. Three computers run a client software. Computer 0 starts a client and a server. Each client sends their own audio to the server. The audio is mixed together at the server and sent back to the clients. However, the server topology shown below introduces latency because all data has to make a detour over the server.

No peer-to-peer communication in classical Jamulus introduces latency

Therefore, in Jamulus direct, a peer-to-peer topology eliminates the detour via the server (see figure below). Each participant in a peer-to-peer system exchanges its data with each other participant. Peer-to-peer is therefore the preferred structure to achieve the lowest possible latency.

Peer to peer connections between three clients

This setup ensures the lowest possible latency between each device.
To achieve low latency, a few challenges had to be mastered to get a well functioning system.
For instance, that peer to peer connections usually are blocked by the firewall of the network router and therefore need a mechanism to open these ports on the firewall.
A further issue is the management of the session. In server based systems, the server usually manages the session. It is the contact point for new clients to register to join the session and also to unregister when leaving.
The peer-to-peer audio transmission showed audio latencies under 30 milliseconds for two locations, with a ping time of 13 milliseconds.
A detailed paper of Jamulus Direct can be downloaded here.

Jamulus-Direct is written for the Raspberry Pi 4 and achieves high quality audio utilizing a USB audio card connected via the RPI4 USB interface. Every audio card with available Linux drivers is suitable (Tested with Focusrite Scarlet 2i2 and Behringer U-Phoria UM2)

There are two ways you can use Jamulus-Direct:
1.) Burn Jamulus-Direct onto a SD-card and dedicate your Raspberry Pi for Jamulus-Direct
(Instructions for dedicated SD-card installation)

2.) Use Jamulus on your existing Raspberry Pi4 with Raspberry Pi OS already installed
(Instructions with already installed Raspberry Pi OS)

Nvidia Xavier-AGX Modular Vision System

The Institute of Embedded Systems (InES) at ZHAW, with great experience in hardware development for NVIDIA computing modules, is now shipping its modular vision system based on high-performance NVIDIA Xavier AGX.

To shorten the time to market, the prototyping system consists of a greatly reduced motherboard that can be equipped with different types of M2-footprint modules to add functions like HDMI in and out, FPD-Link III in and out, USB-c etc.

Due to the modular architecture, the user may configure a personalized system by adding from a choice of different modules. Due to the low complexity and flexibility of the provided interfaces, more custom-made modules can easily be developed.

AGXC512: Nvidia-Xavier-AGX Carrier Board
(Configuration Example)
HDMI4KI2: 4k HDMI card with 2 Inputs
HDMI4KO2: 4k HDMI card with 2 outputs
FPDI4: FPD-Link III deserializer card with 4 inputs
USBC2: USB-C card with 2 x USB-C
NWG1: GBit Ethernet card

The system consists of a minimal mother board, which only includes the necessary circuitry to startup and program the Xavier.

To keep size and costs low, the main board includes USB 2.0, HDMI and one 16-lane PCIe standard card slot. All other interfaces are provided by nine PCIe M2-footprint sockets which share not only PCIe lines, but also dedicated Xavier-AGX interfaces like MIPI- CSI, USB and I2C. An overview of possible configurations and currently available devices is given in the table below.

Front view with Nvidia-Xavier-AGX on top
Slot connections to Xavier and currently available Modules
Modules may easily be exchanged
  • The Video-Out socket provides two HDMI or DisplayPort lines, as well as the required I2C lines. It is also possible to configure one DisplayPort and one HDMI output. This slot also carries USB2.0, if the user wants to implement HD-BaseT.
  • The USB socket allows two USB interfaces with USB connectors of choice (USB-A-B-C, USB on the go) I2C, CAN-Bus and serial UART are also supported.
  • Video-In Module 1 and 2 each support 2 CSI ports with 4 lanes or 4 CSI ports with 2 lanes which allows connecting 4k cameras or HDMI to CSI converters
  • The Network Module allows the connection of 1GBIT Ethernet over RGMII or another PCIe lane for PCIe high speed PHYs
Available Modules for Xavier prototyping board

For more information, availability and pricing, please see contacts on the right.

Linux Driver for the LT6911UXC HDMI to MIPI CSI-2 Converter

The ZHAW Institute of Embedded Systems (InES), High Performance Multimedia Group, developed a 4k video for Linux driver for the Lontium LT6911UXC HDMI to MIPI CSI-2 converter IC.
The driver was written for NVIDIA Jetson Processors and enables the following features of the LT6911UXC

  • Supports 4k HDMI 2.0 to MIPI CSI-2, requiring only one CSI port
  • Up to 4k resolution
  • Only 4 CSI lanes (one port) are required to receive 4k@30fps
  • Converts 4:2:2 YcBCr to CSI-2 YUV streams 1)
  • Converts RGB to RGB CSI-2 streams 1)

A driver for the advanced LT6911GX with HDMI 2.1 support and 4k@60fps on a single CSI-2 port is in the pipeline at InES-HPMM group.

The driver can be downloaded from here:

1) If RGB or YUV streams are accepted depends on the corresponding LT6911UXC firmware provided by Lontium. Contact Lontium for the firmware.

Keywords: TC358743, HDMI to CSI converter, HDMI to CSI bridge, LT6911UXC, LT6911GX, MIPI, Jetson, Xavier

« Older posts