Embedded devices like the Nvidia Tegra X1/2 offer tremendous video processing capabilities. But often there are bottlenecks hindering you from taking advantage of their full potential. One solution to this problem is to employ the general purpose compute capabilities of the GPU (GPGPU). For this purpose, we have developed a GStreamer Plug-In that lets you add a customized video processing functionality to any pipeline with full GPU support.

A possible application is shown in the image below. Two video inputs are combined to a single video output as a picture-in-picture video stream. A 4k image is depicted in the background and on top of it a downscaled FullHD input is streamed.

In order to cope with the huge amount of data, the video processing is outsourced to the GPU. The use of CUDA allows you to create new algorithms from scratch or integrate existing libraries. The plugin enables you to benefit of the unique architecture of the TX1/2, where CPU and GPU share access to the same memory. Therefore, memory access time is reduced  and unnecessary copies are avoided. The next image shows a pipeline of the example mentioned above.

At the beginning of the pipeline, where the data rates are the highest, the GPU and internal Hardware encoders are used. The CPU can then handle the compressed data easily and gives access to the huge number of existing GStreamer Plug-Ins. For example it is capable of preparing a live video stream for clients.

The GStreamer Plug-In can also serve as a basis for other applications like format conversion, debayering or video filters.

Feel free to contact us on this topic.

The HDMI2CSI board for capturing 4K HDMI now supports both TX1 and TX2. Video capturing is fully supported for resolutions up to 2160p30 on Input A and 1080p60 on Input B.

Driver development will continue on L4T 28.1. The previous 24.2.1 branch is considered deprecated.

Get started with the Readme: https://github.com/InES-HPMM/linux-l4t-4.4
and find detailed instructions (for building the Kernel etc.) on the Wiki: https://github.com/InES-HPMM/linux-l4t-4.4/wiki/hdmi2csi

Main changes:

• Driver for tc358840: Now using the updated version that is already in the 28.1 kernel (with a small modification)
• Device tree: Adapted to be compatible with 28.1 (if you come from previous L4T, please note the new way of flashing a device tree in U-Boot! Also the structure is different with separate repositories for kernel and device tree)
• Vi driver: Using the new version from Nvidia instead of our implementation, since it now supports “ganged mode” for combining multiple VI ports
• Custom resolutions: The EDID can be read and written from the Linux userspace (See [1]) to support different resolutions/timings on the fly

If you want to use Userptr/Dmabuf mode in GStreamer v4l2src, you still need to rebuild GStreamer. The reason is that GStreamer by default uses libv4l for the REQBUF ioctl. The libv4l implementation of this ioctl does NOT support userptr/dmabuf. But you can just build GStreamer without libv4l and it will use correct implementations for the ioctls and work.

Original release:

https://blog.zhaw.ch/high-performance/2016/06/01/open-source-driver-for-hdmi2csi-module-released/

The High-Performance Multimedia Group has developed a High Definition Multimedia Interface (HDMI®) to MIPI®Camera Serial Interface Type 2 (CSI-2) converter module (HDMI2CSI) as a plug in to the NVIDIA Jetson TX1 development kit.
The HDMI2CSI module supports 4K video resolution for next-generation embedded Ultra High Definition video applications. The HDMI2CSI module offers two 4K/2K HDMI video and audio streams to be simultaneously converted in MIPI CSI-2 video and TDM audio format that can be processed by the Jetson TX1 processor.
The Jetson TX1 board is equipped with 3 four-lanes MIPI high-speed camera serial interfaces (CSI-2) which are used by the HDMI2CSI board to input HDMI video. The module utilizes two MIPI CSI-2 ports of the Jetson TX1 board (8 lanes) to input a 4K HDMI video stream. For a second 2K HDMI video stream, the remaining MIPI CSI-2 port is used (4 lanes).
Eight channels of HDMI audio streams per HDMI input are also supported and can be transmitted over TDM or I2S.
4K capable drivers for the HDMI2CSI are available as open source.

Fig. 1: The HDMI2CSI Board

Fig. 2:  The HDMI2CSI Board attached to the NVidia TX1 Evaluation Board

Technical Data:

• Based on Toshiba TC358840 Camera Serial Interface converter ICs
• HDMI 1.4b
• 4096 x 2160 (4Kx2K) @ 24 fps
• 3840 x 2160, @ 30 fps
• 4 x I2S Audio Interface with 16, 18, 20 or 24-bit
• 8 Channel TDM (Time Division Multiplexed Audio Interface) with 16, 18, 20 or 24-bit
• HDCP 1.4 support
• EDID support via I2C

For purchasing orders please contact Pender Electronic Design

(c) ZHAW Institute of Embedded Systems