The Matroska (.mkv) container format is used for recording sensor data to allow multiple tracks using various codecs. Recordings contain tracks for storing color, depth, IR images, and IMU data.

High level details on the .mkv container format can be found on the Matroska website.

Using Third Party Tools

Tools like ffmpeg or the mkvinfo command from the MKVToolNix toolkit can be used to inspect and extract information from recording files.

For example, the following command extracts just the depth track as a 16-bit PNG sequence into the same folder:

ffmpeg -i output.mkv -map 0:1 -vsync 0 depth%04d.png

The -map 0:1 parameter extracts track index 1 which is depth for most recordings. -map 0:0 would be used if the recording did not contain a color track.

The -vsync 0 parameter forces ffmpeg to extract frames as is rather than trying to match 30fps, 15fps, or 5fps frame rates.

IMU Sample Structure

If extracting IMU data from the file without using the playback API, the data will be in a binary format. Below is the structure of the IMU data. All fields are little endian:

Identifying Tracks

It may be necessary to identify which track contains color, depth, IR, etc. when using third party tools to read Matroska files. Track numbers vary based on camera mode and which set of tracks is enabled. Tags are used to identify the meaning of each track.

The following tags will be attached to the specific Matroska elements that can be used to locate the corresponding track or attachment. Tools like ffmpeg and mkvinfo can be used to view these tags.