A – Filter Description

Orbbec SDK implementation includes post-processing filters to enhance the quality of depth data and to reduce noise levels. All the filters are implemented in the library core as independent modules to be used in the customer code.

A1 – Decimation Filter

A1A – Function

This filter effectively reduces the depth scene complexity.
The filter runs on kernel sizes [2×2] to [8×8] pixels. The image size is scaled down proportionally in both dimensions to preserve the aspect ratio. After the resulted frame is produced, the frame intrinsic parameters are recalculated to compensate for the resolution changes.
The filter also provides some hole filling capability, as the filter uses valid (non-zero) pixels only.

A1B – Parameters

A1C – Data Domain

Applicable on both depth and disparity maps.

A2 – HDR Merge Filter

A2A – Function

This filter is used jointly with depth HDR function. By merging consecutive depth images of alternating exposure values, we can over come challenges in acquiring depth values for under-illuminated and over-illuminated objects simultaneously.

A2B- Parameters

None.

A2C – Data Domain

Applicable on both depth and disparity maps.

A3 – Sequence ID Filter

A3A – Function

This filter is used jointly with depth HDR function and outputs only the sequence with specified sequence id.

A3B- Parameters

A3C – Data Domain

Applicable on both depth and disparity maps.

A4 – Threshold Filter

A4A – Function

This filter preserves depth values of interest and omits depth values out of scope. The depth threshold values Min and Max are both defined in millimeters. As this definition implies, the threshold filter operates only on depth maps. This filter is automatically bypassed if applied on disparity maps.

A4B- Parameters

A4C – Data Domain

Applicable only on depth maps.

A5 – Noise Removal Filter

A5A – Function

This filter removes noise pixels and gives rise to a less-filled map. There are two parameters Min Diff and Max Size defined for tuning the filtering performance.

Min Diff is the parameter regulating spatial filtering range in depth. A large Min Diff value results in less effective removal performance and thus preserves more depth information after filtering.

Max Size is the parameter for distinguishing between noise and non-noise pixels. A large Max Size value amplifies filtering performance and leads to removal operation of more significant sizes.

A5B- Parameters

A5C – Data Domain

Applicable on both depth and disparity maps.

A6 – Spatial Filter

A6A – Function

This filter performs multiple iterations of processing as specified by the magnitude parameter, to enhance the smoothness of depth data. It is also capable of filling small holes in depth maps.

The Alpha parameter determines the significance of smoothing operation. This filter is also capable of preserving edges, where the Diff Threshold parameter distinguishes between homogeneous sections and edges. The smoothing operation is repeated for multilple times, where the Magnitude parameter defines the number of interations. In each iteration, holes will be filled using valid pixels in a neighborhood of a specified radius.

A6B- Parameters

A6C – Data Domain

Applicable on both depth and disparity maps.

A7 – Temporal Filter

A7A – Function

This filter is intended to improve the depth data persistency by manipulating per-pixel values based on previous frames.

The filter performs a single pass on the data, adjusting the depth values while also updating the tracking history.

Note that due to its reliance on historic data the filter may introduce visible blurring/smearing artefacts, and therefore is best-suited for static scenes.

A7B- Parameters

A7C – Data Domain

Applicable on both depth and disparity maps.

A8 – Hole Filling Filter

A8A – Function

This filter fills all holes in the depth map using the specified mode.

A8B- Parameters

A8C – Data Domain

Applicable on both depth and disparity maps.

B – Practice Considerations

Post-processing modules are encapsulated into self-contained processing blocks, which can thus meet the following key requirements:

1. Synchronous/Asynchronous invocation
2. Internal frames memory/lifetime management

All post-processing filters are capable to receive and process frames from different sources, though in the general case it is not practical for the following reasons:

• Performance overhead as each time a new type/source of the frame is recognized, some filters require re-initialization.
• The Temporal filter effectiveness relies on the frames history being preserved. Switching the frame sources invalidates the preserved history and incapacitates the filter.

Therefore establishing and maintaining a filter pipe per camera source is strongly recommended.

C – Use Post-processing Filters

C1 – Best Practice

The recommended scheme for post-processing is elaborated below:

• Depth Frame: essential, input depth frame for post-processing

=>

• Decimation Filter: optional, reducing complexity by subsampling depth maps and losing depth details

=>

• HDR Merge Filter: optional, combined with depth HDR function and merging consecutive depth images of alternating exposure values

=>

• Sequence ID Filter: optional, combined with depth HDR function and selecting specific sequence

=>

• Threshold Filter: optional, preserving depth values of interest and omitting depth values out of scope

=>

• Noise Removal Filter: essential, removing scattering depth pixels

=>

• Spatial Filter: optional, spatially smoothing adjacend depth pixels while preserving edge information

=>

• Temporal Filter: optional, temporally smoothing depth values to improve their temporal consistency

=>

• Hole Filling Filter: not recommended, recovering missing depth values from pixels in neighborhood

=>

• Filtered Depth: essential, output depth frame after post-processing.

The default settings for post-processing filters in Orbbec SDK tools and demos is listed below:

C2 – Orbbec Viewer

Orbbec Viewer –  Post-processing Filters

C3 – Orbbec SDK

C3A – C APIs and Code Sample

This section describes only two selected post-processing filters. For other filters, please refer to libobsensor\h\Filter.h:

// header file: libobsensor\h\Filter.h

/**
* @brief Create a decimation filter.
* @param[out] error Log error messages.
* @return A depth_filter object.
*/
ob_filter *ob_create_decimation_filter(ob_error **error);

/**
* @brief Get the decimation filter scale range.
*
* @param[in] filter A decimation filter object.
* @param[out] error Log error messages.
*/
ob_uint8_property_range ob_decimation_filter_get_scale_range(ob_filter *filter, ob_error **error);

/**
* @brief Set the decimation filter scale value.
*
* @param[in] filter A decimation object.
* @param[in] value decimation filter scale value.
* @param[out] error Log error messages.
*/
void ob_decimation_filter_set_scale_value(ob_filter *filter, uint8_t value, ob_error **error);

/**
* @brief Get the decimation filter scale value.
*
* @param[in] filter A decimation object.
* @param[out] error Log error messages.
* @return decimation filter scale value.
*/
uint8_t ob_decimation_filter_get_scale_value(ob_filter *filter, ob_error **error);

// header file: libobsensor\h\Filter.h

/**
* @brief Create a noise removal filter.
* @param[out] error Log error messages.
* @return A depth_filter object.
*/
ob_filter *ob_create_noise_removal_filter(ob_error **error);

/**
* @brief Get the noise removal filter disp diff range.
*
* @param[in] filter A noise removal filter object.
* @param[out] error Log error messages.
* @return ob_uint16_property_range the disp_diff value of property range.
*/
ob_uint16_property_range ob_noise_removal_filter_get_disp_diff_range(ob_filter *filter, ob_error **error);

/**
* @brief Get the noise removal filter max size range.
*
* @param[in] filter noise removal filter object.
* @param[out] error Log error messages.
* @return ob_int_property_range the _max_size value of property range.
*/
ob_int_property_range ob_noise_removal_filter_get_max_size_range(ob_filter *filter, ob_error **error);

/**
* @brief Set the noise removal filter params.
*
* @param[in] filter noise removal filter object.
* @param[in] params ob_noise_removal_filter_params.
* @param[out] error Log error messages.
*/
void ob_noise_removal_filter_set_filter_params(ob_filter *filter, ob_noise_removal_filter_params params, ob_error **error);

/**
* @brief Get the noise removal filter params.
*
* @param[in] filter noise removal filter object.
* @param[out] error Log error messages.
* @return ob_noise_removal_filter_params.
*/
ob_noise_removal_filter_params ob_noise_removal_filter_get_filter_params(ob_filter *filter, ob_error **error);

#include "window.hpp"
#include <iostream>

extern "C" {
#include <stdlib.h>
#include <libobsensor/h/Error.h>
#include <libobsensor/h/Frame.h>
#include <libobsensor/h/ObTypes.h>
#include <libobsensor/h/Pipeline.h>
#include <libobsensor/h/StreamProfile.h>
#include <libobsensor/h/Device.h>
}

/*
*This sample is written in C++ code, based on the C language version API of OrbbecSDK.
*/

void check_error(ob_error *error) {
if(error) {
printf("ob_error was raised: \n\tcall: %s(%s)\n", ob_error_function(error), ob_error_args(error));
printf("\tmessage: %s\n", ob_error_message(error));
printf("\terror type: %d\n", ob_error_exception_type(error));
ob_delete_error(error);
exit(EXIT_FAILURE);
}
}

int main(int argc, char **args) {
Window      *win      = nullptr;  // render window, based on opencv
ob_error    *error    = NULL;     // Used to return SDK interface error information
ob_pipeline *pipeline = nullptr;  // pipeline, used to open the depth stream after connecting the device

// Create a pipeline to open the depth stream after connecting the device
pipeline = ob_create_pipeline(&error);
check_error(error);

// Create config to configure the resolution, frame rate, and format of the depth stream
ob_config *config = ob_create_config(&error);
check_error(error);

// Configure the depth stream
ob_stream_profile      *depth_profile = NULL;
ob_stream_profile_list *profiles      = ob_pipeline_get_stream_profile_list(pipeline, OB_SENSOR_DEPTH, &error);
check_error(error);

// Find the corresponding profile according to the specified format, first look for the y16 format
depth_profile = ob_stream_profile_list_get_video_stream_profile(profiles, 640, OB_HEIGHT_ANY, OB_FORMAT_Y16, 30, &error);
// If the specified format is not found, search for the default profile to open the stream
if(error) {
depth_profile = ob_stream_profile_list_get_profile(profiles, OB_PROFILE_DEFAULT, &error);
ob_delete_error(error);
error = nullptr;
}

// enable stream
ob_config_enable_stream(config, depth_profile, &error);
check_error(error);

ob_filter* dec_filter = ob_create_decimation_filter(&error);
check_error(error);

//When the value is set to 1, it indicates no decimation.
ob_decimation_filter_set_scale_value(dec_filter, 1, &error);
check_error(error);

//Create noiseRemovalFilter
ob_filter *noise_filter = ob_create_noise_removal_filter(&error);
check_error(error);

//Get the dispDiff range, it includes maximum, minimum, step size, current, and default values.
ob_uint16_property_range disp_diff_range = ob_noise_removal_filter_get_disp_diff_range(noise_filter, &error);
check_error(error);
//Get the disp maxSize range,it includes maximum, minimum, step size, current, and default values.
ob_int_property_range max_size_range = ob_noise_removal_filter_get_max_size_range(noise_filter, &error);
check_error(error);
ob_noise_removal_filter_params filter_params = ob_noise_removal_filter_get_filter_params(noise_filter, &error);
check_error(error);
//Assign values based on the range of dispDiff.
filter_params.disp_diff = 10;
//Assign values based on the range of maxSize.
filter_params.max_size = 100;
//set params
ob_noise_removal_filter_set_filter_params(noise_filter, filter_params, &error);
check_error(error);

std::vector<ob_filter *> ob_filters;
ob_filters.push_back(dec_filter);
ob_filters.push_back(noise_filter);

// Start the pipeline with config
ob_pipeline_start_with_config(pipeline, config, &error);
check_error(error);

// Create a window for rendering, and set the resolution of the window
uint32_t width = ob_video_stream_profile_width(depth_profile, &error);
check_error(error);
uint32_t height = ob_video_stream_profile_height(depth_profile, &error);
check_error(error);
win = new Window("DepthViewer", width, height);
check_error(error);

bool resize_win = ob_filter_is_enable(dec_filter, &error);
check_error(error);

// Wait in a loop, exit after the window receives the "esc" key
while(*win) {
// Wait for up to 100ms for a frameset in blocking mode.
ob_frame *frameset = ob_pipeline_wait_for_frameset(pipeline, 100, &error);
check_error(error);

if(frameset == nullptr) {
continue;
}

ob_frame *depth_frame = ob_frameset_depth_frame(frameset, &error);
check_error(error);

if(depth_frame != nullptr) {
for(ob_filter *filter: ob_filters) {
depth_frame = ob_filter_process(filter, depth_frame, &error);
check_error(error);
}
}

uint32_t width, height;
if(depth_frame != nullptr) {
// for Y16 format depth frame, print the distance of the center pixel every 30 frames
uint32_t index = ob_frame_index(depth_frame, &error);
check_error(error);
ob_format format = ob_frame_format(depth_frame, &error);
check_error(error);
if(index % 30 == 0 && format == OB_FORMAT_Y16) {
uint32_t width = ob_video_frame_width(depth_frame, &error);
check_error(error);
uint32_t height = ob_video_frame_height(depth_frame, &error);
check_error(error);
float scale = ob_depth_frame_get_value_scale(depth_frame, &error);
check_error(error);
uint16_t *data = (uint16_t *)ob_frame_data(depth_frame, &error);
check_error(error);

// pixel value multiplied by scale is the actual distance value in millimeters
float center_distance = data[width * height / 2 + width / 2] * scale;

// attention: if the distance is 0, it means that the depth camera cannot detect the object（may be out of detection range）
printf("Facing an object %.2f mm away.\n", center_distance);
}

if(resize_win) {
uint32_t width = ob_video_frame_width(depth_frame, &error);
check_error(error);
uint32_t height = ob_video_frame_height(depth_frame, &error);
check_error(error);
win->resize(width, height);
resize_win = false;
}

// add frame to render
// attention: the frame will be released inside the window，for user's code should release it by call ob_delete_frame()
win->addToRender(depth_frame);
}
ob_delete_frame(frameset, &error);
check_error(error);
};

for(ob_filter *filter: ob_filters) {
ob_delete_filter(filter, &error);
check_error(error);
}

// stop the pipeline
ob_pipeline_stop(pipeline, &error);
check_error(error);

// destroy the window
delete win;

// destroy profile
ob_delete_stream_profile(depth_profile, &error);
check_error(error);

// destroy profile list
ob_delete_stream_profile_list(profiles, &error);
check_error(error);

// destroy the pipeline
ob_delete_pipeline(pipeline, &error);
check_error(error);

return 0;
}

C3B – C++ APIs and Code Sample

This section describes only two selected post-processing filters. For other filters, please refer to libobsensor\hpp\Filter.hpp:

// header file: libobsensor\hpp\Filter.hpp

/**
* @brief Decimation filter,reducing complexity by subsampling depth maps
* and losing depth details.
*/
class OB_EXTENSION_API DecimationFilter : public Filter {
public:
DecimationFilter();

/**
* @brief Set the decimation filter scale value.
*
* @param type The decimation filter scale value.
*/
void setScaleValue(uint8_t value);

/**
* @brief Get the decimation filter scale value.
*/
uint8_t getScaleValue();

/**
* @brief Get the property range of the decimation filter scale value.
*/
OBUint8PropertyRange getScaleRange();
};

// header file: libobsensor\hpp\Filter.hpp

/**
* @brief The noise removal filter, removing scattering depth pixels.
*/
class OB_EXTENSION_API NoiseRemovalFilter : public Filter {
public:
NoiseRemovalFilter();

/**
* @brief Set the noise removal filter params.
*
* @param[in] params ob_noise_removal_filter_params.
*/
void setFilterParams(OBNoiseRemovalFilterParams filterParams);

/**
* @brief Get the noise removal filter params.
*
* @return OBNoiseRemovalFilterParams.
*/
OBNoiseRemovalFilterParams getFilterParams();

/**
* @brief Get the noise removal filter disp diff range.
* @return OBUint16PropertyRange The disp diff of property range.
*/
OBUint16PropertyRange getDispDiffRange();

/**
* @brief Get the noise removal filter max size range.
* @return OBUint16PropertyRange The max size of property range.
*/
OBUint16PropertyRange getMaxSizeRange();
};

#include "window.hpp"

#include "libobsensor/hpp/Pipeline.hpp"
#include "libobsensor/hpp/Error.hpp"

int main(int argc, char **argv) try {
// Create a pipeline with default device
ob::Pipeline pipe;

// Get all stream profiles of the depth camera, including stream resolution, frame rate, and frame format
auto profiles = pipe.getStreamProfileList(OB_SENSOR_DEPTH);

std::shared_ptr<ob::VideoStreamProfile> depthProfile = nullptr;
try {
// Find the corresponding profile according to the specified format, first look for the y16 format
depthProfile = profiles->getVideoStreamProfile(640, OB_HEIGHT_ANY, OB_FORMAT_Y16, 30);
}
catch(ob::Error &e) {
// If the specified format is not found, search for the default profile to open the stream
depthProfile = std::const_pointer_cast<ob::StreamProfile>(profiles->getProfile(OB_PROFILE_DEFAULT))->as<ob::VideoStreamProfile>();
}

// By creating config to configure which streams to enable or disable for the pipeline, here the depth stream will be enabled
std::shared_ptr<ob::Config> config = std::make_shared<ob::Config>();
config->enableStream(depthProfile);

ob::DecimationFilter decFilter;
// When the value is set to 1, it indicates no decimation.
decFilter.setScaleValue(1);
ob::NoiseRemovalFilter noiseFilter;
//open it
noiseFilter.enable(true);
//Get the dispDiff range, it includes maximum, minimum, step size, current, and default values.
OBUint16PropertyRange dispDiffRange = noiseFilter.getDispDiffRange();
//Get the disp maxSize range,it includes maximum, minimum, step size, current, and default values.
OBUint16PropertyRange dispMaxSizeRange = noiseFilter.getMaxSizeRange();
OBNoiseRemovalFilterParams filterParams = noiseFilter.getFilterParams();
//Assign values based on the range of dispDiff.
filterParams.disp_diff = 10;
//Assign values based on the range of maxSize.
filterParams.max_size = 100;
//set params
noiseFilter.setFilterParams(filterParams);

//Add filter to list
std::vector<ob::Filter> postFilters;
postFilters.push_back(decFilter);
postFilters.push_back(noiseFilter);

// Start the pipeline with config
pipe.start(config);

// Create a window for rendering, and set the resolution of the window
Window app("PostProcessing", depthProfile->width(), depthProfile->height());

bool resizeWindow = false;
if(decFilter.isEnabled()) {
resizeWindow = true;
}
while(app) {
// Wait for up to 100ms for a frameset in blocking mode.
auto frameSet = pipe.waitForFrames(100);
if(frameSet == nullptr) {
continue;
}

auto depthFrame = frameSet->depthFrame();
if(depthFrame) {
for(ob::Filter postFilter: postFilters) {
auto newFrame = postFilter.process(depthFrame);
depthFrame    = newFrame->as<ob::DepthFrame>();
}
}

// for Y16 format depth frame, print the distance of the center pixel every 30 frames
if(depthFrame->index() % 30 == 0 && depthFrame->format() == OB_FORMAT_Y16) {
uint32_t  width  = depthFrame->width();
uint32_t  height = depthFrame->height();
float     scale  = depthFrame->getValueScale();
uint16_t *data   = (uint16_t *)depthFrame->data();

// pixel value multiplied by scale is the actual distance value in millimeters
float centerDistance = data[width * height / 2 + width / 2] * scale;

// attention: if the distance is 0, it means that the depth camera cannot detect the object（may be out of detection range）
std::cout << "Facing an object " << centerDistance << " mm away. " << std::endl;
}

if(resizeWindow) {
app.resize(depthFrame->width(), depthFrame->height());
resizeWindow = false;
}

// Render frame in the window
app.addToRender(depthFrame);
}

// Stop the pipeline
pipe.stop();

return 0;
}
catch(ob::Error &e) {
std::cerr << "function:" << e.getName() << "\nargs:" << e.getArgs() << "\nmessage:" << e.getMessage() << "\ntype:" << e.getExceptionType() << std::endl;
exit(EXIT_FAILURE);
}

C4 – ROS  Wrapper

C4A – ROS 1

To enable or disable post-processing filters in a ROS environment, modify the launch file parameters as shown below. Adjust each parameter according to the recommendations in the Best Practice section.

<launch>
<!-- Additional pre-existing configurations... -->

<!-- Configuration of post-processing filters -->
<arg name="enable_decimation_filter" default="false"/>
<arg name="enable_hdr_merge" default="false"/>
<arg name="enable_sequenced_id_filter" default="false"/>
<arg name="enable_threshold_filter" default="false"/>
<arg name="enable_noise_removal_filter" default="true"/>
<arg name="enable_spatial_filter" default="false"/>
<arg name="enable_temporal_filter" default="false"/>
<arg name="enable_hole_filling_filter" default="false"/>

<!-- Additional subsequent configurations... -->
</launch>

C4B – ROS 2

For ROS 2 applications, follow the instructions declarations in your launch file to configure post-processing filters

# Other pre-existing configurations...

# Configuration of post-processing filters
DeclareLaunchArgument('enable_decimation_filter', default_value='false'),
DeclareLaunchArgument('enable_hdr_merge', default_value='false'),
DeclareLaunchArgument('enable_sequence_id_filter', default_value='false'),
DeclareLaunchArgument('enable_threshold_filter', default_value='false'),
DeclareLaunchArgument('enable_noise_removal_filter', default_value='true'),
DeclareLaunchArgument('enable_spatial_filter', default_value='false'),
DeclareLaunchArgument('enable_temporal_filter', default_value='false'),
DeclareLaunchArgument('enable_hole_filling_filter', default_value='false'),

# Additional subsequent configurations...

C5 – Example Captures under Typical Configurations

C5A – Gemini 335 & 330

Configuration	Enabled filters of default settings: Noise Removal	Enabled filters of default settings: Noise Removal & Spatial & Temporal
point cloud
point cloud with RGB texture

C5B – Gemini 335L & 330L

Configuration	Enabled filters of default settings: Noise Removal	Enabled filters of default settings: Noise Removal & Spatial & Temporal
point cloud
point cloud with RGB texture