Depth from HDR imaging

1.Introduction

For image sensors, the primary challenge lies in capturing both dimly lit indoor scenes and brightly illuminated outdoor settings in the same high-quality image, such as photographing a person standing by the window indoors. Typical image sensors generally adjust exposure or gain to ensure quality imaging in low-light or high-illumination conditions. Robots face various challenges in this regard, such as autonomous mobile robots (AMR) that must operate precisely in environments where sunlight and shadow coexist; robotic vacuum cleaners working near windows face issues with (bright and dark area in the same view)areas overly bright from sunlight and objects too dark from backlighting. In scenarios where dark and bright areas coexist, using long exposure could result in over-exposure of bright area, while short exposure might fail to capture details in darker area. The ORBBEC Gemini 330 Series stereoscopic cameras also encounter similar challenge because their 3D images are calculated from a pair of high quality, left and right images.

Figure 1. The scene of near windows. Bright and black area in the same view.

High Dynamic Range (HDR) imaging in computer vision and cinematography is a set of techniques aiming at handling enormously wide exposure range (i.e., images with greater contrast between light and dark) compared to traditional digital imaging technologies, by merging multiple frames. For 3D depth cameras, higher dynamic range means higher depth fill-rate and higher accuracy which will significantly enhance a robot’s obstacle avoidance capabilities. This paper introduces fundamental concepts related to HDR functionality and introduce how HDR is utilized in the ORBBEC Gemini 330 Series cameras.

2.Sensor Settings

2.1  Exposure and Gain

Exposure and gain are two main parameters in photography and imaging technologies that directly affect the quality and clarity of images.

The ORBBEC Gemini 330 series cameras individually control the exposure for visible RGB light and infrared pairs. For Orbbec depth or IR cameras, exposure is measured in microsecond (µs).  It range from 1 µs to approximately 166 ms. Longer exposure times can enhance image brightness and signal-to-noise ratio (SNR) in low-light environments, thereby improving image clarity, and vice versa. Indoor scenes generally benefit from longer exposure time, while outdoor scenes require much shorter one.

Figure 2. Effect of modifying camera exposure. The larger the exposure, the brighter the images.

Gain adjustment does not affect the amount of light collected by the sensor but amplifies the existing electrical signal. Increasing gain can enhance image brightness in low-light conditions but also increase noise and reduce image quality. Furthermore, using shorter exposure time combined with higher gain settings can effectively minimize effects such as motion blur caused by rapid movement.

Figure 3. Effect of modifying camera gain. The larger the gain, the brighter the images.

2.2  Over & Under Exposure

When signals are converted into digital frames, they undergo quantization based on the effective dynamic range of the sensor and the available bit depth. Values falling below the minimum threshold or exceeding the maximum threshold are truncated. This can result in areas of overexposure or underexposure. Overexposed regions are typically displayed as predominantly white, while underexposed areas appear almost black. This issue is prominent in stereo cameras. Figure 6 illustrates the effect of overexposure and underexposure by presenting two bad cases:

Figure 4. Two bad cases of under exposure (top)  and over exposure (bottom): left – infrared image, right – depth image.

As described in the commonly used Linear-Signal-Model, light signal undergoes quantization and gets converted into digital frames, while underexposure or overexposure may happen during this process. For a k-bit imaging sensor, normally gray values are expected to range from 0 to 2^k-1, despite the following unexpected pixels:

• almost black ones, when target surfaces are black, or too far away etc.
• nearly staturated ones with high-refelctive surfaces etc.

In both cases, rich features which reliable stereoscopic matching relys on, are missingso that depth result may be not available or with erroneous value  over such areas.

In order to solve the problem of underexposure or overexposure, common methods include: adjusting the camera’s manual exposure, controlling the camera’s automatic exposure configuration, using HDR camera or using image enhancement technology to restore the image, etc. Image enhancement technology will not be discussed in this paper.

2.3  Manual Exposure

Manual exposure is an exposure mode supported by the Gemini 330 series cameras, allowing users to adjust the exposure time and gain to modify image brightness. This method does not automatically adjust based on changes in ambient lighting. It allows users to execute precise exposure control. For instance, to photograph fast-moving objects, one can set a shorter exposure time to prevent motion blur.

Figure 5. Same scene captured with short exposure (left) vs  longer exposure (right)

You can also to setup in Orbbec viewer under  IR Left/IR Right/Depth control panel.  They user need to turn auto-exposure off before manually adjustment.

Figure 6. Disabling auto-exposure and adjusting exposure and gain in Orbbec Viewer

Similarly, you can refer to the following code to set exposure and gain.

// Create a pipeline with default device
ob::Pipeline pipe;

// Get the device from the pipeline
auto device = pipe.getDevice();

// set ir exposure value to 3000
device->setIntProperty(OB_PROP_IR_EXPOSURE_INT, 3000);

// set ir gain value to 16
device->setIntProperty(OB_PROP_IR_GAIN_INT, 16);

2.4 Automatic Exposure

The goal of auto-exposure is to achieve appreciate brightness level, or so-called target brightness level in different lighting conditions and scenes, so that the videos or images captured are neither too dark nor too bright. For example, robots need to work indoors and outdoors, and the camera is usually set to automatic exposure mode.

Figure 7. Effect of auto-exposure  indoors(top) and outdoors(bottom): left – infrared image, right – depth image.

By default Gemini 330 series cameras are configured in auto-exposure mode. Under this mode, the camera will automatically adjust exposure and gain values until the average image intensity reaches certain preset value. Auto exposure can also be turned on/off in Orbbec viewer.

Auto exposure creteria can be setup as well, incase preset average intensity value is not proper for use case. If the IR images is too dark or too bright, user can manually change the Mean Intensity Set Point, which can be set under IR Left/IR Right/Depth control panel.

Figure 8. enabling auto-exposure and adjusting Mean Intensity Set Point in Orbbec Viewer

Similarly, you can refer to the following code to set auto-exposure.

// Create a pipeline with default device
ob::Pipeline pipe;

// Get the device from the pipeline
auto device = pipe.getDevice();

// Enable IR sensor auto exposue
device->setBoolProperty(OB_PROP_IR_AUTO_EXPOSURE_BOOL, true);

// set the IR sensor mean intensity set point value to 60
device->setIntProperty(OB_PROP_IR_BRIGHTNESS_INT, 60);

However, automatic exposure cannot handle certain extreme scenarios, such as half bright and half dark in the view. In this case, you may consider using HDR as described in Section 3.

2.5 Exposure Limitation

Natrually, the maximum exposure time of a camera is constrained by its frame rate, as indicated in Table 1. In automatic exposure mode, the maximum exposure time is less than the Max exposure time based on frame rate. In manual exposure mode, there is no such limitation for user input, but if it exceeds the limitation for current frame rate, the active frame rate will be reduced accordingly. For example, if the camera’s frame rate is set to 30fps and the manual exposure time is set to 30,000us, yet some indoor scenes still appear too dark, the exposure time can be extended to 60,000us. This adjustment will cause the camera’s output frame rate to drop to 15fps.

Table 1. Max exposure per framerate

3.High Dynamic Range

High Dynamic Range (HDR) imaging allows imaging systems to capture images in extremely dark and bright scenes alike. To overcome the limitations of traditional dynamic range, this technology typically involves capturing multiple images at different exposure levels and combining them into a single HDR image. Given the limitations of the Gimini 330 hardware（MX6800）,  it is impossible/difficult to synthesize multi-exposure color and depth images on the ASIC inside the camera. We propose a software solution running on the host CPU to implement this feature. It utilizes data from two consecutive frames and directly synthesizes these two depth images, thereby enhances the dynamic range of 16-bit depth images.

This feature requires firmware v1.2.01+  and SDK v1.9.0+.

3.1 Methodology

As described before, HDR works with given exposure and gain values which was left to the user to set according to specific application scenario. So the main task here lies in merging outputs from different exposure and gain settings.

For a given set of depth image with n frames, let d_i (i=0, \ldots, n-1)be the i-th depth value at any pixel, and the merged depth value could be represented via the following

where \mathscr{W}is the weighting function. In Orbbec’s HDR implementation,  only two consecutive frames is chosen thus nequals to 2. And\mathscr{W}is a simple non-zero check function. So the former can be rewritten as

To be noted, the aforementioned merging process is implemented in Orbbec-SDK which can be simply customized.

3.2  Orbbec viewer Setting

To enhance the camera functionality with High Dynamic Range (HDR) capabilities, we have implemented new HDR control features on compatible devices. In the Orbbec Viewer software, users must activate the depth or infrared streams to ensure optimal performance of the HDR function. To accommodate the HDR processing, which involves merging consecutive frames, we have adjusted the device configuration to double the standard frame rate to 60 FPS while maintaining an effective output of 30 FPS due to the HDR synthesis process. Each HDR output is created by merging frames with alternating “high exposure + low exposure” and “low exposure + high exposure” settings. This process introduces a minimal latency of two frames, but frame rate is still 60FPS.

Users are advised to manually adjust the exposure and gain settings for both high and low exposure frames within the depth control settings to optimize HDR performance. Additionally, for precise monitoring of the camera’s functionality, the HDR-Merge feature can be disabled. This allows for an accurate assessment of the camera’s operational characteristics under varied environmental conditions.

Figure 9. Configuring the hardware for HDR in Orbbec Viewer

The camera will initiate streaming, alternating between the two predetermined exposure settings. By opening the metadata panel and pausing the stream, users can access additional details. It is important to note that per-frame metadata provides critical information and is essential for the HDR images.

Figure 10. Examining frame metadata of an HDR stream

The relevant metadata items are:

• Hdr Sequence Name – name of the HDR sequence
• Hdr Sequence Size – length of the HDR sequence
• Hdr Sequence Index– step number inside the HDR sequence of the current frame (zero based)
• Exposure – exposure that was used for current frame
• Gain – gain that was used for current frame

In order to generate synthesized depth HDR stream, HDR Merge post-processing switch needs to be enabled:

Figure 11. Enabling Depth HDR Merge under post-processing

After activating the HDR Merge feature, the standard depth stream is replaced by an artificially generated HDR depth stream. This modification eliminates flickering in the depth stream; however, flickering in the infrared image stream persists. Since the merging of infrared image frames is a common image processing operation, this functionality has not been incorporated into the OrbbecSDK.

In addition to HDR depth fusion, the SDK provides a simple way of splitting the flickering stream into two individual sequence componnents by identifying its ID. This can be  done using Sequence ID Filter switch.

Figure 12. Filtering the stream by sequence ID

3.3 Code examples

To activate HDR functionality, set different exposures or gains, and use built-in HDR Merge post-processing algorithm, refer to the following code example:

#include "window.hpp"

#include "libobsensor/hpp/Pipeline.hpp"
#include "libobsensor/hpp/Error.hpp"

int main(int argc, char **argv) try {
// Create a pipeline with default device
ob::Pipeline pipe;

// Get the device from the pipeline
auto device = pipe.getDevice();

// Check if the device supports HDR merge
if(!device->isPropertySupported(OB_STRUCT_DEPTH_HDR_CONFIG, OB_PERMISSION_READ_WRITE)) {
std::cerr << "Current default device does not support HDR merge" << std::endl;
return -1;
}

// Configure which streams to enable or disable for the Pipeline by creating a Config
std::shared_ptr<ob::Config> config = std::make_shared<ob::Config>();

// Get all stream profiles of the depth camera, including stream resolution, frame rate, and frame format
auto depthProfiles = pipe.getStreamProfileList(OB_SENSOR_DEPTH);
auto depthProfile  = depthProfiles->getProfile(OB_PROFILE_DEFAULT);
config->enableStream(depthProfile);

// Create HdrMerge post processor to merge depth frames betweens different hdr sequence ids.
// The HdrMerge also supports processing of infrared frames.
ob::HdrMerge hdrMerge;

// configure and enable Hdr stream
OBHdrConfig obHdrConfig;
obHdrConfig.enable     = true;  // enable HDR merge
obHdrConfig.exposure_1 = 7500;
obHdrConfig.gain_1     = 24;
obHdrConfig.exposure_2 = 100;
obHdrConfig.gain_2     = 16;
device->setStructuredData(OB_STRUCT_DEPTH_HDR_CONFIG, &obHdrConfig, sizeof(OBHdrConfig));

// Start the pipeline with config
pipe.start(config);

// Create a window for rendering and set the resolution of the window
bool   resizeWindows = true;
Window app("HDR-Merge", 1280, 720);
bool   mergeRequired = true;

std::cout << "Press 'M' to toggle HDR merge." << std::endl;
while(app) {
auto key = app.waitKey(10);
if(key == 'M' || key == 'm') {
mergeRequired = !mergeRequired;
if(mergeRequired) {
std::cout << "HDR merge enabled." << std::endl;
}
else {
std::cout << "HDR merge disabled." << std::endl;
}
}

auto frameSet = pipe.waitForFrames(100);
if(frameSet == nullptr) {
continue;
}

auto depthFrame = frameSet->depthFrame();
if(depthFrame == nullptr) {
continue;
}

if(mergeRequired) {
// Using HdrMerge post processor to merge depth frames
auto mergedDepthFrame = hdrMerge.process(depthFrame);
if(mergedDepthFrame == nullptr) {
continue;
}

// add merged depth frame to render queue
app.addToRender(mergedDepthFrame);
}
else {
// add original depth frame to render queue
app.addToRender(depthFrame);
}
}

// Stop the Pipeline, no frame data will be generated
pipe.stop();

// close hdr merge
obHdrConfig.enable = false;
device->setStructuredData(OB_STRUCT_DEPTH_HDR_CONFIG, &obHdrConfig, sizeof(OBHdrConfig));

return 0;
}
catch(ob::Error &e) {
std::cerr << "function:" << e.getName() << "\nargs:" << e.getArgs() << "\nmessage:" << e.getMessage() << "\ntype:" << e.getExceptionType() << std::endl;
exit(EXIT_FAILURE);
}

3.4 Results

The built-in HDR function of the Gemini 330 series cameras improves the depth fill rate.

In the following depth capturing senario, which covers a high-reflection panel and a low-reflection bracket, the Gemini 330 series camera is set to automatic exposure.  It is very obvious that in the high-reflection surface area, most of the pixels cannot get valid depth date.

On the other hand, when we turn on the HDR and HDR Merge functions, the depth fill rate in the high-reflection panel area is higher, and the depth value is more accurate.

Figure 14.  HDR stream in mixed high& low reflection scene: high exposure frame (top), low exposure frame (middle), merged depth (bottom)

One main concern about HDR is its impact on depth accuracy. One example showing the influence is as follows. Pieces of highly reflective panel were sticked to a flat wall about 1043mm away from a Gemini 335 camera as shown in the upper-left RGB image. The panels are too reflective to form reliable speckles thus no valid depth can be estimated in some regions, especially in “High Accuracy” or “Hand” presets where only reliable estimations remain. The lower left depth image depicts an depth image captured in “Hand” preset and with Auto Exposure checked.

Figure 15. Pieces of highly reflective panel were sticked tightly to a flat wall to test the accuracy.

As a comparison, a depth frame with HDR is also captured in the same scenario showing higher depth fill-rate (as show in the lower right of Figure 13) but no decreased accuracy as shown in Figure 14. Black and red lines are depth values at the same row with only different HDR setting. The red line with HDR on appeas to be more accurate.

Figure 16. Section profile of depth image captured in the same scenario as in Figure 13.

4.Summary

Gemini 330 series cameras provide high-quality depth imaging. Its built-in automatic exposure mode can meet user needs in most scenarios, both indoorand outdoor. For autonomous mobile robots equipped with these cameras, the automatic exposure generally provides sufficient adaptability to handle varying lighting conditions effectively. Since the auto-exposure algorithm is implemented in the firmware, it does not consume resources from the host equipments and simplifys user operation.

Manual exposure is typically used in scenarios requiring precise exposure control, such as factory robotic arms pan-picking, where it is only necessary to enhance the imaging quality of the workpieces.

HDR in depth sensor is designed to address some challenges that neither automatic nor manual exposure can solve. It significently improves the fill rate in tough working environment with complicated illumination or reflection situation, and also improves the depth accuracy at these area. It reduces the risk of two common and serious errors incomputer vision systems: “missed detection” and “false detection”. In some use cases, these errors can even be fatal.

As every technology people developed, it of course has its own limitations, such as frame delay and CPU usage due to extra calculation on host; experience required for determine proper high and low exposure times;  motion artifacts for high speed moving objects.

In scenes with rapid camera movement, adjusting the mean intensity of the auto-exposure can improve image stability. If needed, enabling HDR and HDR merge can enhance detail capture at the cost of a higher frame rate. For applications requiring precise depth data, activate HDR but avoid merging depth frames, using both high and low exposure frames to meet specific algorithmic needs. This setup ensures detailed and accurate depth information under varying lighting conditions.

5.References

• HDR imaging with OpenCV – https://docs.opencv.org/3.4/d2/df0/tutorial_py_hdr.html
• High dynamic range – https://en.wikipedia.org/wiki/High_dynamic_range