RGB-depth cameras play a critical role in applications such as 3D reconstruction, scanning, and Simultaneous Localization and Mapping (SLAM), which must undergo rigorous testing to ensure optimal performance. While qualitative testing can confirm basic functionality, it is the quantitative testing that truly matters for applications requiring precise depth readings and measurements.
In this blog, we encapsulate the key insights Xin shared at the Vision Spectra Conference 2024 on the methods and metrics used to evaluate depth cameras, with a focus on both qualitative and quantitative testing approaches, offering an in-depth exploration of the performance metrics associated with various depth-sensing technologies.
A Deeper Dive into Quantitative Testing
Quantitative testing is essential for capturing the numerical data required for high-accuracy applications. Xin highlighted the three primary depth-sensing technologies available today:
- Structured Light Depth Cameras
- Stereo Vision Depth Cameras
- Time of Flight (TOF) Depth Cameras
Each of these technologies operates on unique principles to measure distance. For instance, structured light cameras calculate depth by comparing a captured image with a pre-recorded calibration image, while TOF cameras measure depth based on the time it takes for a laser to bounce back from an object.
Accuracy and Error Analysis
To accurately assess these cameras, we have developed a comprehensive testing workflow. This involves:
- Setting up a testing environment with a white flat wall as the target.
- Using an optical rail to ensure the camera and laser distance meter move rigidly together.
- Capturing data from varying distances (0.5 meters to 6 meters) at intervals of 100 millimeters.
- Turning off all post-processing filters to ensure raw data comparison.
We selected multiple regions of interest (ROI) within the captured depth images to evaluate the presence of lens distortion and other anomalies. This multi-ROI approach helps us understand if the camera’s performance varies across different areas of the frame.
Data Presentation and Analysis
The analysis of the captured data involves several key metrics:
- Accuracy: Calculated by comparing depth readings to ground truth values obtained from a laser distance meter.
- Root Mean Square Error (RMSE): A widely used method to represent error.
- Standard Deviation (STD): Indicates the stability and concentration of data.
We visualize this data using charts and histograms, which provide a clear picture of each camera’s performance. For instance, our findings show that TOF cameras generally offer the best accuracy, while structured light cameras perform well at close ranges.
Exploring Spatial and Temporal Precision Metrics
Spatial precision refers to the consistency of depth readings across different parts of the image frame. We calculate the mean STD within selected ROIs and plot these values against distance. Interestingly, while TOF cameras lead in accuracy, structured light cameras show superior spatial precision at close ranges, followed by stereo vision cameras.
Temporal precision evaluates the stability of depth readings over time. In our tests, we captured depth images every second for ten minutes at a fixed distance. A stable camera should show minimal variation in these readings over time, indicating reliability for long-term applications.
Practical Insights and Comparisons
Our testing framework not only highlights the strengths and limitations of each depth-sensing technology but also provides practical insights for real-world applications. For example, TOF cameras excel in accuracy over longer distances, making them ideal for industrial applications, while structured light cameras are more cost-effective for close-range uses, such as Apple’s Face ID.
Xin also shared specific examples of cameras with performance issues, such as lens distortion leading to significant depth inaccuracies at the edges of the frame. He used heatmaps to demonstrate how TOF cameras showed increased error rates at distances beyond 4 meters, and histograms to illustrate the temporal instability of certain stereo vision cameras over prolonged use. These analyses are crucial for identifying and addressing potential problems in camera design and manufacturing, helping us refine our technologies and improve overall performance.
In conclusion, we are committed to advancing precise and reliable depth-sensing technology. Our rigorous testing methodologies ensure that our cameras meet the highest standards of accuracy and performance, paving the way for innovative applications across various industries.
Watch Xin’s full webinar here.
