Depth Cameras: Stereo, Time-of-Flight, and Structured Light Compared

A depth camera returns a per-pixel range image, but each of the three dominant technologies measures that range differently. Stereo triangulates from parallax, time-of-flight (ToF) measures the round-trip phase or delay of emitted light, and structured light triangulates against a projected pattern. Those physics set the error curve, the ambient-light tolerance, and the failure modes you will see on the robot. This guide gives the math, the trade-offs, the wiring and ROS2 configuration, and a validation workflow you can run on your own hardware.

Quick Takeaways

• Stereo (especially active stereo with an IR projector) is the best general-purpose choice for mobile robots. It works outdoors, but depth error grows with the square of distance.
• Indirect ToF gives dense, low-latency depth with error nearly independent of range. It suffers from multipath and sunlight saturation.
• Structured light gives the best close-range accuracy (sub-millimeter on industrial units). It fails in direct sunlight and degrades past about 1–3 m on most compact units.
• Pick by working distance and lighting first. Resolution and price come second.

Depth Sensing Fundamentals

All three technologies output a depth map Z(u, v). You back-project each pixel into 3D with the pinhole model:

X = (u - cx) * Z / fx
Y = (v - cy) * Z / fy
Z = Z

Here fx, fy are focal lengths in pixels and (cx, cy) is the principal point from CameraInfo. The error in Z is what separates the technologies.

Stereo Vision

Two rectified cameras with baseline B see the same point at horizontal pixel offsets that differ by the disparity d:

d = x_L - x_R = f * B / Z
Z = f * B / d

Differentiating gives the error model:

dZ = (Z^2 / (f * B)) * dd

Depth error scales with Z² and shrinks with a longer baseline B and higher f. Example with f = 640 px, B = 50 mm, and a sub-pixel matching error dd = 0.08 px:

Passive stereo needs surface texture. A white wall produces no valid matches. Active stereo adds an IR projector that paints texture onto the scene. That is how the Intel RealSense D400 series works. The projector only helps matching. It does not encode depth, so unlike structured light it needs no known pattern geometry.

Time-of-Flight

Indirect ToF (iToF) modulates an NIR emitter (typically 850 or 940 nm) at frequency f_mod. The sensor samples the returned signal at four phase offsets (A0, A90, A180, A270):

phi = atan2(A270 - A90, A0 - A180)
d   = c * phi / (4 * pi * f_mod)
d_unambiguous = c / (2 * f_mod)

At f_mod = 200 MHz the unambiguous range is only 0.75 m. Real sensors run two or more frequencies and unwrap the phase to extend range. Depth noise scales roughly as 1 / (f_mod × SNR), so higher modulation frequency gives finer precision but a shorter ambiguity interval.

Direct ToF (dToF) timestamps individual photon returns with SPAD arrays:

d = c * t / 2

A timing resolution of 100 ps corresponds to 15 mm. dToF handles long range better but yields lower resolution on compact parts.

Structured Light

A projector casts a known pattern (speckle, stripes, or Gray codes). The camera observes the pattern displaced by depth. The geometry is stereo with the second camera replaced by the projector. Measured against a reference plane at distance Z0:

1/Z = 1/Z0 + d / (f * B)

Industrial units use temporal pattern sequences (multiple projected frames) for sub-0.1 mm accuracy on static scenes. Consumer units use a single fixed speckle pattern for real-time capture at lower accuracy.

Head-to-Head Comparison

Failure Modes and Mitigation

Practical mitigations:

• Use global-shutter stereo sensors on fast platforms.
• Add 940 nm emitters plus narrow band-pass filters for outdoor ToF.
• Run a temporal median filter (3–5 frames) and a speckle filter on all three types.
• Mask depth where confidence or amplitude is low. On iToF, amplitude is the confidence.

Hardware Integration

Interfaces and Power

Wiring checklist:

• Budget 5 V at 1.5 A per USB depth camera. IR projector bursts cause brown-outs on weak USB ports.
• Mount the camera on a rigid bracket. A 0.5° flex at 4 m is 35 mm of lateral error.
• Hardware-sync multiple cameras on the same GPIO trigger to avoid motion skew between views.
• For multi-ToF rigs, assign different modulation frequencies or time-multiplex the emitters.

ROS2 Configuration

Launching a RealSense (Active Stereo)

# 848x480 @ 30 FPS is the sweet spot for D435-class depth accuracy.
# align_depth registers depth to the color frame; pointcloud.enable publishes sensor_msgs/PointCloud2.
ros2 launch realsense2_camera rs_launch.py \
  depth_module.depth_profile:=848x480x30 \
  align_depth.enable:=true \
  pointcloud.enable:=true

Orbbec ToF cameras use a similar launch pattern:

ros2 launch orbbec_camera femto_bolt.launch.py

Depth Guard Node (Nearest-Obstacle Monitor)

This node subscribes to the depth image and CameraInfo, back-projects a central ROI, and publishes the nearest obstacle distance for an emergency-stop layer.

#!/usr/bin/env python3
import numpy as np
import rclpy
from rclpy.node import Node
from rclpy.qos import qos_profile_sensor_data
from sensor_msgs.msg import Image, CameraInfo
from std_msgs.msg import Float32
from cv_bridge import CvBridge


class DepthGuard(Node):
    def __init__(self):
        super().__init__('depth_guard')
        # Valid range gate in meters; rejects near-field saturation and far-field noise.
        self.declare_parameter('min_range_m', 0.3)
        self.declare_parameter('max_range_m', 6.0)
        # Fraction of image width/height used as the central region of interest.
        self.declare_parameter('roi_fraction', 0.4)

        self.bridge = CvBridge()
        self.K = None  # 3x3 intrinsics from CameraInfo.k (row-major: fx,0,cx,0,fy,cy,0,0,1)

        # Sensor-data QoS (best-effort) matches camera driver publishers.
        self.create_subscription(
            CameraInfo, '/camera/camera/depth/camera_info',
            self.info_cb, qos_profile_sensor_data)
        self.create_subscription(
            Image, '/camera/camera/depth/image_rect_raw',
            self.depth_cb, qos_profile_sensor_data)
        self.pub = self.create_publisher(Float32, 'nearest_obstacle_m', 10)

    def info_cb(self, msg: CameraInfo):
        self.K = np.array(msg.k, dtype=np.float64).reshape(3, 3)

    def depth_cb(self, msg: Image):
        if self.K is None:
            return
        raw = self.bridge.imgmsg_to_cv2(msg, desired_encoding='passthrough')
        # REP 118: 16UC1 depth is in millimeters, 32FC1 is in meters.
        z = raw.astype(np.float32) * (0.001 if msg.encoding == '16UC1' else 1.0)

        h, w = z.shape
        frac = self.get_parameter('roi_fraction').value
        r0, r1 = int(h * (0.5 - frac / 2)), int(h * (0.5 + frac / 2))
        c0, c1 = int(w * (0.5 - frac / 2)), int(w * (0.5 + frac / 2))
        roi = z[r0:r1, c0:c1]

        lo = self.get_parameter('min_range_m').value
        hi = self.get_parameter('max_range_m').value
        valid = np.isfinite(roi) & (roi > lo) & (roi < hi)
        if valid.sum() < 50:      # too few valid pixels: do not publish a misleading value
            return

        # 2nd percentile instead of min() rejects isolated flying pixels.
        nearest = float(np.percentile(roi[valid], 2))

        # Back-project the median valid pixel to get forward/lateral geometry (X = (u-cx)Z/fx).
        vs, us = np.nonzero(valid)
        u = us.mean() + c0
        fx, cx = self.K[0, 0], self.K[0, 2]
        lateral = (u - cx) * nearest / fx
        self.get_logger().debug(f'nearest={nearest:.2f} m, lateral={lateral:.2f} m')

        self.pub.publish(Float32(data=nearest))


def main():
    rclpy.init()
    rclpy.spin(DepthGuard())


if __name__ == '__main__':
    main()

Real-World Workflow: Validating a Depth Camera Before Deployment

Datasheet numbers assume ideal targets. Measure noise on your own unit and surfaces.

Step 1: Plane-Fit Noise Test

Point the camera at a flat matte wall at 1 m, 2 m, and 3 m. Record 100 frames per distance. Fit a plane and compute the residual standard deviation.

import numpy as np

def plane_fit_rms(points_xyz: np.ndarray) -> float:
    """points_xyz: (N, 3) array in meters. Returns RMS distance to best-fit plane."""
    centroid = points_xyz.mean(axis=0)
    # SVD of centered points; the smallest singular vector is the plane normal.
    _, _, vt = np.linalg.svd(points_xyz - centroid, full_matrices=False)
    normal = vt[-1]
    residuals = (points_xyz - centroid) @ normal   # signed distance to plane
    return float(np.sqrt(np.mean(residuals ** 2)))

Compare the result to your error model. For stereo, the RMS should track Z²/(f·B) · dd. For ToF, it should stay roughly flat with range.

Step 2: Accuracy (Bias) Test

Place a target at a laser-measured distance (for example, 2.000 m). The mean depth minus ground truth is the bias. iToF units often show a few millimeters to centimeters of bias that depends on target reflectivity. Calibrate or compensate.

Step 3: Stress the Failure Modes

Run the same test against black felt, a glass door, brushed aluminum, and a sunlit surface. Log the valid-pixel ratio for each. A sensor that returns 90% valid pixels on a wall and 30% on glass tells you where the navigation stack needs a second sensor.

Step 4: Pick the Sensor by Task

Frequently Asked Questions

Which depth camera works best outdoors?

Stereo is the best choice. Passive matching uses sunlight as illumination, so the algorithm does not depend on projected IR. ToF and structured light lose signal because sunlight carries significant NIR energy that saturates the sensor or washes out the projected pattern. Narrow band-pass filters and higher emitter power help ToF, but stereo remains the most reliable option in direct sun.

Is ToF more accurate than stereo?

It depends on range. At close distance (under 1–2 m), stereo with a good baseline often matches or beats compact ToF. At longer range, stereo error grows with Z² while iToF error stays nearly constant as a percentage of distance, so ToF pulls ahead. ToF accuracy also degrades from multipath in corners and on glossy surfaces.

Can I use multiple depth cameras on one robot?

Yes, with precautions. Active stereo cameras interfere little because the projector only adds texture. ToF and structured light cameras interfere strongly when fields of view overlap. Use different modulation frequencies, time-multiplexed emitters, or hardware-synced triggers, and keep overlap small.

What is the best depth camera for ROS2 SLAM?

Active stereo cameras such as the RealSense D435 family are the common default. They have mature ROS2 drivers, work across lighting conditions, and supply a depth stream aligned to color for RGB-D SLAM pipelines. Choose iToF only for short-range indoor mapping where dense, low-latency depth outweighs sunlight sensitivity.