EM-ViT: Adapting Monocular Depth Estimation in Novel Scenes through Disparity Refinement and Robot Ego-Motion
arXiv Code
We present EM-ViT, a lightweight Monocular Depth Estimation (MDE) model for robotics, which learns to adapt relative depth geometric priors to metric scale using a virtual stereo framework.
☆ Our model is self-supervised via photometric reconstruction loss, sparse LiDAR measurements, and employs gradient matching loss to preserve the relative fine-grained details of the affine-invariant disparity input.
Abstract
The geometric understanding of a scene is a crucial component for safe and efficient robotic operations in novel and unstructured environments. The recent emergence of foundation models for Monocular Depth Estimation (MDE) has provided an opportunity to enrich robotic perception capabilities with dense long-range depth predictions, replacing expensive onboard hardware (e.g., LiDARs and stereo cameras). However, these offline models are primarily trained on affine-invariant depth representations, and their generalization to novel scenes remains unclear. In this work we present EM-ViT, a monocular depth estimation model that learns to adapt online relative depth inputs from a foundation model prior to the metric scale with observations collected during robotic operations. Motivated by the stereo camera parameterization, our method decomposes depth prediction via dense disparity refinement to recover the scene's relative structure and an effective baseline scalar prediction to determine the missing metric scale. The learning process leverages the robot's Ego-Motion (EM) and, optionally, sparse LiDAR points to rapidly train a lightweight Vision Transformer (ViT), while an internal gradient matching loss preserves the fine-grained semantic structure of the input disparity in real-time. We demonstrate the efficacy of our method on publicly available sequences, such as KITTI, and show that our online learning pipeline can adapt to novel scenes faster and with higher accuracy than existing MDE foundation models.
Online Evaluation and MDE Adaptation
We evaluate our system on extensive experiments from robotic collected data in indoor and outdoor environments, as well as, publicly available datasets, like KITTI and CitrusFarm.
Figures above showcase the MDE results from various scenes during online training, while below are moments during our MDE adaptation in a garage environment.
EM-ViT demonstrates faster and more accurate MDE metric adaptability using our scale-to-structure disentaglement method among offline and online fine-tuned baselines. When comparing with online fine-tuned DAV2 approach, it reduces L1\( _{10m} \) error by 32% and requires 16% less training time.
We also present our method's offline training results in the garage environment, compared with the unadapted and adapted DAV2 ViT-S models.
In contrast to a direct metric head, where the scale would be encoded redundantly at every pixel, in our virtual setup dense disparity \(\hat{d}\) contains all shape information alone, since the effective baseline \(\hat{B}\) is decoupled by any depth ratio,
All online evaluations have been conducted on a NVIDIA AGX Orin 32GB computer,
while we used an NVIDIA RTX 3060Ti GPU for the offline evaluation.
Evaluation Studies
Online Disparity Refinement
In many cases, robust MDE foundation models are lacking in correct disparity or depth map estimation, especially when operating in novel scenes.
✓ EM-ViT is able to provide real-time disparity corrections, through the proposed dense and relative disparity reconstruction method, improving out-of-domain monocular depth estimation.
We replace our disparity refinement module with a \( L_2 \)-solver, for metric fine-tuning of the affine-invariant input disparity.
✓ Even trained on three image sequences, EM-ViT achieves disparity corrections and reduces the L1\(_{80m}\) error by 40cm compared with the used solver for the affine-invariant input disparity.
Robotic Data Availability and Depth Map Fidelity
We evaluate EM-ViT under different number of received images per sequence (above) and available ground truth LiDAR points (left).
✓ Our model is able to accurate refine and obtain depth maps utilizing the geometric priors, even with limited sampling rates and ground truth, reaching down to the use of a single beam (i.e. rangefinder).
Synthetic datasets with dense depth annotations have enabled high in fidelity metric MDE results. Unfortunatelly, this is unlikely in real-world robotic applications, where sparse LiDAR annotations are mainly available.
✓ EM-ViT showcases higher depth map fidelity due to the proposed disparity reconstruction method and the gradient matching \( \mathcal{L}_{GM} \) supervision.
Disparity Map and Effective Baseline Decomposition
We employ true disparities and baseline values from a KITTI scene and compare the per-pixel baseline values distribution among the two comparing disparities \(d_a\) and \(\hat{d}\). to examine the generalized disparity generation of the foundation model.
✓ Our method achieves concentrated variance on unseen images close to the true baseline, by having 1cm of mean baseline mean absolute error in contrast to the 13cm of the input disparity (top two figures).
✓ Also, our model predicts baseline values \(\hat{B}\) that are closer in value to the true baseline (figure bottom-left).
✓ By obtaining the MAD normalized disparities (figure bottom-right), our system reduces by 16% the overall standard deviation of the per-pixel baseline values.