HDVO: Hybrid Dense Direct Visual Odometry

Structure of hybrid dense direct visual odometry — Structure of HDVO. A data-based stereo depth network feeds a model-based DDVO module that refines the relative pose over N iterations. The dense photometric loss is filtered by an occlusion mask and a texture mask before back-propagating into the depth network. At inference the parts marked with a snowflake are frozen and only the relative pose is optimised with the second-order ESM optimiser.

Abstract

Dense direct visual odometry (DDVO) is a well-known robust model-based visual odometry method that computes robot self-localization from visual data. However, obtaining a metric localization requires depth information, which is unavailable in most applications and especially outdoors. For this reason we focus on a hybrid dense direct visual odometry (HDVO) that combines a model-based localization module and a data-based stereo depth estimation module. Since depth is estimated from a stereo sensor, the proposed HDVO achieves better localization performance than methods using only monocular RGB data. Moreover, the learning part of HDVO is optimized with the photometric loss of DDVO without any depth supervision signal.

For this loss, occlusion and low texture affect the optimization convergence. This paper investigates a binary mask based on stereo-temporal consistency to address the occlusion problem, along with another binary mask based on local patch consistency to solve the low-texture problem. Finally, it is shown that HDVO achieves a significant advantage in terms of both inference speed and odometry error on public benchmarks.

Contributions

Unified model-based and data-based odometry. A deep stereo depth network and a classical dense direct visual odometry module are trained end-to-end through a single photometric loss, without any depth ground truth.
Metric scale from stereo. Stereo depth removes the scale ambiguity of monocular depth priors, which is what makes the estimated trajectory directly usable across scenarios and sensors without ground-truth rescaling.
Efficient test-time optimization. Only the pose is optimized at inference, so the method keeps real-time performance while retaining the generalization benefit of on-line refinement.
Robust loss plus binary masks. A Huber loss replaces the unstable learned per-pixel weights, while the STC occlusion mask and the LAM texture mask filter the extreme outliers that re-weighting alone cannot handle.

Why Mask the Photometric Loss

The dense photometric loss is the matching error between the reference and current frames. Three kinds of pixels break this matching: occluded pixels that have no correspondence at all, pixels affected by brightness discrepancy between camera views, and pixels in homogeneous regions that cannot be distinguished from their neighbours. All three act as outliers during optimization.

Illustration of occlusion, brightness discrepancy and homogeneous texture in the loss matrix — Each coloured cell is one pixel of the loss matrix. Occlusion, brightness discrepancy and homogeneous texture all produce loss values that cannot be driven to zero.

Computation steps of the stereo-temporal consistency occlusion mask — STC occlusion mask: a pixel is kept only if it is recovered consistently by both the stereo warping and the temporal warping of the same reference view.

Local area matching texture mask — LAM texture mask: local patch consistency identifies ambiguous low-texture pixels such as sky, highlights and shadows.

Input RGB frame from Virtual KITTI2 — Input frame

LAM homogeneous texture mask — LAM texture mask

Results on KITTI Odometry

HDVO reaches the lowest translation error among model-based, end-to-end and hybrid methods on KITTI sequences 09 and 10, while producing trajectories at true metric scale. Most competing hybrid and learning-based methods report results rescaled with ground-truth odometry, which makes them unusable as-is for real vehicle localization.

KITTI odometry translation and rotation error on sequences 09 and 10
Method	seq. 09 t_err (%)	seq. 09 r_err (°/100m)	seq. 10 t_err (%)	seq. 10 r_err (°/100m)
ORB-SLAM2 (stereo)	0.85	0.26	0.56	0.24
DVSO	0.83	0.21	0.74	0.21
D3VO	0.78	–	0.62	–
DF-VO	2.07	0.23	2.06	0.36
DINO-VO	1.64	0.93	0.96	0.71
HDVO (IROS 2022)	0.87	0.28	0.87	0.46
HDVO (ours)	0.73	0.36	0.42	0.24

Trajectory comparison on KITTI sequence 09 — KITTI sequence 09

Trajectory comparison on KITTI sequence 10 — KITTI sequence 10

Absolute trajectory error versus inference speed on KITTI sequence 09 — ATE vs. speed

Translation error versus inference speed on KITTI sequence 09 — Translation error vs. speed

Rotation error versus inference speed on KITTI sequence 09 — Rotation error vs. speed

Effect of the Masks

Both masks improve depth and pose accuracy, and their combination is consistently the best. Removing occlusions matters more than removing low-texture regions, but the two are complementary. On Virtual KITTI2 scene 20 the multi-mask reduces the translation error from 13.90% to 0.95%.

Ablation of the STC and LAM masks
Sequence	Mask	abs rel ↓	rmse ↓	t_err (%) ↓	r_err (°/100m) ↓
KITTI 09	none	0.2827	8.414	2.77	0.68
	LAM	0.2535	7.538	2.41	0.71
	STC	0.1042	3.934	2.68	0.81
	multi-mask	0.0665	3.305	1.54	0.45
Virtual KITTI2 20	none	0.7648	62.14	13.90	3.75
	LAM	0.3273	63.24	5.29	0.87
	STC	0.3298	64.75	1.68	0.76
	multi-mask	0.1284	59.45	0.95	0.46

Estimated trajectory without any mask — No mask

Estimated trajectory with the LAM texture mask — LAM mask

Estimated trajectory with the STC occlusion mask — STC mask

Estimated trajectory with the combined multi-mask — Multi-mask

Inference Cost

Measured with the plain PyTorch implementation using the CoEx stereo backbone. HDVO stays real-time on embedded platforms, with room for further acceleration through ONNX or TensorRT.

Inference time of HDVO on different platforms
Platform	Time (ms) ↓	FPS ↑
NVIDIA A40	21.06	47.5
NVIDIA L40	26.67	37.5
Jetson Orin	80.89	12.4
Jetson Thor	55.97	17.9

Generalization

Beyond ground vehicles, HDVO is evaluated on two drone datasets: Mid-Air, where depth reaches beyond 1000 m, and EuRoC MAV, which combines fast motion with a short stereo baseline. The dynamic and weather experiments are zero-shot: the model is trained on KITTI Odometry and evaluated on Virtual KITTI2 without fine-tuning.

Odometry results on the Mid-Air drone dataset — Mid-Air

Odometry results on the EuRoC MAV drone dataset — EuRoC MAV

Zero-shot trajectory on Virtual KITTI2 scene 20 with many moving vehicles — Scene 20 — 77 dynamic objects

Zero-shot trajectory on Virtual KITTI2 scene 02 — Scene 02 — low dynamic activity

Relative pose error on Virtual KITTI2 scene 20 under different weather conditions
Weather	RPE trans. (m) ↓	RPE rot. (°) ↓
Base	0.170	0.051
Fog	0.204	0.066
Morning	0.177	0.046
Overcast	0.183	0.044
Rain	0.191	0.048
Sunset	0.173	0.041

Trajectory under fog on Virtual KITTI2 scene 20 — Fog

Trajectory under rain on Virtual KITTI2 scene 20 — Rain

Trajectory at sunset on Virtual KITTI2 scene 20 — Sunset

Earlier Conference Versions

This journal article extends two conference papers: the IROS 2022 paper that introduced stereo depth to fix the scale problem of the depth prior, and the ITSC 2023 paper that proposed the rule-based mask generation module.

BibTeX

@article{HDVO-TIV-2026,
  title     = {HDVO: Hybrid Dense Direct Visual Odometry},
  author    = {Liu, Ziming and Malis, Ezio and Martinet, Philippe},
  journal   = {IEEE Transactions on Intelligent Vehicles},
  volume    = {11},
  pages     = {930--945},
  year      = {2026},
  publisher = {IEEE},
  doi       = {10.1109/TIV.2026.3708447}
}

Acknowledgements

This project was funded by 3IA Côte d'Azur, Université Côte d'Azur (Nice, France), and was carried out at Inria.