FutureNav: Unified World-Action Modeling
for Vision-and-Language Navigation

Lingfeng Zhang1,2,4,*       Zeying Gong3,4,*       Xiaoshuai Hao4,†,✉       Haoxiang Fu5       Qiang Zhang3      
Mingliang Zhou4       Hangjun Ye4       Xiaojun Liang2       Junwei Liang3       Wenbo Ding1,✉      

1 Tsinghua University     2 Pengcheng Laboratory    
3 HKUST(GZ)     4 Xiaomi EV     5 NUS     
* Equal Contribution      † Project Leader      ✉ Corresponding Author

Paper   Video   Code   Model

  Real-World Demonstrations

Testing our method in real-world environments.


  Simulation Demonstrations

Testing our method in simulation environments, including R2R and RxR.

  Abstract

Vision-and-language navigation (VLN) in continuous environments requires an agent to ground instructions in egocentric observations while maintaining spatial understanding across long action sequences. Recent navigation foundation models have shown strong progress by scaling vision-language models, but they often learn navigation primarily as direct action generation, without explicitly modeling world states or predicting their future evolution. We introduce \emph{Unified Navigation World Action Model} (FutureNav), a VLM-based unified world-action modeling framework. Specifically, FutureNav jointly encodes text, visual, and spatial features and feeds them into the LLM, and optimizes four objectives for simultaneous world and action modeling: an action policy objective for navigation action prediction, inverse and forward dynamics objectives for modeling state transitions, and a future generation objective for predicting future spatial states. This unified architecture strengthens action prediction while explicitly modeling the world, without sacrificing inference speed. Extensive experiments show that, with only a 4B-scale backbone, FutureNav achieves state-of-the-art performance on multiple VLN benchmarks and substantially outperforms prior VLN methods, paving the way toward future world-action models for VLN. We will release the code and models to support future research.

  Overview

Overview of FutureNav. Our model unifies four navigation capabilities in a single VLM-based world-action framework: action policy learning for direct navigation control, inverse dynamics for inferring actions from observation changes, forward dynamics for action-conditioned state transition modeling, and future generation for predicting upcoming spatial states. This unified design improves world understanding and action prediction while achieving state-of-the-art performance on VLN-CE benchmarks.

  Framework

Architecture of FutureNav. A VLM backbone encodes the instruction and egocentric observations, while a frozen spatial encoder provides spatial-aware features for world-state understanding. The model is trained to support four world-action capabilities: action policy learning for navigation control, inverse dynamics for inferring actions from observation changes, forward dynamics for predicting action-conditioned state transitions, and future generation for anticipating upcoming spatial states.

  Quantitative Results

Main experiment of FutureNav. Comparison with prior VLN-CE methods on the R2R-CE and RxR-CE val-unseen splits. FutureNav uses only a single RGB sensor. StreamVLN* uses EnvDrop as external data. NaVILA* excludes human-following data. † denotes models trained only with VLN-CE expert trajectories and without any additional data. Δ(%) rows report relative gains over JanusVLN; NE is computed in the reverse direction since lower is better.


  Qualitative Results


Simulated qualitative comparison on VLN-CE. We compare FutureNav with the policy-only baseline under matched instructions and observations. The baseline tends to make locally plausible but less stable decisions, while FutureNav better preserves spatial progress, follows landmark-conditioned turns, and stops near the intended target.
Zero-shot real-world qualitative results of FutureNav. Without real-world navigation fine-tuning, the model follows natural language instructions in real indoor and outdoor scenes, grounds navigation decisions to visible landmarks, and stops at the specified destination.