FLAME: Learning to Navigate with Multimodal LLM in Urban Environments

Yunzhe Xu, Yiyuan Pan, Zhe Liu, Hesheng Wang
Shanghai Jiaotong University
Paper arXiv Code Data


  • Our agent, powered solely by a Multimodal LLM (MLLM), demonstrates effectiveness in correlating specific environmental features with verbal navigation instruction.

  • We propose a tailored three-phase tuning technique for adapting Flamingo into navigation scenarios using synthetic data, fully unleashing MLLM's power.

  • Our approach outperforms the latest state-of-the-art (SOTA) methods by 7.3% TC on Touchdown, proving that MLLMs can significantly outperform specialized VLN models.

    • Workflow Animation

    Abstract

    Large Language Models (LLMs) have demonstrated potential in Vision-and-Language Navigation (VLN) tasks, yet current applications face challenges. While LLMs excel in general conversation scenarios, they struggle with specialized navigation tasks, yielding suboptimal performance compared to specialized VLN models. We introduce FLAME (FLAMingo-Architected Embodied Agent), a novel Multimodal LLM-based agent and architecture designed for urban VLN tasks that efficiently handles multiple observations. Our approach implements a three-phase tuning technique for effective adaptation to navigation tasks, including single perception tuning for street view description, multiple perception tuning for trajectory summarization, and end-to-end training on VLN datasets. The augmented datasets are synthesized automatically. Experimental results demonstrate FLAME's superiority over existing methods, surpassing state-of-the-art methods by a 7.3% increase in task completion rate on Touchdown dataset. This work showcases the potential of Multimodal LLMs (MLLMs) in complex navigation tasks, representing an advancement towards practical applications of MLLMs in embodied AI.

    Method Details

    FLAME Architecture: Based on Flamingo, FLAME operates autoregressively and efficiently handles multiple perceptions without increasing context length, ensuring efficiency in end-to-end training and inference.

    Three-phase Tuning for Navigation and Synthetic Data Generation: We propose a three-phase tuning technique to adapt Flamingo model to navigation tasks using augmented data: 1) Single perception tuning: Learning to describe street views. 2) Multiple perception tuning: Learning to summarize agent trajectories. 3) End-to-End training and evaluation on VLN datasets. To support the first two tuning phases, we utilize GPT-4 to synthesize captions and route summaries. We also synthesize navigation rationales for urban VLN datasets to validate FLAME's reasoning capability.

    Comparison with SOTAs

    Touchdown Map2seq
    Dev Set Test Set Dev Set Test Set
    Model TC↑ SPD↓ nDTW↑ TC↑ SPD↓ nDTW↑ TC↑ SPD↓ nDTW↑ TC↑ SPD↓ nDTW↑
    RCONCAT (2019) 10.60 20.4 22.50 11.80 20.40 22.90 17.10 - 30.70 14.70 - 27.70
    GA (2019) 12.00 18.70 25.20 11.9 19.00 24.90 18.20 - 33.00 17.00 - 30.10
    VLN-Trans (2021) 15.00 20.30 27.00 16.20 20.80 27.80 18.60 - 31.10 17.00 - 29.50
    ARC+L2S (2020) 19.48 17.05 - 16.68 18.84 - - - - - - -
    ORAR (2022) 30.05 11.12 45.50 29.60 11.79 45.30 49.88 5.87 62.70 47.75 6.53 62.10
    VELMA (2023) 29.83 14.67 43.44 27.38 15.03 41.93 52.75 6.78 66.45 48.70 6.80 62.37
    PM-VLN (2023) 33.00 23.60 - 33.40 23.80 - - - - - - -
    VLN-Video (2024) 34.50 9.60 - 31.70 11.2 - - - - - - -
    Loc4Plan (2024) 34.50 10.50 - 32.90 11.50 - 48.00 7.00 - 45.30 7.20 -
    FLAME 41.28 9.14 55.96 40.20 9.53 54.56 56.95 5.95 71.36 52.44 5.91 67.72

    Comparison with state-of-the-art models on Touchdown and Map2seq datasets. Bold values indicate best performance.

    BibTeX

    @article{xu2024flame,
        title={FLAME: Learning to Navigate with Multimodal LLM in Urban Environments},
        author={Xu, Yunzhe and Pan, Yiyuan and Liu, Zhe and Wang, Hesheng},
        journal={arXiv preprint arXiv:2408.11051},
        year={2024}}