OpenBench: A New Benchmark and Baseline for Semantic Navigation in Smart Logistics

ICRA 2025
Junhui Wang1,2, Dongjie Huo3, Zehui Xu4, Yongliang Shi2, Yimin Yan5, Yuanxin Wang6, Chao Gao2†, Yan Qiao1†, Guyue Zhou2,7†
1 Institute of Systems Engineering and Collaborative Laboratory for Intelligent Science and Systems, Macau University of Science and Technology
2 Institute for AI Industry Research (AIR), Tsinghua University
3 College of Information Science and Technology, Beijing University of Chemical Technology
4 School of Astronautics, Harbin Institute of Technology
5 School of Artificial Intelligence, University of Chinese Academy of Sciences
6 School of Mechanical and Vehicular Engineering, Beijing Institute of Technology
7 School of Vehicle and Mobility, Tsinghua University
†Indicates Corresponding Author
Paper arXiv Code

Highlights

  • New Benchmark for Last-Mile Delivery: A new benchmark is introduced to optimize last-mile delivery in residential environments.
  • Baseline Implementation: The OPEN system is presented as a baseline for last-mile delivery, using off-the-shelf OSM for lightweight map representation.
  • Combination of Foundation Models and Classic Algorithms: The OPEN system combines foundation models and classic algorithms for enhanced semantic navigation.
  • Simulated and Real-World Experiments: Extensive experiments validate the OPEN system's effectiveness in last-mile delivery.

Experimental Video

Benchmark Overview

overview

Overview of the proposed benchmark framework. The diagram presents the simulation environments and corresponding OSM, which are provided for the implementation of semantic navigation systems. This framework necessitates the navigation system to process natural language instructions autonomously, enabling accurate navigation from the initial starting point to the designated customer's front door.

Simulation Environments

sim_nav

Simulation environment for last-mile delivery. Based on the Gazebo simulation platform, we constructed three distinct world models of varying sizes, categorized into three levels: small, medium, and large, depending on the complexity of their environments. Each building within these models has been labeled with house numbers on their doors. Additionally, corresponding OSM data are generated for each world model, reflecting real-world situations.

Baseline Overview

navigation_framework

Overview of the OPEN system for autonomous last-mile delivery. The system initiates with a natural language delivery request, processed by a task planning module powered by an LLM. This module interacts with OSM to extract destination details and generates a structured task sequence. The robot autonomously decides between navigation and exploration modes, generating waypoints for execution by a classical planner.

Experiments

real_nav

Illustration of the real-world experiment. The top-left part presents the OSM and target buildings. The bottom-left part displays the delivery instructions. The right side of the figure shows the navigation trajectories of different methods.

BibTeX

If you find our work useful in your research, please consider citing:

@misc{wang2025openbench,
        title={OpenBench: A New Benchmark and Baseline for Semantic Navigation in Smart Logistics}, 
        author={Junhui Wang and Dongjie Huo and Zehui Xu and Yongliang Shi and Yimin Yan and Yuanxin Wang and Chao Gao and Yan Qiao and Guyue Zhou},
        year={2025},
        eprint={2502.09238},
        archivePrefix={arXiv},
        primaryClass={cs.RO},
        url={https://arxiv.org/abs/2502.09238}
      }