A Survey on End-to-End Autonomous Driving Training from the Perspectives of Data, Strategy, and Platform

Welcome to this curated collection of papers on End-to-End Autonomous Driving (E2E-AD), aimed at researchers, engineers, and enthusiasts in the field of autonomous driving systems. This repository provides a comprehensive selection of papers, focusing primarily on the training methods and ecosystems that drive the development of intelligent autonomous vehicles.

In particular, we focus on the Data-Strategy-Platform framework for E2E-AD systems, offering insights into:

• Data Layer: Addressing data collection, coverage, and governance practices.
• Strategy Layer: Exploring imitation learning, reinforcement learning, and generative approaches.
• Platform Layer: Understanding scalable training infrastructures and cloud-edge collaborations.

Each paper in this repository has been selected for its relevance and contribution to the field, and we hope it serves as a valuable resource for anyone working in or learning about autonomous driving technology.

• Survey paper
• Research Papers
  • Data Layer
    • Sources and Scenario Coverage
    • Supervision and Alignment of Labels
    • Data Valuation and Governance
  • Strategy Layer
    • Imitation Learning
    • Reinforcement Learning
    • Diffusion Policy
    • Multimodal Large Language Model
    • World Model
  • Platform Layer
    • Distributed and Parallel Training Platform
    • Testing and Evaluation Platform
• Datasets & Benchmarks
• Other Awesome Lists
• Citation

🔗Refer to Link

• nuScenes : A large-scale multimodal dataset widely used for various AD tasks, including 3D object detection, tracking, and prediction. It features data from cameras, LiDAR, and radar, along with full sensor suites and map information. Several works like LightEMMA , OpenDriveVLA , and GPT-Driver utilize nuScenes for evaluation or data generation. It is also used for tasks like BEV retrieval and dense captioning.
• 4DLidarOpen : An open multi-modal autonomous driving dataset centered on 4D FMCW LiDAR, providing point-wise radial velocity, multiple LiDAR types, surround-view cameras, ego poses, 3D boxes, track IDs, and benchmarks for detection, BEV segmentation, flow prediction, motion forecasting, and planning.
• XWOD : Extreme Weather Object Detection benchmark with real-world traffic images across rain, snow, fog, haze/sand/dust, flooding, tornado, and wildfire conditions for studying weather-robust autonomous driving perception.
• ScenePilot-Bench : A first-person driving VLM benchmark built on ScenePilot-4K, evaluating scene understanding, spatial perception, motion planning, and safety-aware reasoning across large-scale driving videos.
• Bench2Drive-Robust : A closed-loop robustness benchmark for E2E-AD under deployment perturbations such as camera-stream failures, ego-state errors, and compute-induced control delay.
• MDrive : A closed-loop cooperative driving benchmark for end-to-end multi-agent systems, covering V2X perception sharing, negotiation, Real2Sim conversion, and human-in-the-loop simulation.
• HiDrive : A closed-loop benchmark emphasizing high-level and long-tail driving capabilities such as rule compliance, ethical reasoning, emergency response, and rare-object interaction.
• EgoDyn-Bench : A diagnostic benchmark for evaluating ego-motion understanding and physical grounding in VLMs, MLLMs, and driving VLA models.
• WorldLens : A full-spectrum benchmark for driving world models, evaluating generation, reconstruction, action-following, downstream-task usefulness, and human preference with WorldLens-26K annotations.
• BDD-X (Berkeley DeepDrive eXplanation) : This dataset provides textual explanations for driving actions, making it particularly relevant for training and evaluating interpretable AD models. DriveGPT4 and ADAPT are evaluated on BDD-X. It contains video sequences with corresponding control signals and natural language narrations/reasoning.
• Waymo Open Dataset (WOMD) : A large and diverse dataset with high-resolution sensor data, including LiDAR and camera imagery. Used in works like OmniDrive for Q&A data generation and by LLMs Powered Context-aware Motion Prediction. Also used for scene simulation in ChatSim. WOMD-Reasoning is a language dataset built upon WOMD focusing on interaction descriptions and driving intentions.
• DriveLM : A benchmark and dataset focusing on driving with graph visual question answering. TS-VLM is evaluated on DriveLM. It aims to assess perception, prediction, and planning reasoning through QA pairs in a directed graph, with versions for CARLA and nuScenes.
• LingoQA : A benchmark and dataset specifically designed for video question answering in autonomous driving. It contains over 419k QA pairs from 28k unique video scenarios, covering driving reasoning, object recognition, action justification, and scene description. It also proposes the Lingo-Judge evaluation metric.
• DriveAction : DriveAction leverages real-world driving data actively collected by users of production-level autonomous vehicles to ensure broad and representative scenario coverage, provides high-level discrete behavior labels collected directly from users' actual driving operations, and implements a behavior-based tree-structured evaluation framework that explicitly links vision, language, and behavioral tasks to support comprehensive and task-specific evaluation.
• CARLA Simulator & Datasets : While a simulator, CARLA is extensively used to generate data and evaluate AD models in closed-loop settings. Works like LeapAD , LMDrive , and LangProp use CARLA for experiments and data collection. DriveLM-Carla is a specific dataset generated using CARLA.
• Argoverse : A dataset suite with a focus on motion forecasting, 3D tracking, and HD maps. Argoverse 2 is used in challenges like 3D Occupancy Forecasting.
• KITTI : One of the pioneering datasets for autonomous driving, still used for tasks like 3D object detection and tracking.
• Cityscapes : Focuses on semantic understanding of urban street scenes, primarily for semantic segmentation.
• UCU Dataset (In-Cabin User Command Understanding) : Part of the LLVM-AD Workshop, this dataset contains 1,099 labeled user commands for autonomous vehicles, designed for training models to understand human instructions within the vehicle.
• MAPLM (Large-Scale Vision-Language Dataset for Map and Traffic Scene Understanding) : Also from the LLVM-AD Workshop, MAPLM combines point cloud BEV and panoramic images for rich road scenario images and multi-level scene description data, used for QA tasks.
• NuPrompt : A large-scale language prompt set based on nuScenes for driving scenes, consisting of 3D object-text pairs, used in Prompt4Driving.
• nuDesign : A large-scale dataset (2300k sentences) constructed upon nuScenes via a rule-based auto-labeling methodology for 3D dense captioning.
• LaMPilot : An interactive environment and dataset designed for evaluating LLM-based agents in a driving context, containing scenes for command tracking tasks.
• DRAMA (Joint Risk Localization and Captioning in Driving) : Provides linguistic descriptions (with a focus on reasons) of driving risks associated with important objects.
• Rank2Tell : A multimodal ego-centric dataset for ranking importance levels of objects/events and generating textual reasons for the importance.
• HighwayEnv : A collection of environments for autonomous driving and tactical decision-making research, often used for RL-based approaches and LLM decision-making evaluations (e.g., by DiLu, MTD-GPT).

These repositories offer broader collections of resources that may overlap with or complement the focus of this list.

• Awesome-LLM4AD (LLM for Autonomous Driving):
  • GitHub: https://github.com/Thinklab-SJTU/Awesome-LLM4AD
  • Alternative Link:http://codesandbox.io/p/github/sorokinvld/Awesome-LLM4AD
  • Description: A curated list of research papers about LLM-for-Autonomous-Driving, categorized by planning, perception, question answering, and generation. Continuously updated.
• Awesome-VLLMs (Vision Large Language Models):
  • GitHub: JackYFL/awesome-VLLMs
  • Description: Collects papers on Visual Large Language Models, with a dedicated section for "Vision-to-action" including "Autonomous driving" (Perception, Planning, Prediction).
• Awesome-Data-Centric-Autonomous-Driving:
  • GitHub: https://github.com/LincanLi98/Awesome-Data-Centric-Autonomous-Driving
  • Description: Focuses on data-driven AD solutions, including datasets, data mining, and closed-loop technologies. Mentions the role of LLMs/VLMs in scene understanding and decision-making.
• Awesome-World-Model (for Autonomous Driving and Robotics):
  • GitHub: https://github.com/LMD0311/Awesome-World-Model
  • Description: Records, tracks, and benchmarks recent World Models for AD or Robotics, supplementary to a survey paper. Includes many VLA-related and generative model papers.
• Awesome-Multimodal-LLM-Autonomous-Driving:
  • GitHub: https://github.com/IrohXu/Awesome-Multimodal-LLM-Autonomous-Driving
  • Description: A systematic investigation of Multimodal LLMs in autonomous driving, covering background, tools, frameworks, datasets, and future directions.
• Awesome VLM Architectures:
  • GitHub: gokayfem/awesome-vlm-architectures
  • Description: Contains information on famous Vision Language Models (VLMs), including details about their architectures, training procedures, and datasets.

If you find this repository helpful, a citation to our paper would be greatly appreciated:

@ARTICLE{xu2026survey,
  author={Xu, Chengkai and Cui, Yiming and Liu, Jiaqi and Guo, Yicheng and Qin, Cheng and Zhang, Geyuan and Dong, Xinwei and Fang, Shiyu and Hang, Peng and Sun, Jian},
  journal={IEEE Transactions on Intelligent Transportation Systems}, 
  title={A Survey on End-to-End Autonomous Driving Training From the Perspectives of Data, Strategy, and Platform}, 
  year={2026},
  volume={},
  number={},
  pages={1-20},
  keywords={Modeling;Training;Optimization;Autonomous driving;Safety;Surveys;Vehicles;Testing;Learning (artificial intelligence);Reinforcement learning;Autonomous vehicle;end-to-end;artificial intelligence;intelligent transportation system},
  doi={10.1109/TITS.2026.3695999}
}