Wei Zhan

Wei Zhan serves as a Co-Director of Berkeley DeepDrive, a research center at UC Berkeley focusing on AI for autonomy, mobility and robotic applications. He is an Assistant Professional Researcher at UC Berkeley leading the autonomy group of MSC Lab with 10+ Ph.D. students and Postdoc. His research interest lies in AI for autonomous systems leveraging control, robotics, computer vision and machine learning to tackle challenges involving a large variety of sophisticated dynamics, interactive human behavior and complex scenes in a scalable way, and make the autonomous system sustainably evolve.

He received his Ph.D. degree from UC Berkeley. His publications received the Best Student Paper Award in IV’18, Best Paper Award – Honorable Mention of IEEE Robotics and Automation Letters, and Best Paper Award Finalist of ICRA’24. One of his publications also got ICLR’23 notable top 5% oral presentation. He led the construction of INTERACTION dataset and the organization of its prediction challenges in NeurIPS’20 and ICCV’21.

He is actively looking for talented applicants (as Postdoc, Ph.D. student, visiting student or local Berkeley Master’s/undergrad student) to working with. Please fill in the form if you are interested.

Policy Customization

• Residual Q-Learning – offline and online policy customization without value: NeurIPS’23, arxiv, Website, Code

Autonomous Racing – Learning to Plan and Control at the Limits

• Active exploration for modeling dynamics and racing behavior: IEEE Trans-CST ’24, arxiv
• Skill-Critic – refining learned skills for reinforcement learning: RA-Letters ’24, arxiv, Website
• BeTAIL- behavior transformer adversarial imitation learning: arxiv, Website
• Double-iterative GP for model error compensation: IFAC’23, arxiv
• Outracing human racers with MPC: arxiv

Language and LLM for Decision and Planning

• LanguageMPC – LLMs as decision makers: arxiv, Website

LLM for Code Diagnosis and Repair

• Diagnosis and repair of motion planners by LLM: arxiv

Generalizable Robot Learning with Language

• PhyGrasp – generalizing grasping with physics-informed large multimodal models: arxiv, Website

Human-to-Robot Cross-Embodiment

• Representation learning from general human demonstrations to robot manipulation : arxiv, Website

Open X-Embodiment

• Open X-Embodiment – Robotic Learning Datasets and RT-X Models: ICRA’24 (Best Paper Finalist), arxiv, Blog, Dataset, Website, Code

3D Perception with Temporal, Multi-View Images

• SOLOFusion – temporal multi-view 3D detection: ICLR’23 (notable top 5% oral presentation), arxiv, Code
• NeRF-Det – learning geometry-aware volumetric representation: ICCV’23, arxiv, Code

3D Reconstruction and Localization with Efficient Representation

• Q-SLAM – quadric representations for monocular SLAM: arxiv
• Quadric representations for LiDAR odometry, mapping and localization: RA-Letters ’23, arxiv

Self-Supervised Learning for End-to-End Autonomy

• Cohere3D – temporal coherence for self-supervision for perception, prediction and planning: arxiv
• PreTraM – self-supervision connecting trajectory and map: ECCV’22, arxiv, Code
• Prediction with synthetic data pretraining: arxiv
• Image2Point – 2D pretraining for 3D understanding: ECCV’22, arxiv, Code

Recent and Continued

Efficient, Automated Data Engine and Training Pipeline

• Free Data selection with general-purpose models: NeurIPS’23, arxiv.
• Active finetuning: CVPR’23, arxiv
• Doubly-robust self-training: NeurIPS’23, arxiv
• Cross-modality semi-supervised learning for 3D detection: ECCV’22, arxiv, Code

3D Perception with Early-Stage LiDAR-Camera Fusion

• SparseFusion – fusing multi-modal sparse representations: ICCV’23, arxiv, Code
• Fusing BEV point cloud and front-view image: IV’18, arxiv, Code

Online Semantic HD map Construction and Scene Understanding

• Auto construction of semantic HD maps: IROS’21, arxiv

Multi-Agent, Interactive Prediction with Interpretability

• Multi-agent prediction combining egocentric and allocentric views: CoRL’21
• Social posterior collapse in variational autoencoder: NeurIPS’21, arxiv
• Interventional behavior prediction: IROS’22, arxiv
• Interpretable goal-conditioned interactive prediction: IROS’22, arxiv

INTERACTION Dataset and Benchmark

• INTERACTION dataset with critical scenes and densely interactive behavior: Website, arxiv, IROS’19
• INTERPRET challenge benchmarking conditional, multi-agent prediction: Website

LiDAR-based Perception

• SqueezeSegV3 – spatially-adaptive convolution for segmentation: ECCV’20, arxiv, Code
• Labels Are Not Perfect – inferring spatial uncertainty in detection: IEEE Trans-ITS ’21, IROS’20, arxiv
• Multi-task learning: IROS’21

2D Perception

• Sparse R-CNN: IEEE Trans-PAMI ’23, CVPR’21, arxiv
• Autoscale: IJCV ’22, arxiv

Vehicle Dynamics and Control

• Dual Extended Kalman Filter for state and parameter estimation: ITSC’21
• Remote control with slow sensor: Sensors ’19