AdaptLPS

AdaptLPS Demo

The demo demonstrates the performance of our AdaptLPS model on three different source-target dataset pairs: SemanticKITTI → nuScenes, SemanticKITTI → PandaSet and nuScenes → PandaSet. AdaptLPS is the first unsupervised domain adaptation approach for panoptic segmentation of LiDAR point clouds. View the demo by selecting a dataset pair and a scan from the menu below. Use left mouse click to rotate the camera and mouse wheel to zoom in and out.

Select a scan

No domain adaptation

AdaptLPS (Ours)

Abstract

Scene understanding is a pivotal task for autonomous vehicles to safely navigate in the environment. Recent advances in deep learning enable accurate semantic reconstruction of the surroundings from LiDAR data. However, these models encounter a large domain gap while deploying them on vehicles equipped with different LiDAR setups which drastically decreases their performance. Fine-tuning the model for every new setup is infeasible due to the expensive and cumbersome process of recording and manually labeling new data. Unsupervised Domain Adaptation (UDA) techniques are thus essential to fill this domain gap and retain the performance of models on new sensor setups without the need for additional data labeling. In this paper, we propose AdaptLPS, a novel UDA approach for LiDAR panoptic segmentation that leverages task-specific knowledge and accounts for variation in the number of scan lines, mounting position, intensity distribution, and environmental conditions. We tackle the UDA task by employing two complementary domain adaptation strategies, data-based and model-based. While data-based adaptations reduce the domain gap by processing the raw LiDAR scans to resemble the scans in the target domain, model-based techniques guide the network in extracting features that are representative for both domains. Extensive evaluations on three pairs of real-world autonomous driving datasets demonstrate that AdaptLPS outperforms existing UDA approaches by up to 6.41 pp in terms of the PQ score.

How Does It Work?

Our approach comprises two domain adaptation strategies, namely, data-based and model-based. The data-based domain adaptation minimizes the domain gap between the raw LiDAR scans from the source and target domains using three techniques, namely, (i) pose correction which accounts for the different mounting positions of LiDARs, (ii) virtual scan generation which simulates the point cloud from the target domain using the source domain, and (iii) intensity mapping which learns a residual to map the LiDAR intensities between the source and target domains. In contrast, model-based domain adaptation minimizes the domain gap between the panoptic segmentation models by aligning the source and target distributions using our novel multi-scale feature-space optimal transport augmented with instance-aware sampling. During inference, we further reduce the domain gap using our efficient PDC-Lite technique which re-calibrates the first batch normalization layer to remove any bias and variance accumulated from the source domain.

Figure: Illustration of our AdaptLPS domain adaptation approach. We employ data-based (pose correction, virtual scan generation, and intensity mapping) and model-based (optimal transport and instance-aware sampling) domain adaptation strategies to reduce the domain gap between source and target datasets.

Using extensive evaluations on three different dataset pairs, we demonstrate that our AdaptLPS strategy consistently outperforms existing domain adaptation techniques on both panoptic and semantic segmentation tasks. Lastly, we present a detailed ablation study along with a comprehensive qualitative analysis that highlights the improvement brought about by the different constituent elements of our domain adaption strategy.

Videos

Code

A software implementation of this project based on PyTorch can be found in our GitHub repository for academic usage and is released under the GPLv3 license. For any commercial purpose, please contact the authors.

Publications

Borna Bešić, Nikhil Gosala, Daniele Cattaneo, Abhinav Valada
Unsupervised Domain Adaptation for LiDAR Panoptic Segmentation
arXiv preprint arXiv:2109.15286, 2020.

(PDF) (BibTeX)

Acknowledgement

This work was funded by the Eva Mayr-Stihl Stiftung and the Federal Ministry of Education and Research (BMBF) of Germany under ISA 4.0.

People