Details
| Original language | English |
|---|---|
| Pages (from-to) | 153-161 |
| Number of pages | 9 |
| Journal | ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences |
| Issue number | X-2-2024 |
| Publication status | Published - 10 Jun 2024 |
| Event | 2024 ISPRS TC II Mid-term Symposium on The Role of Photogrammetry for a Sustainable World - Las Vegas, United States Duration: 11 Jun 2024 → 14 Jun 2024 |
Abstract
Panoptic segmentation unifies semantic and instance segmentation and thus delivers a semantic class label and, for so-called thing classes, also an instance label per pixel. The differentiation of distinct objects of the same class with a similar appearance is particularly challenging and frequently causes such objects to be incorrectly assigned to a single instance. In the present work, we demonstrate that information on the 3D geometry of the observed scene can be used to mitigate this issue: We present a novel CNN-based method for panoptic segmentation which processes RGB images and depth maps given as input in separate network branches and fuses the resulting feature maps in a late fusion manner. Moreover, we propose a new depth-aware dice loss term which penalises the assignment of pixels to the same thing instance based on the difference between their associated distances to the camera. Experiments carried out on the Cityscapes dataset show that the proposed method reduces the number of objects that are erroneously merged into one thing instance and outperforms the method used as basis by +2.2% in terms of panoptic quality.
Keywords
- Dice Loss, Panoptic Segmentation, RGB Depth Fusion
ASJC Scopus subject areas
- Physics and Astronomy(all)
- Instrumentation
- Environmental Science(all)
- Environmental Science (miscellaneous)
- Earth and Planetary Sciences(all)
- Earth and Planetary Sciences (miscellaneous)
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
In: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, No. X-2-2024, 10.06.2024, p. 153-161.
Research output: Contribution to journal › Conference article › Research › peer review
}
TY - JOUR
T1 - Depth-Aware Panoptic Segmentation
AU - Nguyen, Tuan
AU - Mehltretter, Max
AU - Rottensteiner, Franz
N1 - Publisher Copyright: © Author(s) 2024.
PY - 2024/6/10
Y1 - 2024/6/10
N2 - Panoptic segmentation unifies semantic and instance segmentation and thus delivers a semantic class label and, for so-called thing classes, also an instance label per pixel. The differentiation of distinct objects of the same class with a similar appearance is particularly challenging and frequently causes such objects to be incorrectly assigned to a single instance. In the present work, we demonstrate that information on the 3D geometry of the observed scene can be used to mitigate this issue: We present a novel CNN-based method for panoptic segmentation which processes RGB images and depth maps given as input in separate network branches and fuses the resulting feature maps in a late fusion manner. Moreover, we propose a new depth-aware dice loss term which penalises the assignment of pixels to the same thing instance based on the difference between their associated distances to the camera. Experiments carried out on the Cityscapes dataset show that the proposed method reduces the number of objects that are erroneously merged into one thing instance and outperforms the method used as basis by +2.2% in terms of panoptic quality.
AB - Panoptic segmentation unifies semantic and instance segmentation and thus delivers a semantic class label and, for so-called thing classes, also an instance label per pixel. The differentiation of distinct objects of the same class with a similar appearance is particularly challenging and frequently causes such objects to be incorrectly assigned to a single instance. In the present work, we demonstrate that information on the 3D geometry of the observed scene can be used to mitigate this issue: We present a novel CNN-based method for panoptic segmentation which processes RGB images and depth maps given as input in separate network branches and fuses the resulting feature maps in a late fusion manner. Moreover, we propose a new depth-aware dice loss term which penalises the assignment of pixels to the same thing instance based on the difference between their associated distances to the camera. Experiments carried out on the Cityscapes dataset show that the proposed method reduces the number of objects that are erroneously merged into one thing instance and outperforms the method used as basis by +2.2% in terms of panoptic quality.
KW - Dice Loss
KW - Panoptic Segmentation
KW - RGB Depth Fusion
UR - http://www.scopus.com/inward/record.url?scp=85199892117&partnerID=8YFLogxK
U2 - 10.48550/arXiv.2405.10947
DO - 10.48550/arXiv.2405.10947
M3 - Conference article
AN - SCOPUS:85199892117
SP - 153
EP - 161
JO - ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences
JF - ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences
SN - 2194-9042
IS - X-2-2024
T2 - 2024 ISPRS TC II Mid-term Symposium on The Role of Photogrammetry for a Sustainable World
Y2 - 11 June 2024 through 14 June 2024
ER -