Details
| Original language | English |
|---|---|
| Pages (from-to) | 137-144 |
| Number of pages | 8 |
| Journal | International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives |
| Volume | 48 |
| Issue number | 2 |
| Publication status | Published - 11 Jun 2024 |
| Event | ISPRS TC II Mid-term Symposium on the Role of Photogrammetry for a Sustainable World - Las Vegas, United States Duration: 11 Jun 2024 → 14 Jun 2024 |
Abstract
Neural Radiance Fields (NeRFs) have become a rapidly growing research field with the potential to revolutionize typical photogrammetric workflows, such as those used for 3D scene reconstruction. As input, NeRFs require multi-view images with corresponding camera poses as well as the interior orientation. In the typical NeRF workflow, the camera poses and the interior orientation are estimated in advance with Structure from Motion (SfM). But the quality of the resulting novel views, which depends on different parameters such as the number and distribution of available images, the accuracy of the related camera poses and interior orientation, but also the reflection characteristics of the depicted scene, is difficult to predict. In addition, SfM is a time-consuming pre-processing step, and its robustness and quality strongly depend on the image content. Furthermore, the undefined scaling factor of SfM hinders subsequent steps in which metric information is required. In this paper, we evaluate the potential of NeRFs for industrial robot applications. To start with, we propose an alternative to SfM pre-processing: we capture the input images with a calibrated camera that is attached to the end effector of an industrial robot and determine accurate camera poses with metric scale based on the robot kinematics. We then investigate the quality of the novel views by comparing them to ground truth, and by computing an internal quality measure based on ensemble methods. For evaluation purposes, we acquire multiple datasets that pose challenges for reconstruction typical of industrial applications, like reflective objects, poor texture, and fine structures. We show that the robot-based pose determination reaches similar accuracy as SfM in non-demanding cases, while having clear advantages in more challenging scenarios. We also report results of the novel view quality for different NeRF approaches, showing that an additional online pose refinement may be disadvantageous. Finally, we present first results of applying the ensemble method to estimate the quality of the synthetic novel view in the absence of a ground truth.
Keywords
- industrial applications, neural radiance fields, novel view synthesis, robotics, uncertainty estimation
ASJC Scopus subject areas
- Computer Science(all)
- Information Systems
- Social Sciences(all)
- Geography, Planning and Development
Cite this
- Standard
- Harvard
- Apa
- Vancouver
- BibTeX
- RIS
In: International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives, Vol. 48, No. 2, 11.06.2024, p. 137-144.
Research output: Contribution to journal › Conference article › Research › peer review
}
TY - JOUR
T1 - Novel View Synthesis with Neural Radiance Fields for Industrial Robot Applications
AU - Hillemann, Markus
AU - Langendörfer, Robert
AU - Heiken, Max
AU - Mehltretter, Max
AU - Schenk, Andreas
AU - Weinmann, Martin
AU - Hinz, Stefan
AU - Heipke, Christian
AU - Ulrich, Markus
N1 - Publisher Copyright: © Author(s) 2024.
PY - 2024/6/11
Y1 - 2024/6/11
N2 - Neural Radiance Fields (NeRFs) have become a rapidly growing research field with the potential to revolutionize typical photogrammetric workflows, such as those used for 3D scene reconstruction. As input, NeRFs require multi-view images with corresponding camera poses as well as the interior orientation. In the typical NeRF workflow, the camera poses and the interior orientation are estimated in advance with Structure from Motion (SfM). But the quality of the resulting novel views, which depends on different parameters such as the number and distribution of available images, the accuracy of the related camera poses and interior orientation, but also the reflection characteristics of the depicted scene, is difficult to predict. In addition, SfM is a time-consuming pre-processing step, and its robustness and quality strongly depend on the image content. Furthermore, the undefined scaling factor of SfM hinders subsequent steps in which metric information is required. In this paper, we evaluate the potential of NeRFs for industrial robot applications. To start with, we propose an alternative to SfM pre-processing: we capture the input images with a calibrated camera that is attached to the end effector of an industrial robot and determine accurate camera poses with metric scale based on the robot kinematics. We then investigate the quality of the novel views by comparing them to ground truth, and by computing an internal quality measure based on ensemble methods. For evaluation purposes, we acquire multiple datasets that pose challenges for reconstruction typical of industrial applications, like reflective objects, poor texture, and fine structures. We show that the robot-based pose determination reaches similar accuracy as SfM in non-demanding cases, while having clear advantages in more challenging scenarios. We also report results of the novel view quality for different NeRF approaches, showing that an additional online pose refinement may be disadvantageous. Finally, we present first results of applying the ensemble method to estimate the quality of the synthetic novel view in the absence of a ground truth.
AB - Neural Radiance Fields (NeRFs) have become a rapidly growing research field with the potential to revolutionize typical photogrammetric workflows, such as those used for 3D scene reconstruction. As input, NeRFs require multi-view images with corresponding camera poses as well as the interior orientation. In the typical NeRF workflow, the camera poses and the interior orientation are estimated in advance with Structure from Motion (SfM). But the quality of the resulting novel views, which depends on different parameters such as the number and distribution of available images, the accuracy of the related camera poses and interior orientation, but also the reflection characteristics of the depicted scene, is difficult to predict. In addition, SfM is a time-consuming pre-processing step, and its robustness and quality strongly depend on the image content. Furthermore, the undefined scaling factor of SfM hinders subsequent steps in which metric information is required. In this paper, we evaluate the potential of NeRFs for industrial robot applications. To start with, we propose an alternative to SfM pre-processing: we capture the input images with a calibrated camera that is attached to the end effector of an industrial robot and determine accurate camera poses with metric scale based on the robot kinematics. We then investigate the quality of the novel views by comparing them to ground truth, and by computing an internal quality measure based on ensemble methods. For evaluation purposes, we acquire multiple datasets that pose challenges for reconstruction typical of industrial applications, like reflective objects, poor texture, and fine structures. We show that the robot-based pose determination reaches similar accuracy as SfM in non-demanding cases, while having clear advantages in more challenging scenarios. We also report results of the novel view quality for different NeRF approaches, showing that an additional online pose refinement may be disadvantageous. Finally, we present first results of applying the ensemble method to estimate the quality of the synthetic novel view in the absence of a ground truth.
KW - industrial applications
KW - neural radiance fields
KW - novel view synthesis
KW - robotics
KW - uncertainty estimation
UR - http://www.scopus.com/inward/record.url?scp=85197339144&partnerID=8YFLogxK
U2 - 10.48550/arXiv.2405.04345
DO - 10.48550/arXiv.2405.04345
M3 - Conference article
AN - SCOPUS:85197339144
VL - 48
SP - 137
EP - 144
JO - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives
JF - International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences - ISPRS Archives
SN - 1682-1750
IS - 2
T2 - ISPRS TC II Mid-term Symposium on the Role of Photogrammetry for a Sustainable World
Y2 - 11 June 2024 through 14 June 2024
ER -