Multimodal Feature Fusion for Wide-Angle Image Generation

Quanxing Peng

Abstract: Image stitching combines visible light images from various perspectives to create wide-angle composites. However, adverse weather degrades these images, compromising stitching quality. Infrared sensors, which capture thermal radiation, excel in such conditions by highlighting targets. To overcome these challenges, we propose a multimodal fusion approach that integrates the robustness of infrared imaging with the rich textures of visible light. Our method uses a coarse-to-fine offset estimation based on infrared structural features and visible texture details, followed by a non-parametric Direct Linear Transformation for accurate geometric alignment, and finally fuses the stitched images to enhance scene perception. Tested on a real dataset of 530 multimodal pairs and a synthetic set of 200 pairs, our approach reduces average corner point error by 53%, eliminates ghosting, and boosts information entropy by 24.6% over DATFuse-UDIS++, demonstrating superior robustness and accuracy.

Keywords: multimodal image fusion, infrared?visible stitching, wide-angle panorama, homography estimation, deep learning