Abstract

Recent research has demonstrated that the fusion of complementary information captured by multi-modal sensors (visible and infrared cameras) enables robust pedestrian detection under various surveillance situations (e.g., daytime and nighttime). In this paper, we investigate a number of fusion architectures in an attempt to identify the optimal way of incorporating multispectral information for joint semantic segmentation and pedestrian detection. We made two important findings: (1) the sum fusion strategy, which computes the sum of two feature maps at the same spatial locations, delivers the best performance of multispectral detection, while the most commonly used concatenation fusion surprisingly performs the worst; and (2) two-stream semantic segmentation without multispectral fusion is the most effective scheme to infuse semantic information as supervision for learning human-related features. Based on these studies, we present a unified multispectral fusion framework for joint training of semantic segmentation and target detection that outperforms state-of-the-art multispectral pedestrian detectors by a large margin on the KAIST benchmark dataset.

© 2018 Optical Society of America

Multispectral image fusion based pedestrian detection using a multilayer fused deconvolutional single-shot detector

Yunfan Chen and Hyunchul Shin
J. Opt. Soc. Am. A 37(5) 768-779 (2020)

MAFFNet: real-time multi-level attention feature fusion network with RGB-D semantic segmentation for autonomous driving

Tongfei Lv, Yu Zhang, Lin Luo, and Xiaorong Gao
Appl. Opt. 61(9) 2219-2229 (2022)

Pedestrian detection using a translation-invariant wavelet residual dense super-resolution

Wei-Yen Hsu and Pei-Ci Chen
Opt. Express 30(23) 41279-41295 (2022)

Learning feature fusion for target detection based on polarimetric imaging

Sihao Gao, Yu Cao, Wenjing Zhang, Qian Dai, Jun Li, and Xiaojun Xu
Appl. Opt. 61(7) D15-D21 (2022)

Polarization-driven camouflaged object segmentation via gated fusion

Bingyang Fu, Tieyong Cao, Yunfei Zheng, Zheng Fang, Lei Chen, Yang Wang, Yekui Wang, and Yong Wang
Appl. Opt. 61(27) 8017-8027 (2022)