Details

Autor(en) / Beteiligte

• Zuxuan Wu
• Yanwei Fu
• Yu-Gang Jiang
• Sigal, Leonid

Titel

Harnessing Object and Scene Semantics for Large-Scale Video Understanding

Ist Teil von

• 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, p.3112-3121

Ort / Verlag

IEEE

Erscheinungsjahr

2016

Link zum Volltext

• IEEE_Xplore

Quelle

IEEE Electronic Library (IEL)

Beschreibungen/Notizen

• Large-scale action recognition and video categorization are important problems in computer vision. To address these problems, we propose a novel object-and scene-based semantic fusion network and representation. Our semantic fusion network combines three streams of information using a three-layer neural network: (i) frame-based low-level CNN features, (ii) object features from a state-of-the-art large-scale CNN object-detector trained to recognize 20K classes, and (iii) scene features from a state-of-the-art CNN scene-detector trained to recognize 205 scenes. The trained network achieves improvements in supervised activity and video categorization in two complex large-scale datasets - ActivityNet and FCVID, respectively. Further, by examining and back propagating information through the fusion network, semantic relationships (correlations) between video classes and objects/scenes can be discovered. These video class-object/video class-scene relationships can in turn be used as semantic representation for the video classes themselves. We illustrate effectiveness of this semantic representation through experiments on zero-shot action/video classification and clustering.