Temporal Attention Mechanism with Conditional Inference for Large-Scale Multi-Label Video Classification

ECCV Workshop on the YouTube-8M Large-Scale Video Understanding Challenge (2018)


Here we show neural network based methods, which combine multimodal sequential inputs effectively and classify the inputs into multiple categories. Two key ideas are 1) to select informative frames among a sequence using attention mechanism and 2) to utilize correlation information between labels to solve multi-label classification problems. The attention mechanism is used in both modality(spatio) and sequential (temporal) dimensions to ignore noisy and meaningless frames. Furthermore, to tackle fundamental problems induced by independently predicting each label in conventional multi-label classification methods, the proposed method considers the dependencies among the labels by decomposing joint probability of labels into conditional terms. From the experimental results (5th in the Kaggle competition), we discuss how the suggested methods operate in the YouTube-8M Classification Task, what insights they have, and why they succeed or fail.


김은솔 (카카오브레인), 온경운 (서울대학교), 김종석 (카카오브레인), 허유정 (서울대학교), 최성호 (서울대학교), 이현동(서울대학교/컬럼비아 대학교), 장병탁 (서울대학교)


multimodal sequential learning Attention multi-label classification video understanding

발행 날짜