Abstract:
Video relation detection forms a new and challenging problem in computer vision, where subjects and objects need to be spatio-temporally localized and a predicate label needs to be assigned if and only if there is an interaction between the two. Despite recent progress in video relation detection, overall performance is still marginal and it remains unclear what the key factors are towards solving the problem. Following examples set in the object detection and action localization literature, we perform a deep dive into the error diagnosis of current video relation detection approaches. We introduce a diagnostic tool for analyzing the sources of detection errors. Our tool evaluates and compares current approaches beyond a single scalar metric, the mean Average Precision, by defining different error types specific to video relation detection, used for false positive analyses. Moreover, we examine different factors of influence on the performance in false negative analyses, including relation length, number of subject/object/predicate instances, and subject/object size. Finally, we present the effect on the video related performances with an oracle fix for each error type. On the existing video relation benchmarks, we show where current approaches excel and fall short, allowing us to pinpoint the most important future directions in the field.
