Abstract:
Multi-Modal Self-Supervised Learning from videos have been shown to improve model's performance on various downstream tasks. However, such Self-Supervised pre-training requires large batch sizes and large amount of computation resources due to the noise present in the uncurated data. This is partly due to that the prevalent training scheme are trained on coarse-grained setting, in which vectors representing the whole video clips or natural language sentences are used for computing similarity. Such scheme makes training noisy as part of the video clips can be totally not correlated with the other-modality input such as text description. In this paper, we propose a fine-grained Multi-Modal Self-Supervised training scheme that computes similarity between embeddings at finer-scale (such as individual feature map embeddings and embeddings of phrases),and uses attention mechanism to reduce noisy pairs' weighting in the loss function. We show that with the proposed pre-training scheme, we can train smaller models, with smaller batch-size and much less computational resource to achieve downstream tasks performances comparable to State-Of-The-Art, for tasks including action recognition and text-image retrievals.
