IOS: Inter-Operator Scheduler for CNN Acceleration

Abstract: To accelerate CNN inference, existing deep learning frameworks focus on optimizing intra-operator parallelization. However, a single operator can no longer fully utilize the available parallelism given the rapid advances in high-performance hardware, resulting in a large gap between the peak performance and the real performance. This performance gap is more severe under smaller batch sizes. In this work, we extensively study the parallelism between operators and propose Inter-Operator Scheduler (IOS) to automatically schedule multiple operators' parallel execution through a novel dynamic programming algorithm. IOS consistently outperforms state-of-the-art libraries (e.g., TensorRT) by 1.1 to 1.5x on modern CNN benchmarks. The code to reproduce each experiment is available at: https://github.com/mit-han-lab/inter-operator-scheduler.

26/04/2020

high-dynamic-range imaging, point-spread-function engineering, end-to-end learning, computational imaging, deep learning, optics, photography

5:01

06/12/2021

Multi-Precision Policy Enforced Training (MuPPET) : A Precision-Switching Strategy for Quantised Fixed-Point Training of CNNs

neural architecture search, mobilenet, tensorrt, latency, classification, imagenet, viterbi, network width search, ows, aows

4:55

06/12/2020

non-rigid tracking, learnable optimization, differentiable solver, non-rigid icp, gauss newton, pcg, preconditioning, non-linear optimization, 4d perception, deep learning

4:56

19/10/2020

Colby Banbury, Chuteng Zhou, Igor Fedorov and
Ramon Matas, Urmish Thakker, Dibakar Gope, Vijay Janapa Reddi, Matthew Mattina, Paul Whatmough

Keywords Paper

2:47

05/04/2021

MicroNets: Neural Network Architectures for Deploying TinyML Applications on Commodity Microcontrollers

Colby Banbury, Chuteng Zhou, Igor Fedorov and
Ramon Matas, Urmish Thakker, Dibakar Gope, Vijay Janapa Reddi, Matthew Mattina, Paul Whatmough

Sai Praneeth Karimireddy, Martin Jaggi, Satyen Kale and
Mehryar Mohri, Sashank Reddi, Sebastian Stich, Ananda Theertha Suresh

Lingxiao Ma, Zhiqiang Xie, Zhi Yang and
Jilong Xue, Youshan Miao, Wei Cui, Wenxiang Hu, Fan Yang, Lintao Zhang, Lidong Zhou

Reinforcement Learning, Learning to Optimize, Combinatorial Optimization, Compilers, Code Optimization, Neural Networks, ML for Systems, Learning for Systems

4:55

26/04/2020

video deblurring, deep convolutional neural network, motion blur estimation, optical flow, temporal sharpness prior, image restoration

0:53

06/12/2020