Neural Architecture Search as Sparse Supernet

02/02/2021

Neural Architecture Search as Sparse Supernet

Yan Wu, Aoming Liu, Zhiwu Huang, Siwei Zhang, Luc Van Gool

Keywords:

Abstract Paper Similar Papers

Abstract: This paper aims at enlarging the problem of Neural Architecture Search (NAS) from Single-Path and Multi-Path Search to automated Mixed-Path Search. In particular, we model the NAS problem as a sparse supernet using a new continuous architecture representation with a mixture of sparsity constraints. The sparse supernet enables us to automatically achieve sparsely-mixed paths upon a compact set of nodes. To optimize the proposed sparse supernet, we exploit a hierarchical accelerated proximal gradient algorithm within a bi-level optimization framework. Extensive experiments on Convolutional Neural Network and Recurrent Neural Network search demonstrate that the proposed method is capable of searching for compact, general and powerful neural architectures.

The video of this talk cannot be embedded. You can watch it here:

https://slideslive.com/38948634

(Link will open in new window)

0

0

0

0

Share

This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment

no comments yet

Similar Papers

03/05/2021

A Design Space Study for LISTA and Beyond

Tianjian Meng, Xiaohan Chen, Yifan Jiang, Zhangyang Wang

Keywords Paper

0

0

0

0

5:50

14/06/2020

GP-NAS: Gaussian Process Based Neural Architecture Search

Zhihang Li, Teng Xi, Jiankang Deng and
Gang Zhang, Shengzhao Wen, Ran He

Keywords Paper

neural architecture search, gaussian process, image classification, face recognition

0

0

0

0

0:59

06/12/2020

Adapting Neural Architectures Between Domains

Yanxi Li, Zhaohui Yang, Yunhe Wang, Chang Xu

Keywords Paper

0

0

0

0

3:20

19/08/2021

Neural Architecture Search of SPD Manifold Networks

Rhea Sanjay Sukthanker, Zhiwu Huang, Suryansh Kumar and
Erik Goron Endsjo, Yan Wu, Luc Van Gool

Keywords Paper

Machine Learning, Classification, Deep Learning, Networks

0

0

0

0

14:16

18/07/2021

On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent

Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson and
Blake Woodworth, Nati Srebro, Amir Globerson, Daniel Soudry

Keywords Paper

, Probabilistic Methods, MCMC, Theory, Deep learning Theory

0

0

0

0

15:38

26/04/2020

Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers

Junjie LIU, Zhe XU, Runbin SHI and
Ray C. C. Cheung, Hayden K.H. So

Keywords Paper

neural network pruning, sparse learning, network compression, architecture search

0

0

0

0

4:49

14/06/2020

MTL-NAS: Task-Agnostic Neural Architecture Search Towards General-Purpose Multi-Task Learning

Yuan Gao, Haoping Bai, Zequn Jie and
Jiayi Ma, Kui Jia, Wei Liu

Keywords Paper

neural architecture search, general-purpose multi-task learning, task-agnostic search space, single-shot gradient-based search algorithm, minimal entropy regularization

0

0

1

0

1:00

02/02/2021

HINT: Hierarchical Invertible Neural Transport for Density Estimation and Bayesian Inference

Jakob Kruse, Gianluca Detommaso, Ullrich Köthe, Robert Scheichl

Keywords Paper

0

0

0

0

19:03

14/06/2020

Densely Connected Search Space for More Flexible Neural Architecture Search

Jiemin Fang, Yuzhu Sun, Qian Zhang and
Yuan Li, Wenyu Liu, Xinggang Wang

Keywords Paper

neural architecture search (nas), densely connected search space

0

0

0

0

1:00

06/12/2020

The phase diagram of approximation rates for deep neural networks

Dmitry Yarotsky, Anton Zhevnerchuk

Keywords Paper

0

0

0

0

3:07

06/12/2020

ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding

Yibo Yang, Hongyang Li, Shan You and
Fei Wang, Chen Qian, Zhouchen Lin

Keywords Paper

0

0

0

0

3:19

06/12/2021

Rethinking Neural Operations for Diverse Tasks

Nicholas Roberts, Mikhail Khodak, Tri Dao and
Liam Li, Christopher Ré, Ameet S Talwalkar

Keywords Paper

deep learning, machine learning

0

0

0

0

10:26

18/07/2021

Sparsifying Networks via Subdifferential Inclusion

Sagar Verma, Jean-Christophe Pesquet

Keywords Paper

Optimization, Convex Optimization

0

0

0

0

5:10

06/12/2020

Network Diffusions via Neural Mean-Field Dynamics

shushan He, Hongyuan Zha, Xiaojing Ye

Keywords Paper

0

0

0

0

3:21

03/05/2021

Deep Networks and the Multiple Manifold Problem

Sam Buchanan, Dar Gilboa, John Wright

Keywords Paper

low-dimensional structure, overparameterized neural networks, deep learning

0

0

0

0

5:14

26/04/2020

Empirical Studies on the Properties of Linear Regions in Deep Neural Networks

Xiao Zhang, Dongrui Wu

Keywords Paper

deep learning, linear region, optimization

0

0

0

0

4:53

06/12/2021

NTopo: Mesh-free Topology Optimization using Implicit Neural Representations

Jonas Zehnder, Yue Li, Stelian Coros, Bernhard Thomaszewski

Keywords Paper

deep learning, optimization, machine learning, self-supervised learning, representation learning

0

0

0

0

9:24

14/06/2020

Dataless Model Selection With the Deep Frame Potential

Calvin Murdock, Simon Lucey

Keywords Paper

deep learning, sparse approximation theory, deep network architectures, model selection, sparsity, mutual coherence

0

0

0

1

5:00

19/10/2020

NASE: Learning knowledge graph embedding for link prediction via neural architecture search

Xiaoyu Kou, Bingfeng Luo, Huang Hu, Yan Zhang

Keywords Paper

kg embedding, neural architecture search, knowledge graph

0

0

0

0

6:29

26/04/2020

Multiplicative Interactions and Where to Find Them

Siddhant M. Jayakumar, Wojciech M. Czarnecki, Jacob Menick and
Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, Razvan Pascanu

Keywords Paper

multiplicative interactions, hypernetworks, attention

0

0

0

0

5:34

06/12/2020

Detecting Interactions from Neural Networks via Topological Analysis

Liu Liu, Qingquan Song, Kaixiong Zhou and
Ting-Hsiang Wang, Ying Shan, Xia Hu

Keywords Paper

Algorithms -> Bandit Algorithms; Reinforcement Learning and Planning -> Reinforcement Learning; Theory -> Learning Theory, Reinforcement Learning and Planning -> Exploration

0

0

0

0

3:25

26/04/2020

Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning

Ruqi Zhang, Chunyuan Li, Jianyi Zhang and
Changyou Chen, Andrew Gordon Wilson

Keywords Paper

0

0

0

0

14:59

26/04/2020

Fast Neural Network Adaptation via Parameter Remapping and Architecture Search

Jiemin Fang, Yuzhu Sun, Kangjian Peng* and
Qian Zhang, Yuan Li, Wenyu Liu, Xinggang Wang

Keywords Paper

1

0

0

0

4:39

18/07/2021

What's in the Box? Exploring the Inner Life of Neural Networks with Robust Rules

Jonas Fischer, Anna Olah, Jilles Vreeken

Keywords Paper

Deep Learning

0

0

0

0

5:34

06/12/2021

Parametric Complexity Bounds for Approximating PDEs with Neural Networks

Tanya Marwah, Zachary Lipton, Andrej Risteski

Keywords Paper

theory, deep learning, optimization

0

0

0

0

12:32

14/06/2020

When NAS Meets Robustness: In Search of Robust Architectures Against Adversarial Attacks

Minghao Guo, Yuzhe Yang, Rui Xu and
Ziwei Liu, Dahua Lin

Keywords Paper

adversarial robustness, neural architecture search, adversarial examples, deep learning architectures, adversarial attacks

0

0

0

0

1:01

18/07/2021

Optimal Transport Kernels for Sequential and Parallel Neural Architecture Search

Vu Nguyen, Tam Le, Makoto Yamada, Michael A Osborne

Keywords Paper

Algorithms, AutoML

0

0

0

0

5:22

14/06/2020

SP-NAS: Serial-to-Parallel Backbone Search for Object Detection

Chenhan Jiang, Hang Xu, Wei Zhang and
Xiaodan Liang, Zhenguo Li

Keywords Paper

object detection, nas, autonomous driving scene

0

0

0

0

1:01

02/02/2021

Anytime Inference with Distilled Hierarchical Neural Ensembles

Adria Ruiz, Jakob Verbeek

Keywords Paper

0

0

0

0

15:09

03/05/2021

Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective

Wuyang Chen, Xinyu Gong, Zhangyang Wang

Keywords Paper

number of linear regions, neural tangent kernel, Neural Architecture Search

0

0

0

0

5:01

26/04/2020

Robust Subspace Recovery Layer for Unsupervised Anomaly Detection

Chieh-Hsin Lai, Dongmian Zou, Gilad Lerman

Keywords Paper

robust subspace recovery, unsupervised anomaly detection, outliers, latent space, autoencoder

0

0

0

0

4:56

06/12/2021

Generic Neural Architecture Search via Regression

Yuhong Li, Cong Hao, Pan Li and
Jinjun Xiong, Deming Chen

Keywords Paper

deep learning, machine learning, self-supervised learning, vision, language

0

0

0

0

14:56

06/12/2021

Estimating the Unique Information of Continuous Variables

Ari Pakman, Amin Nejatbakhsh, Dar Gilboa and
Abdullah Makkeh, Luca Mazzucato, Michael Wibral, Elad Schneidman

Keywords Paper

deep learning, optimization, generative model

0

0

0

0

12:39

06/12/2020

Collegial Ensembles

Etai Littwin, Ben Myara, Sima Sabah and
Joshua Susskind, Shuangfei Zhai, Oren Golan

Keywords Paper

0

0

0

0

3:17

06/12/2020

A Study on Encodings for Neural Architecture Search

Colin White, Willie Neiswanger, Sam Nolen, Yash Savani

Keywords Paper

Optimization -> Convex Optimization; Optimization -> Non-Convex Optimization; Optimization -> Stochastic Optimization, Algorithms -> Large Scale Learning

0

0

0

0

3:04

19/08/2021

Generative Adversarial Neural Architecture Search

Seyed Saeed Changiz Rezaei, Fred X. Han, Di Niu and
Mohammad Salameh, Keith Mills, Shuo Lian, Wei Lu, Shangling Jui

Keywords Paper

Machine Learning, Adversarial Machine Learning, Applications of Reinforcement Learning, Combinatorial Search and Optimisation

0

0

0

0

13:42

18/07/2021

Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation

Aurick Zhou, Sergey Levine

Keywords Paper

Deep Learning, Bayesian Deep Learning

0

0

0

0

5:05

06/12/2020

Adaptive Shrinkage Estimation for Streaming Graphs

Nesreen K. Ahmed, Nick Duffield

Keywords Paper

0

0

0

0

3:23

12/07/2020

Operation-Aware Soft Channel Pruning using Differentiable Masks

Minsoo Kang, Bohyung Han

Keywords Paper

Applications - Computer Vision

0

0

0

0

14:56

26/04/2020

Locally Constant Networks

Guang-He Lee, Tommi S. Jaakkola

Keywords Paper

0

0

0

0

4:44