Sparsity Aware Normalization for GANs

02/02/2021

Sparsity Aware Normalization for GANs

Idan Kligvasser, Tomer Michaeli

Keywords:

Abstract Paper Similar Papers

Abstract: Generative adversarial networks (GANs) are known to benefit from regularization or normalization of their critic (discriminator) network during training. In this paper, we analyze the popular spectral normalization scheme, find a significant drawback and introduce sparsity aware normalization (SAN), a new alternative approach for stabilizing GAN training. As opposed to other normalization methods, our approach explicitly accounts for the sparse nature of the feature maps in convolutional networks with ReLU activations. We illustrate the effectiveness of our method through extensive experiments with a variety of network architectures. As we show, sparsity is particularly dominant in critics used for image-to-image translation settings. In these cases our approach improves upon existing methods, in less training epochs and with smaller capacity networks, while requiring practically no computational overhead.

The video of this talk cannot be embedded. You can watch it here:

https://slideslive.com/38948065

(Link will open in new window)

0

0

0

0

Share

This is an embedded video. Talk and the respective paper are published at AAAI 2021 virtual conference. If you are one of the authors of the paper and want to manage your upload, see the question "My papertalk has been externally embedded..." in the FAQ section.

Comments

Post Comment

no comments yet

Similar Papers

05/01/2021

InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning

Kwot Sin Lee, Ngoc-Trung Tran, Ngai-Man Cheung

Keywords Paper

0

0

0

0

5:01

06/12/2021

Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data

Liming Jiang, Bo Dai, Wayne Wu, Chen Change Loy

Keywords Paper

generative model

0

0

0

0

13:51

26/04/2020

Consistency Regularization for Generative Adversarial Networks

Han Zhang, Zizhao Zhang, Augustus Odena, Honglak Lee

Keywords Paper

Generative Adversarial Networks, Consistency Regularization, GAN

0

0

0

0

6:15

06/12/2020

Differentiable Augmentation for Data-Efficient GAN Training

Shengyu Zhao, Zhijian Liu, Ji Lin and
Jun-Yan Zhu, Song Han

Keywords Paper

0

0

0

0

3:22

14/06/2020

Orthogonal Convolutional Neural Networks

Jiayun Wang, Yubei Chen, Rudrasis Chakraborty, Stella X. Yu

Keywords Paper

orthogonal convolution, orthogonality, regularization, filter redundancy, robustness, classification, retrieval, semi-supervised, gans, inpainting

0

0

0

0

1:00

14/06/2020

Towards Achieving Adversarial Robustness by Enforcing Feature Consistency Across Bit Planes

Sravanti Addepalli, Vivek B.S., Arya Baburaj and
Gaurang Sriramanan, R. Venkatesh Babu

Keywords Paper

adversarial robustness, adversarial defense, adversarial training, fast adversarial training, adversary-free training, adversarial attacks, efficient adversarial training, generalization, feature consistency, deep neural networks

0

0

0

0

1:01

02/02/2021

Adversarial Training Reduces Information and Improves Transferability

Matteo Terzi, Alessandro Achille, Marco Maggipinto, Gian Antonio Susto

Keywords Paper

0

0

0

0

19:54

06/12/2020

Improving GAN Training with Probability Ratio Clipping and Sample Reweighting

Yue Wu, Pan Zhou, Andrew Wilson and
Eric Xing, Zhiting Hu

Keywords Paper

0

0

0

0

3:22

06/12/2021

Data-Efficient Instance Generation from Instance Discrimination

Ceyuan Yang, Yujun Shen, Yinghao Xu, Bolei Zhou

Keywords Paper

machine learning, generative model

0

0

0

0

6:53

14/06/2020

Watch Your Up-Convolution: CNN Based Generative Deep Neural Networks Are Failing to Reproduce Spectral Distributions

Ricard Durall, Margret Keuper, Janis Keuper

Keywords Paper

spectral regularization, gan, deepfake, up-convolution, generative models, frequency spectrum

0

0

0

0

1:00

06/12/2020

Training Generative Adversarial Networks with Limited Data

Tero Karras, Miika Aittala, Janne Hellsten and
Samuli Laine, Jaakko Lehtinen, Timo Aila

Keywords Paper

0

0

0

0

3:24

03/05/2021

Training GANs with Stronger Augmentations via Contrastive Discriminator

Jongheon Jeong, Jinwoo Shin

Keywords Paper

visual representation learning, contrastive learning, unsupervised learning, data augmentation, generative adversarial networks

0

0

0

0

5:48

03/05/2021

Group Equivariant Generative Adversarial Networks

Neel Dey, Antong Chen, Soheil Ghafurian

Keywords Paper

Geometric Deep Learning, Generative Adversarial Networks, Group Equivariance

0

0

0

0

5:06

06/12/2021

ReSSL: Relational Self-Supervised Learning with Weak Augmentation

Mingkai Zheng, Shan You, Fei Wang and
Chen Qian, Changshui Zhang, Xiaogang Wang, Chang Xu

Keywords Paper

self-supervised learning, contrastive learning

0

0

0

0

6:35

06/12/2021

Why Spectral Normalization Stabilizes GANs: Analysis and Improvements

Zinan Lin, Vyas Sekar, Giulia Fanti

Keywords Paper

generative model

0

0

0

0

14:40

26/04/2020

Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee

Wei Hu, Zhiyuan Li, Dingli Yu

Keywords Paper

deep learning theory, regularization, noisy labels

0

0

0

0

5:13

26/04/2020

Stochastic Conditional Generative Networks with Basis Decomposition

Ze Wang, Xiuyuan Cheng, Guillermo Sapiro, Qiang Qiu

Keywords Paper

0

0

0

0

4:00

14/06/2020

ViewAL: Active Learning With Viewpoint Entropy for Semantic Segmentation

Yawar Siddiqui, Julien Valentin, Matthias Nießner

Keywords Paper

active learning, semantic segmentation, deep learning, view consistency

0

0

0

0

1:01

14/06/2020

DUNIT: Detection-Based Unsupervised Image-to-Image Translation

Deblina Bhattacharjee, Seungryong Kim, Guillaume Vizier, Mathieu Salzmann

Keywords Paper

style transfer, object detection, unsupervised image to image translation, domain adaptation

0

0

0

0

1:01

14/06/2020

MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks

Animesh Karnewar, Oliver Wang

Keywords Paper

msg-gan, multi-scale gradients, generative adversarial network, high-res image synthesis, gan training stability

0

0

0

0

1:00

03/05/2021

You Only Need Adversarial Supervision for Semantic Image Synthesis

Edgar Schoenfeld, Vadim Sushko, Dan Zhang and
Juergen Gall, Bernt Schiele, Anna Khoreva

Keywords Paper

GANs, Semantic Image Synthesis, Image Generation, Deep Learning

0

0

0

0

5:11

06/12/2020

Top-KAST: Top-K Always Sparse Training

Sid Jayakumar, Razvan Pascanu, Jack Rae and
Simon Osindero, Erich Elsen

Keywords Paper

0

0

0

0

3:18

12/07/2020

Implicit competitive regularization in GANs

Florian Schaefer, Hongkai Zheng, Anima Anandkumar

Keywords Paper

Deep Learning - Theory

0

0

0

0

15:07

12/07/2020

Concise Explanations of Neural Networks using Adversarial Training

Prasad Chalasani, Jiefeng Chen, Amrita Roy Chowdhury and
Xi Wu, Somesh Jha

Keywords Paper

Accountability, Transparency and Interpretability

0

0

0

0

14:16

19/08/2021

Enhance Image as You Like with Unpaired Learning

Xiaopeng Sun, Muxingzi Li, Tianyu He, Lubin Fan

Keywords Paper

Computer Vision, 2D and 3D Computer Vision, Applications of Unsupervised Learning

0

0

0

0

11:20

06/12/2021

ResNEsts and DenseNEsts: Block-based DNN Models with Improved Representation Guarantees

Kuan-Lin Chen, Ching-Hua Lee, Harinath Garudadri, Bhaskar D Rao

Keywords Paper

optimization, vision

0

0

0

0

13:27

14/06/2020

A Characteristic Function Approach to Deep Implicit Generative Modeling

Abdul Fatir Ansari, Jonathan Scarlett, Harold Soh

Keywords Paper

generative adversarial networks, generative models, probability metrics, characteristic functions, unsupervised learning

0

0

0

0

4:56

06/12/2020

Bayesian Attention Modules

Xinjie Fan, Shujian Zhang, Bo Chen, Mingyuan Zhou

Keywords Paper

0

0

0

0

3:32

06/12/2021

Adversarial Robustness without Adversarial Training: A Teacher-Guided Curriculum Learning Approach

Anindya Sarkar, Anirban Sarkar, Sowrya Gali, Vineeth N Balasubramanian

Keywords Paper

robustness, adversarial robustness and security

0

0

0

0

15:21

14/06/2020

Deep Residual Flow for Out of Distribution Detection

Ev Zisselman, Aviv Tamar

Keywords Paper

neural-networks, out-of-distribution detection, flow models, neural generative models, machine learning architectures.

0

0

0

0

0:59

19/08/2021

GSPL: A Succinct Kernel Model for Group-Sparse Projections Learning of Multiview Data

Danyang Wu, Jin Xu, Xia Dong and
Meng Liao, Rong Wang, Feiping Nie, Xuelong Li

Keywords Paper

Machine Learning, Learning Sparse Models, Multi-instance; Multi-label; Multi-view learning, Unsupervised Learning

0

0

0

0

11:48

03/05/2021

Reweighting Augmented Samples by Minimizing the Maximal Expected Loss

Mingyang Yi, LU HOU, Lifeng Shang and
Xin Jiang, Qun Liu, Zhi-Ming Ma

Keywords Paper

sample reweighting, data augmentation

0

0

0

0

4:58

06/12/2021

A Little Robustness Goes a Long Way: Leveraging Robust Features for Targeted Transfer Attacks

Jacob Springer, Melanie Mitchell, Garrett Kenyon

Keywords Paper

deep learning, optimization, robustness, adversarial robustness and security, transformers

0

0

0

0

9:29

02/02/2021

Discriminative Region Suppression for Weakly-Supervised Semantic Segmentation

Beomyoung Kim, Sangeun Han, Junmo Kim

Keywords Paper

0

0

0

0

14:27

06/12/2021

Channel Permutations for N:M Sparsity

Jeff Pool, Chong Yu

Keywords Paper

optimization

0

0

0

0

12:41

14/06/2020

HRank: Filter Pruning Using High-Rank Feature Map

Mingbao Lin, Rongrong Ji, Yan Wang and
Yichen Zhang, Baochang Zhang, Yonghong Tian, Ling Shao

Keywords Paper

network pruning, neural network compression and acceleration, high-rank feature map, efficient deep learning computing

0

0

0

0

4:57

26/04/2020

Mixup Inference: Better Exploiting Mixup to Defend Adversarial Attacks

Tianyu Pang, Kun Xu, Jun Zhu

Keywords Paper

Trustworthy Machine Learning, Adversarial Robustness, Inference Principle, Mixup

0

0

0

0

4:59

14/06/2020

Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement

Chunle Guo, Chongyi Li, Jichang Guo and
Chen Change Loy, Junhui Hou, Sam Kwong, Runmin Cong

Keywords Paper

low-light image, deep learning, zero-reference learning, curve estimation, non-reference loss, computational photography, image enhancement, image restoration, image processing, computer vision

0

0

0

0

0:58

05/01/2021

Exploiting Spatial Relation for Reducing Distortion in Style Transfer

Jia-Ren Chang, Yong-Sheng Chen

Keywords Paper

0

0

0

0

4:54

06/12/2021

Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training

Minguk Kang, Woohyeon Shim, Minsu Cho, Jaesik Park

Keywords Paper

generative model

0

0

0

0

9:03