Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

@article{He2015DelvingDI,
  title={Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification},
  author={Kaiming He and X. Zhang and Shaoqing Ren and Jian Sun},
  journal={2015 IEEE International Conference on Computer Vision (ICCV)},
  year={2015},
  pages={1026-1034},
  url={https://api.semanticscholar.org/CorpusID:13740328}
}
This work proposes a Parametric Rectified Linear Unit (PReLU) that generalizes the traditional rectified unit and derives a robust initialization method that particularly considers the rectifier nonlinearities.
[PDF] Semantic Reader
17,798 Citations
Highly Influential Citations
1,245
Background Citations
4,397
Methods Citations
8,322
Results Citations
143

Figures and Tables from this paper

17,798 Citations

A Deep Convolutional Neural Network with Selection Units for Super-Resolution

The proposed deep network with SUs, called SelNet, was top-5th ranked in NTIRE2017 Challenge, which has a much lower computation complexity compared to the top-4 entries, and experiment results show that the proposed SelNet outperforms the authors' baseline only with ReLU, and other state-of-the-art deep-learning-based SR methods.

Empirical Evaluation of Rectified Activations in Convolutional Network

The experiments suggest that incorporating a non-zero slope for negative part in rectified activation units could consistently improve the results, and are negative on the common belief that sparsity is the key of good performance in ReLU.

FReLU: Flexible Rectified Linear Units for Improving Convolutional Neural Networks

Experimental results show that FReLU achieves fast convergence and competitive performance on both plain and residual networks, and is designed to be simple and effective without exponential functions to maintain low-cost computation.

Overcoming Overfitting and Large Weight Update Problem in Linear Rectifiers: Thresholded Exponential Rectified Linear Units

Thresholded exponential rectified linear unit (TERELU) activation function that works better in alleviating in overfitting: large weight update problem, and also gives good amount of non-linearity as compared to other linear rectifiers.

Beyond ImageNet: Deep Learning in Industrial Practice

This chapter focuses on convolutional neural networks, which have since the seminal work of Krizhevsky et al. revolutionized image classification and started surpassing human performance on some benchmark data sets and can be successfully applied to other areas and problems with some local structure in the data.

Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers

This work develops a new type of transformation that is fully compatible with a variant of ReL Us -- Leaky ReLUs, and achieves validation accuracies with deep vanilla networks that are competitive with ResNets, and significantly higher than those obtained with the Edge of Chaos method.

Deep neural networks with Elastic Rectified Linear Units for object recognition

Deep Transfer Learning for Art Classification Problems

This paper shows how DCNNs, which have been fine tuned on a large artistic collection, outperform the same architectures which are pre-trained on the ImageNet dataset only, when it comes to the classification of heritage objects from a different dataset.

ResNet Sparsifier: Learning Strict Identity Mappings in Deep Residual Networks

Epsilon-ResNet is proposed that allows us to automatically discard redundant layers, which produces responses that are smaller than a threshold epsilon, with a marginal or no loss in performance.

Global-connected network with generalized ReLU activation

...

46 References

Deeply-Supervised Nets

The proposed deeply-supervised nets (DSN) method simultaneously minimizes classification error while making the learning process of hidden layers direct and transparent, and extends techniques from stochastic gradient methods to analyze the algorithm.

ImageNet classification with deep convolutional neural networks

A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.

Convolutional neural networks at constrained time cost

This paper investigates the accuracy of CNNs under constrained time cost, and presents an architecture that achieves very competitive accuracy in the ImageNet dataset, yet is 20% faster than “AlexNet” [14] (16.0% top-5 error, 10-view test).

Return of the Devil in the Details: Delving Deep into Convolutional Nets

It is shown that the data augmentation techniques commonly applied to CNN-based methods can also be applied to shallow methods, and result in an analogous performance boost, and it is identified that the dimensionality of the CNN output layer can be reduced significantly without having an adverse effect on performance.

Going deeper with convolutions

We propose a deep convolutional neural network architecture codenamed Inception that achieves the new state of the art for classification and detection in the ImageNet Large-Scale Visual Recognition…

Very Deep Convolutional Networks for Large-Scale Image Recognition

This work investigates the effect of the convolutional network depth on its accuracy in the large-scale image recognition setting using an architecture with very small convolution filters, which shows that a significant improvement on the prior-art configurations can be achieved by pushing the depth to 16-19 weight layers.

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin.

DeepFace: Closing the Gap to Human-Level Performance in Face Verification

This work revisits both the alignment step and the representation step by employing explicit 3D face modeling in order to apply a piecewise affine transformation, and derive a face representation from a nine-layer deep neural network.

Some Improvements on Deep Convolutional Neural Network Based Image Classification

This paper summarizes the entry in the Imagenet Large Scale Visual Recognition Challenge 2013, which achieved a top 5 classification error rate and achieved over a 20% relative improvement on the previous year's winner.

On rectified linear units for speech processing

This work shows that it can improve generalization and make training of deep networks faster and simpler by substituting the logistic units with rectified linear units.