Stable and Symmetric Filter Convolutional Neural Network

Raymond Yeh   Mark Hasegawa-Johnson   Minh N. Do  

University of Illinois Urbana-Champaign  

Abstract

First we present a proof that convolutional neural networks (CNN) with max-norm regularization, max-pooling, and Relu non-linearity are stable to additive noise. Second, we explore the use of symmetric and antisymmetric filters in a baseline CNN model on digit classification, which enjoys the stability to additive noise. Experimental results indicate that the symmetric CNN outperforms the baseline model for nearly all training sizes and matches the state-of-the-art deep-net in the cases of limited training examples.

Materials

[Paper]

Results

For a transformation, $\Phi$, to be stable to additive noise $x'(u) = x(u) + \epsilon(u)$, it needs a Lipschitz continuity condition as defined in [bruna2013invariant], $$||\Phi x-\Phi x'||_2 \leq C \cdot ||x-x'||_2$$ for a constant $C > 0$, and for all $x$ and $x'$. $\Phi x$ denotes the transformed feature. We have shown that a CNN with the following operations, 1) convolution with max-norm regularization, 2) element-wise Relu non-linearity, and 3) max-pooling satisfies the Lipschitz continuity condition. Next, we explored the use of symmetric and antisymmetric filters in a baseline CNN model on digit classification, which enjoys the stability to additive noise and linear-phase condition. Experimental results indicate that the symmetric CNN outperforms the baseline model for nearly all training sizes and matches the state-of-the-art deep-net in the cases of limited training examples on the MNIST digit recognition task.

Citation

@inproceedings{
    yeh2016stable,
    title={Stable and symmetric filter convolutional neural network},
    author={Yeh, Raymond and Hasegawa-Johnson, Mark and Do, Mink N},
    booktitle={2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
    pages={2652--2656},
    year={2016},
    organization={IEEE}
}