Initializing ReLU networks in an expressive subspace of weights
Dayal Singh and G J Sreejith, Under reviewWe analyze signal propagation in ReLU networks with anti-correlated weights and demonstrate that it has an order-to-chaos phase transition, unlike the uncorrelated case. Furthermore, we demonstrate that ReLU networks initialized at this criticality train faster.