NCA Group Meeting with Tony Yang

Thursday Jul 7, 2022, 1:15 PM → 2:30 PM America/New_York

4th Floor Classroom/4-Simons Foundation (160 5th Avenue)

Speaker: Tony Yang, doctoral candidate, Princeton University, and Summer Research Associate in the NCA Group, CCN


Title: A Generalized Framework for Unsupervised Feature Learning with Hebbian and Anti-Hebbian Plasticity

Abstract: Neural networks with Hebbian excitation and anti-Hebbian inhibition form an interesting class of biologically plausible unsupervised learning algorithms. Such networks can be interpreted as online gradient descent-ascent algorithms for solving min-max problems that are dual to unsupervised learning principles formulated with no explicit reference to neural networks. In this talk, I will introduce our generalized formulation, correlation game, which contains the similarity matching principles, independent component analysis, and canonical correlation analysis (CCA) as special cases.

In this framework, the feedforward synapses amplify correlations between presynaptic and postsynaptic neural activity, while lateral synapses diminish correlations among neurons. The synaptic weights are Legendre dual variables of input-output and output-output correlations. With this framework, we show the strong duality when the lateral connection matrix is positive definite, a condition that prohibits multistability of neural activity dynamics. Furthermore, the neural network algorithm typically converges when inhibitory plasticity is faster than excitatory plasticity, which can also be intuitively understood using the min-max problem's structure.

I will also introduce a PyTorch-based package that can automatically derive bio-plausible neural network algorithms from objective functions and perform online or offline learning with GPU acceleration. We obtained a neural network algorithm with disynaptic inhibition and synaptic competition from this generalized framework, and applied it to vision and language domains. Experiments show that our networks can learn meaningful image features with controllable sparsity and are comparable to popular generative models such as latent Dirichlet allocation (LDA) on topic models.