Introduction to Spiking Neural Networks

While the project is equal part reinforcement learning and spiking neural networks, reinforcement learning is a popular field and has been extensively covered by researchers worldwide (Ivanov & D’yakonov, 2019),(Li, 2018). Hence, I have chosen instead to review the literature around spiking neural networks.

The Generations of Neural Networks

Neural network models can be classified into three generations, according to their computational units: perceptrons, non-linear units, and spiking neurons (“Wolfgang Maass”, 1997).

Perceptrons can be composed to produce a variety of models, including Boltzmann machines and Hopfield networks. Non-linear units are currently the most widely used computational unit, responsible for the explosion of progress in machine learning research, in particular, the success of deep learning. These units traditionally apply differentiable, non-linear activation functions such across a weighted sum of input values.

There are two reasons second-generation computational units have seen so much success. First, the computational power of these units is greater than that of first-generation neural networks. Networks built with second-generation computational units with one hidden layer are universal approximators for any continuous function with a compact domain and range (“Cybenko, 1989). Second, networks built with these units are trainable with well-researched gradient-based methods, such as backpropagation.

The third generation of neural networks use computational units called spiking neurons. Much like our biological neurons, spiking neurons are connected to each other at synapses, receiving incoming signals at the dendrites and sending spikes to other neurons via the axon. Each computational unit stores some state: in particular, it stores its membrane potential at any point in time. Rather than fire at each propagation cycle, these computational units fire only when their individual membrane potentials crosses its firing threshold. A simple spiking neuron model is given in .

From this section onwards, we shall term second-generation neural networks Artificial Neural Networks (ANNs), and third-generation neural networks Spiking Neural Networks (SNNs).

A Spiking Neuron Model

In spiking neural networks, neurons exchange information via spikes, and the information received depends on:

Firing frequencies
The relative timing of pre and post-synaptic spikes, and neuronal firing patterns
Identity of synapses used
Which neurons are connected, whether their synapses are inhibitory or excitatory, and synaptic strength

Each neuron has a corresponding model that encapsulates its state: the current membrane potential. As with the mammalian brain, incoming spikes increase the value of membrane potential. The membrane potential eventually decays to resting potential in the absence of spikes. These dynamics are often captured via first-order differential equations. Here we define the Spike Response Model (SRM), a simple but widely-used model describing the momentary value of a neuron \(i\).

We define for presynaptic neuron \(j\), \(\epsilon_{ij}(t) = u_{i}(t) - u_{\text{rest}}\). For a few input spikes, the membrane potential responds roughly linearly to the input spikes:

\begin{equation} u_i{t} = \sum_{j}\sum_{f} \epsilon_{ij}(t - t_j^{(f)}) + u_{\text{rest}} \end{equation}

SRM describes the membrane potential of neuron \(i\) as:

\begin{equation} u_i{t} = \eta (t - \hat{t_i}) + \sum_{j}\sum_{f} \epsilon_{ij}(t - t_j^{(f)}) + u_{\text{rest}} \end{equation}

where \(\hat{t_i}\) is the last firing time of neuron \(i\).

We refer to moment when a given neuron emits an action potential as the firing time of that neuron. We denote the firing times of neuron \(i\) by \(t_i^{(f)}\) where \(f = 1,2,\dots\) is the label of the spike. Then we formally denote the spike train of a neuron \(i\) as the sequence of firing times:

\begin{equation} S_i(t) = \sum_{f} \delta\left( t - t_i^{(f)} \right) \end{equation}

where \(\delta(x)\) is the Dirac-delta function with \(\delta(x) = 0\) for \(x \ne 0\) and \(\int_{-\infty}^{\infty} \delta(x)dx = 1\). Spikes are thus reduced to points in time.

dendrites
input device
soma
central processing unit (non-linear processing step). If the total input exceeds a certain threshold, an output signal is generated
axon
output device, delivering signal to other neurons
synapse
junction between two neurons
post/presynaptic cells
If a neuron is sending a signal across a synapse, the sending neuron is the presynaptic cell, and the receiving neuron is the postsynaptic cell
action potentials/spikes
short electrical pulses, typically of amplitude about 100mV and a duration of 1-2ms
spike train
a chain of action potentials (sequence of stereotyped events) that occur at intervals. Since all spikes of a given neuron look the same, the form of the spike does not matter: the number and timing of the spikes encode the information.
absolute refractory period
minimal distance between two spikes. Spike are well separated, and it is impossible to excite a second spike within this refractory period.
relative refractory period
follows the absolute refractory period – a period where it is difficult to excite an action potential

We define for presynaptic neuron \(j\), \(\epsilon_{ij}(t) = u_{i}(t) - u_{rest}\). For a few input spikes, the membrane potential responds roughly linearly to the input spikes:

\begin{equation} u_i{t} = \sum_{j}\sum_{f} \epsilon_{ij}(t - t_j^{(f)}) + u_{rest} \end{equation}

If \(u_i(t)\) reaches threshold \(\vartheta\) from below, neuron \(i\) fires a spike.

From the above, we can define the Spike Response Model describing the momentary value of the membrane potential of neuron \(i\):

\begin{equation} u_i{t} = \eta (t - \hat{t_i}) + \sum_{j}\sum_{f} \epsilon_{ij}(t - t_j^{(f)}) + u_{rest} \end{equation}

where \(\hat{t_i}\) is the last firing time of neuron \(i\).

We refer to moment when a given neuron emits an action potential as the firing time of that neuron. We denote the firing times of neuron \(i\) by \(t_i^{(f)}\) where \(f = 1,2,\dots\) is the label of the spike. Then we formally denote the spike train of a neuron \(i\) as the sequence of firing times:

\begin{equation} S_i(t) = \sum_{f} \delta\left( t - t_i^{(f)} \right) \end{equation}

where \(\delta(x)\) is the Dirac \(\delta\) function with \(\delta(x) = 0\) for \(x \ne 0\) and \(\int_{-\infty}^{\infty} \delta(x)dx = 1\). Spikes are thus reduced to points in time.

SRM only takes into account the most recent spike, and cannot capture adaptation.

Neuronal Coding

How do spike trains encode information? At present, a definite answer to this question is not known.

Spike Codes

These are coding strategies based on spike timing.

Spikes or Rates?

A code based on time-to-first-spike is consistent with a rate code: if the mean firing rate of a neuron is high, then the time to first spike is expected to occur early. Stimulus reconstruction with a linear kernel can be seen as a special instance of a rate code. It is difficult to draw a clear borderline between pulse and rate codes. The key consideration in using any code is the ability for the system to react quickly to changes in the input. If pulse coding is relevant, information processing in the brain must be based on spiking neuron models. For stationary input, spiking neuron models can be reduced to rate models, but in other cases, this reduction is not possible.

Motivating Spiking Neural Networks

Since second-generation neural networks have excellent performance, why bother with spiking neural networks? In this section, we motivate spiking neural networks from various perspectives.

Training Spiking Neural Networks

As explained in , it is desirable to train spiking neural networks to perform arbitrary tasks, utilizing power-efficient neuromorphic chips that break the Von Neumann bottleneck. We classify the training strategies by their usage of gradients, and discuss certain optimization techniques.

A Spiking Neuron Model

In spiking neural networks, neurons exchange information via spikes, and the information received depends on:

Firing frequencies
The relative timing of pre and post-synaptic spikes, and neuronal firing patterns
Identity of synapses used
Which neurons are connected, whether their synapses are inhibitory or excitatory, and synaptic strength

Each neuron has a corresponding model that encapsulates its state: the current membrane potential. As with the mammalian brain, incoming spikes increase the value of membrane potential. The membrane potential eventually decays to resting potential in the absence of spikes. These dynamics are often captured via first-order differential equations. Here we define the Spike Response Model (SRM), a simple but widely-used model describing the momentary value of a neuron \(i\).

We define for presynaptic neuron \(j\), \(\epsilon_{ij}(t) = u_{i}(t) - u_{\text{rest}}\). For a few input spikes, the membrane potential responds roughly linearly to the input spikes:

\begin{equation} u_i{t} = \sum_{j}\sum_{f} \epsilon_{ij}(t - t_j^{(f)}) + u_{\text{rest}} \end{equation}

SRM describes the membrane potential of neuron \(i\) as:

\begin{equation} u_i{t} = \eta (t - \hat{t_i}) + \sum_{j}\sum_{f} \epsilon_{ij}(t - t_j^{(f)}) + u_{\text{rest}} \end{equation}

where \(\hat{t_i}\) is the last firing time of neuron \(i\).

We refer to moment when a given neuron emits an action potential as the firing time of that neuron. We denote the firing times of neuron \(i\) by \(t_i^{(f)}\) where \(f = 1,2,\dots\) is the label of the spike. Then we formally denote the spike train of a neuron \(i\) as the sequence of firing times:

\begin{equation} S_i(t) = \sum_{f} \delta\left( t - t_i^{(f)} \right) \end{equation}

where \(\delta(x)\) is the Dirac-delta function with \(\delta(x) = 0\) for \(x \ne 0\) and \(\int_{-\infty}^{\infty} \delta(x)dx = 1\). Spikes are thus reduced to points in time.

Motivating Spiking Neural Networks

Since second-generation neural networks have excellent performance, why bother with spiking neural networks? In this section, we motivate spiking neural networks from various perspectives.

Training Spiking Neural Networks

As explained in , it is desirable to train spiking neural networks to perform arbitrary tasks, utilizing power-efficient neuromorphic chips that break the Von Neumann bottleneck. We classify the training strategies by their usage of gradients, and discuss certain optimization techniques.

Probabilistic SNNs

A probabilistic model defines the outputs of all spiking neurons as jointly distributed binary random processes. The joint distribution is differentiable in the synaptic weights, and principled learning criteria from statistics and information theory such as likelihood and mutual information apply. The maximization of such criteria do not require the implementation of the backpropagation mechanism, and often recover as special cases known biologically plausible algorithms.

Graphical Representation

A SNN consists of a network of \(N\) spiking neurons. At any time \(t = 0,1,2, \dots\) each neouron \(i\) outputs a binary signal \(s_{i,t} = \{0,1\}\), with value \(s_{i,t} = 1\) corresponding to a spike emitted at time \(t\). We collect in vector \(s_{t} = \left( s_{i,t}: i \in V \right)\) the binary signals emitted by all neurons at time \(t\), where \(V\) is the set of all neurons. Each neuron \(i \in V\) receives the signals emitted by a subset \(P_i\) of neurons through directed links, known as synapses. Neurons in a set \(P_i\) are referred to as pre-synaptic for post-synaptic neuron \(i\).

The internal, analog state of each spiking neuron \(i \in V\) at time \(t\) is defined by its membrane potential \(u_{i,t}\).

Long short-term memory and learning-to-learn in networks of spiking neurons (Bellec {\it et al.}, 2018)

Key contribution: Inclusion of adapting neurons into recurrent SNN models (RSNNs) increases computing and learning capability. By using a learning algorithm that combines BPTT with a rewiring algorithm that optimizes the network architecture, performance comes close to LSTM ANNs.

Model composition: LSNNs consist of a populaction \(R\) of integrate-and-fire (LIF) neurons (excitatory and inhibitory), and a second population \(A\) of LIF excitatory neurons whose excitability is temporarily reduced through preceding firing activity. \(R\) and \(A\) receive spike trains from a population \(X\) of external input neurons. Results of computations are read out by a population \(Y\) of external linear readout neurons.

BPTT is done by replacing the non-continuous membrane potential with a pseudo derivative that smoothly increases from 0 to 1.

Learning to Learn LSNNs

LSTM networks are especially suited for L2L since they can accommodate two levelsof learning and representation of learned insight: Synaptic connections and weights can encode,on a higher level, a learning algorithm and prior knowledge on a large time-scale. The short-termmemory of an LSTM network can accumulate, on a lower level of learning, knowledge during thecurrent learning task

Gradient Descent for Spiking Neural Networks (Huh & Sejnowski, 2017)

key idea: Replacing the non-differentiable model for membrane potential:

\begin{equation} \tau \dot{s} = -s + \sum_{k} \delta (t - t_k) \end{equation}

with

\begin{equation} \tau \dot{s} = -s + g \dot{v} \end{equation}

for some gate function \(g\), and \(\dot{v}\) is the time derivative of the pre-synaptic membrane voltage.

Exact gradient calculations can be done with BPTT, or real-time recurrent learning. The resultant gradients are similar to reward-modulated spike-time dependent plasticity.

TODO Surrogate Gradient Learning in Spiking Neural Networks (Neftci {\it et al.}, 2019)

TODO Theories of Error Back-Propagation in the Brain (James Whittington & Rafal Bogacz, 2019)

Temporal coding in spiking neural networks with alpha synaptic function

STDP

STDP is a biologically inspired long-term plasticity model, in which each synapse is given a weight \(0 \le w \le w_{maxx}\) , characterizing its strength, and its change depends on the exact moments \(t_{pre}\) of presynaptic spikes and \(t_{post}\) of postsynaptic spikes:

\begin{equation} \Delta w=\left\{\begin{array}{l}{-\alpha \lambda \cdot \exp \left(-\frac{t_{\mathrm{pre}}-t_{\mathrm{post}}}{\tau_{-}}\right), \text {if } t_{\mathrm{pre}}-t_{\mathrm{post}}>0} \ {\lambda \cdot \exp \left(-\frac{t_{\mathrm{post}}-t_{\mathrm{pre}}}{\tau_{+}}\right), \text {if } t_{\mathrm{pre}}-t_{\mathrm{post}}<0}\end{array}\right. \end{equation}

This additive STDP rule requires also an additional constraint that explicitly prevents the weight from falling below 0 or exceeding the maximum value of 1.

(Alexander Sboev {\it et al.}, 2018)

Loihi

Learning must follow the sum-of-products form:

\begin{equation} Z(t) = Z(t-1) + \sum_m S_m \prod_n F_n \end{equation}

where \(Z\) is the synaptic state variable defined for the source destination neuron pair being updated, and \(F-N\) may be a synaptic state variable, a pre-synaptic trace or a post-synaptic trace defined for the neuron pair.

Generating Spike Trains

Poisson Model (Heeger, 2000)

Independent spike hypothesis: the generation of each spike is independent of all other spikes. If the underlying instantaneous firing rate \(r\) is constant over time, it is a homogeneous Poisson process.

We can write:

\begin{equation} P(\textrm{1 spike during } \delta t) \approx r \delta t \end{equation}

We divide time into short, discrete intervals \(\delta t\). Then, we generate a sequence of random numbers \(x[i]\) uniformly between 0 and 1. For each interval, if \(x[i] \le r \delta t\), generate a spike.

Bibliography

Ivanov, S., & D’yakonov, A., Modern Deep Reinforcement Learning Algorithms, CoRR, (), (2019).

Li, Y., Deep Reinforcement Learning, CoRR, (), (2018).

Maass, W., Networks of spiking neurons: the third generation of neural network models, Neural Networks, 10(9), 1659–1671 (1997). http://dx.doi.org/https://doi.org/10.1016/S0893-6080(97)00011-7

Cybenko, G., Approximation by superpositions of a sigmoidal function, Mathematics of Control, Signals and Systems, 2(4), 303–314 (1989). http://dx.doi.org/10.1007/BF02551274

Pfeiffer, M., & Pfeil, T., Deep learning with spiking neurons: opportunities and challenges, Frontiers in neuroscience, 12(), (2018).

Robert G\“utig, To spike, or when to spike?, Current Opinion in Neurobiology, 25(nil), 134–139 (2014). http://dx.doi.org/10.1016/j.conb.2014.01.004

Stemmler, M., A single spike suffices: the simplest form of stochastic resonance in model neurons, Network: Computation in Neural Systems, 7(4), 687–716 (1996). http://dx.doi.org/10.1088/0954-898x_7_4_005

Comsa, I. M., Potempa, K., Versari, L., Fischbacher, T., Gesmundo, A., & Alakuijala, J., Temporal coding in spiking neural networks with alpha synaptic function, CoRR, (), (2019).

Tavanaei, A., Ghodrati, M., Kheradpisheh, S. R., Masquelier, T., & Maida, A., Deep learning in spiking neural networks, Neural Networks, 111(), 47–63 (2019). http://dx.doi.org/https://doi.org/10.1016/j.neunet.2018.12.002

Merolla, P. A., Arthur, J. V., Alvarez-Icaza, R., Cassidy, A. S., Sawada, J., Akopyan, F., Jackson, B. L., …, A million spiking-neuron integrated circuit with a scalable communication network and interface, Science, 345(6197), 668–673 (2014). http://dx.doi.org/10.1126/science.1254642

Davies, M., Srinivasa, N., Lin, T., Chinya, G., Cao, Y., Choday, S. H., Dimou, G., …, Loihi: a neuromorphic manycore processor with on-chip learning, IEEE Micro, 38(1), 82–99 (2018).

Tang, G., Shah, A., & Michmizos, K. P., Spiking neural network on neuromorphic hardware for energy-efficient unidimensional slam, CoRR, (), (2019).

Severa, W., Parekh, O., Carlson, K. D., James, C. D., & Aimone, J. B., Spiking network algorithms for scientific computing, 2016 IEEE International Conference on Rebooting Computing (ICRC), (), 1–8 (2016).

Sboev, A., Vlasov, D., Rybka, R., & Serenko, A., Spiking neural network reinforcement learning method based on temporal coding and stdp, Procedia Computer Science, 145(nil), 458–463 (2018). http://dx.doi.org/10.1016/j.procs.2018.11.107

Hazan, H., Saunders, D. J., Khan, H., Patel, D., Sanghavi, D. T., Siegelmann, H. T., & Kozma, R., Bindsnet: a machine learning-oriented spiking neural networks library in python, Frontiers in Neuroinformatics, 12(), 89 (2018). http://dx.doi.org/10.3389/fninf.2018.00089

Bing, Z., Baumann, I., Jiang, Z., Huang, K., Cai, C., & Knoll, A., Supervised learning in snn via reward-modulated spike-timing-dependent plasticity for a target reaching vehicle, Frontiers in Neurorobotics, 13(), 18 (2019). http://dx.doi.org/10.3389/fnbot.2019.00018

Lee, C., Panda, P., Srinivasan, G., & Roy, K., Training deep spiking convolutional neural networks with stdp-based unsupervised pre-training followed by supervised fine-tuning, Frontiers in Neuroscience, 12(), 435 (2018). http://dx.doi.org/10.3389/fnins.2018.00435

Urbanczik, R., & Senn, W., A gradient learning rule for the tempotron, Neural Computation, 21(2), 340–352 (2009). http://dx.doi.org/10.1162/neco.2008.09-07-605

Lee, J., Delbruck, T., & Pfeiffer, M., Training Deep Spiking Neural Networks Using Backpropagation, Frontiers in Neuroscience, 10(), (2016). http://dx.doi.org/10.3389/fnins.2016.00508

Bohte, S., Kok, J., & Poutré, J., Spikeprop: backpropagation for networks of spiking neurons., In , (pp. 419–424) (2000). : .

Huh, D., & Sejnowski, T. J., Gradient descent for spiking neural networks, In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 31 (pp. 1433–1443) (2018). : Curran Associates, Inc.

Shrestha, S. B., & Orchard, G., Slayer: spike layer error reassignment in time, In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, & R. Garnett (Eds.), Advances in Neural Information Processing Systems 31 (pp. 1412–1421) (2018). : Curran Associates, Inc.

Rueckauer, B., Lungu, I., Hu, Y., & Pfeiffer, M., Theory and tools for the conversion of analog to spiking convolutional neural networks, CoRR, (), (2016).

Scellier, B., & Bengio, Y., Equilibrium propagation: bridging the gap between energy-based models and backpropagation, Frontiers in Computational Neuroscience, 11(), 24 (2017). http://dx.doi.org/10.3389/fncom.2017.00024

O’Connor, P., Gavves, E., & Welling, M., Training a spiking neural network with equilibrium propagation, In K. Chaudhuri, & M. Sugiyama, Proceedings of Machine Learning Research (pp. 1516–1523) (2019). : PMLR.

Bellec, G., Salaj, D., Subramoney, A., Legenstein, R., & Maass, W., Long short-term memory and learning-to-learn in networks of spiking neurons, CoRR, (), (2018).

Huh, D., & Sejnowski, T. J., Gradient descent for spiking neural networks, CoRR, (), (2017).

Neftci, E. O., Mostafa, H., & Zenke, F., Surrogate gradient learning in spiking neural networks, CoRR, (), (2019).

Whittington, J. C., & Bogacz, R., Theories of error back-propagation in the brain, Trends in Cognitive Sciences, 23(3), 235–250 (2019). http://dx.doi.org/10.1016/j.tics.2018.12.005

Heeger, D., Poisson model of spike generation, Handout, University of Standford, 5(), 1–13 (2000).