SNN hardware accelerators // neuron design is based on superconductor electronics fabric
High-performance and ultra-energy-efficient neural network accelerator architectures
A novel high-fan-in differential superconductor neuron structure designed for ultra-high-performance Spiking Neural Network (SNN) accelerators is presented. Utilizing a high-fan-in neuron structure allows us to design SNN accelerators with more synaptic connections, enhancing the overall network capabilities. The proposed neuron design is based on superconductor electronics fabric, incorporating multiple superconducting loops, each with two Josephson Junctions. This arrangement enables each input data branch to have positive and negative inductive coupling, supporting excitatory and inhibitory synaptic data. Compatibility with synaptic devices and thresholding operation is achieved using a single flux quantum (SFQ) pulse-based logic style. The neuron design, along with ternary synaptic connections, forms the foundation for a superconductor based SNN inference. To demonstrate the capabilities of our design, we train the SNN using snnTorch, augmenting the PyTorch framework. After pruning, the demonstrated SNN inference achieves an impressive 96.1% accuracy on MNIST images. Notably, the network exhibits a remarkable throughput of 8.92 GHz while consuming only 1.5 nJ per inference, including the energy consumption associated with cooling to 4K. These results underscore the potential of superconductor electronics in developing high-performance and ultra-energy-efficient neural network accelerator architectures.
Here is a summary of the key points from the paper:
- The paper presents a novel superconductor-based leaky integrate-and-fire (LIF) neuron design for ultra-high performance spiking neural network (SNN) hardware.
- The neuron uses multiple superconducting loops with Josephson junctions (JJs) to achieve high fan-in capabilities. Each input branch can have positive or negative inductive coupling to support excitatory and inhibitory synaptic inputs.
- The design is compatible with single flux quantum (SFQ) pulse logic and can achieve thresholding behavior like biological neurons. The resistors in each branch establish leaky behavior.
- The neuron design was used to construct a 3-layer fully connected SNN for MNIST image classification. With pruning, it achieved 96.1% accuracy and 8.92 GHz throughput with only 1.5 nJ per inference energy consumption.
- Compared to CMOS SNN implementations, the superconductor design shows much higher throughput in the GHz range rather than MHz. It also has ultra-low energy consumption in the nanojoule range.
- The high fan-in capabilities and compatibility with SFQ pulse logic make this neuron design suitable as a building block for energy-efficient, ultra-fast superconductor neuromorphic hardware.
In this work, we present a hardware implementation of a high-fan-in superconductor neuron structure. The analog simulation of the circuit is demonstrated using JoSIM, showcasing the functionality and behavior of the neuron. We further showcase the capabilities of the neuron in SNN inference using the popular MNIST dataset. The fully connected network achieves an impressive 97.07% accuracy. To improve power efficiency and reduce static power consumption, we utilize pruning techniques on the network, resulting in a slightly reduced accuracy of 96.1%. Despite this accuracy reduction, the pruned network retains the ability to process data at the order of GHz throughput. The inference network is trained using snnTorch, which leverages gradient-based optimization techniques for continuous approximations of the gradients during training. Significantly, the neuron operates asynchronously, allowing the overall flow to proceed without the need for a clock signal, except for the input buffers on the input layer, which are used to synchronize the data flow.
Spiking Neural Networks (SNNs):
- SNNs are a type of artificial neural network that more closely mimics biological neural networks. Instead of continuous values, SNNs communicate using discrete spikes similar to biological action potentials.
- Neurons in an SNN integrate incoming spikes over time and fire their own spikes when their membrane potential reaches a threshold. The timing and number of spikes encodes information.
- SNNs can be trained using methods like spike timing dependent plasticity to modify synaptic weights based on spike timing. Supervised learning methods like backpropagation have also been adapted for SNNs.
- Compared to traditional artificial neural networks, SNNs are considered more biologically plausible and energy efficient due to their sparse, spiking communication. However, training SNNs can be more difficult.
- Neuromorphic hardware like IBM’s TrueNorth chip have been developed to efficiently simulate SNNs in custom analog circuitry. SNNs are also well suited for implementation using superconducting electronics.
- Research on SNNs is very active for applications like computer vision, speech recognition, and robotics. SNNs have achieved state-of-the-art results on datasets like MNIST while being far more energy efficient.
- Key advantages of SNNs include biological plausibility, sparse computation and communication, and efficient neuromorphic implementation. Challenges include developing effective training methods and bridging the accuracy gap with traditional deep networks.
