The amount of time, effort, and money needed to train ever-more-complex neural network models is soaring as researchers push the limits of machine learning. Analog deep learning, a new branch of artificial intelligence, promises faster computation with less energy consumption.
Similar to how transistors are the essential components of digital computers, programmable resistors are the fundamental building blocks of analog deep learning. Researchers have developed a network of analog artificial “neurons” and “synapses” that can do calculations similarly to a digital neural network by repeatedly repeating arrays of programmable resistors in intricate layers.
Then, using difficult AI tasks like image recognition and natural language processing, this network may be trained. The goal of a diverse MIT research team was to increase the speed of a particular kind of artificial analog synapse that they had previously created.
They used a useful inorganic substance in the manufacturing process to give their devices a speed boost of a million times over earlier iterations, which is roughly a million times faster than the synapses in the human brain.
This inorganic component also contributes to the resistor’s exceptional energy efficiency. The new material is compatible with silicon production methods, in contrast to materials employed in the earlier iteration of their device.
This modification has made it possible to fabricate nanometer-scale devices and may open the door to their incorporation into commercial computing hardware for deep-learning applications.
“With that key insight, and the very powerful nanofabrication techniques we have at MIT.nano, we have been able to put these pieces together and demonstrate that these devices are intrinsically very fast and operate with reasonable voltages,” says senior author Jesús A. del Alamo, the Donner Professor in MIT’s Department of Electrical Engineering and Computer Science (EECS). “This work has really put these devices at a point where they now look really promising for future applications.”
“The working mechanism of the device is the electrochemical insertion of the smallest ion, the proton, into an insulating oxide to modulate its electronic conductivity. Because we are working with very thin devices, we could accelerate the motion of this ion by using a strong electric field, and push these ionic devices to the nanosecond operation regime,” explains senior author Bilge Yildiz, the Breene M. Kerr Professor in the departments of Nuclear Science and Engineering and Materials Science and Engineering.
With that key insight and the very powerful nanofabrication techniques we have at MIT.nano, we have been able to put these pieces together and demonstrate that these devices are intrinsically very fast and operate with reasonable voltages. This work has really put these devices at a point where they now look really promising for future applications.
Professor Jesús A. del Alamo
“The action potential in biological cells rises and falls with a timescale of milliseconds since the voltage difference of about 0.1 volt is constrained by the stability of water,” says senior author Ju Li, the Battelle Energy Alliance Professor of Nuclear Science and Engineering and professor of materials science and engineering,
“Here we apply up to 10 volts across a special solid glass film of nanoscale thickness that conducts protons, without permanently damaging it. And the stronger the field, the faster the ionic devices.”
These programmable resistors greatly accelerate neural network training while also significantly lowering the cost and energy required. This might speed up the process through which researchers create deep learning models that can be used for things like fraud detection, self-driving cars, or picture analysis in medicine.
“Once you have an analog processor, you will no longer be training networks everyone else is working on. You will be training networks with unprecedented complexities that no one else can afford to, and therefore vastly outperform them all. In other words, this is not a faster car, this is a spacecraft,” adds lead author and MIT postdoc Murat Onen.
Co-authors include Frances M. Ross, the Ellen Swallow Richards Professor in the Department of Materials Science and Engineering; postdocs Nicolas Emond and Baoming Wang; and Difei Zhang, an EECS graduate student. The research is published today in Science.
Accelerating deep learning
For two key reasons, analog deep learning is faster and more energy-efficient than its digital version. “First, computation is performed in memory, so enormous loads of data are not transferred back and forth from memory to a processor.”
Parallel processes are also carried out by analog processors. An analog processor doesn’t require more time to perform new operations as the size of the matrix increases because all computation happens simultaneously.
A protonic programmable resistor is the main component of the new analog processor technology developed by MIT. These nanoscale-sized resistors are placed like a chessboard in an array. One nanometer is one billionth of a meter.
Learning occurs in the human brain as a result of the strengthening and weakening of synapses, the connections between neurons. This approach, where the network weights are programmed by training algorithms, has been used by deep neural networks for a long time.
Analog machine learning is possible with this new processor by varying the electrical conductivity of protonic resistors. The motion of the protons governs the conductance.
More protons are pushed into a resistor channel to increase conductance, and more protons are pulled out to decrease conductance. This is done by conducting protons but blocking electrons in an electrolyte (similar to a battery’s electrolyte).
The researchers investigated various materials for the electrolyte in order to create a protonic resistor that is programmable, extremely quick, and highly energy efficient. Onen concentrated on inorganic phosphosilicate glass (PSG) instead of other devices’ usage of organic chemicals.
PSG, the powdery desiccant agent used to eliminate moisture and found in little bags in new furniture boxes, is essentially silicon dioxide. It is also the most popular oxide utilized in the production of silicon. To create PSG, silicon is given unique properties for proton conduction by adding a small amount of phosphorus.
Onen postulated that an improved PSG, which would make a perfect solid electrolyte for this purpose, may have a high proton conductivity at room temperature without the requirement for water. He was right.
Surprising speed
Because PSG has a large number of nanometer-sized pores with surfaces that act as pathways for proton diffusion, it allows for rapid proton transport. Additionally, it can endure extremely powerful, pulsed electric fields. Onen argues that this is crucial because increasing the device’s voltage enables protons to flow at dizzying speeds.
“The speed certainly was surprising. Normally, we would not apply such extreme fields across devices, in order to not turn them into ash. But instead, protons ended up shuttling at immense speeds across the device stack, specifically a million times faster compared to what we had before. And this movement doesn’t damage anything, thanks to the small size and low mass of protons. It is almost like teleporting,” he says.
“The nanosecond timescale means we are close to the ballistic or even quantum tunneling regime for the proton, under such an extreme field,” adds Li.
The substance is not harmed by the protons, so the resistor can operate for millions of cycles without degrading. It is crucial for incorporating this new electrolyte into computing hardware since it made it possible to create a programmable protonic resistor that is a million times faster than their previous device and can function well at room temperature.
PSG is an electrically insulating substance, therefore nearly no electric current moves through it as protons move. This makes the device extremely energy efficient, Onen adds.
The researchers intend to re-engineer these programmable resistors for high-volume manufacture now that they have proven their efficacy, according to del Alamo. After that, they can scale up the resistor arrays for use in systems and analyze their properties.
They also intend to research the materials in order to get rid of any obstructions that prevent the voltage needed to effectively transfer protons into, through, and out of the electrolyte.
“Another exciting direction that these ionic devices can enable is energy efficient hardware to emulate the neural circuits and synaptic plasticity rules that are deduced in neuroscience, beyond analog deep neural networks,” adds Yildiz.
“The collaboration that we have is going to be essential to innovate in the future. The path forward is still going to be very challenging, but at the same time it is very exciting,” del Alamo says.
This research is funded, in part, by the MIT-IBM Watson AI Lab.
