Researchers used deep learning techniques to improve the image quality of a metalens camera. The new technology employs artificial intelligence to convert low-quality images into high-quality ones, potentially making these cameras suitable for a wide range of imaging activities, including sophisticated microscope applications and mobile devices.
Metalenses are ultrathin optical devices (typically less than a millimeter thick) that manipulate light using nanostructures. Although their small size could allow for incredibly compact and lightweight cameras without typical optical lenses, it has been challenging to obtain the required image quality with these optical components.
“Our technology allows our metalens-based devices to overcome the limitations of image quality,” said research team leader Ji Chen from Southeast University in China. “This advance will play an important role in the future development of highly portable consumer imaging electronics and can also be used in specialized imaging applications such as microscopy.”
A key part of this work was developing a way to generate the large amount of training data needed for the neural network learning process. Once trained, a low-quality image can be sent from the device to into the neural network for processing, and high-quality imaging results are obtained immediately.
Ji Chen
The researchers used a multi-scale convolutional neural network to improve resolution, contrast, and distortion in images from a small camera (3 cm × 3 cm × 0.5 cm) created by directly integrating a metalens onto a CMOS imaging chip, as described in Optica Publishing Group’s journal Optics Letters.
“Metalens-integrated cameras can be directly incorporated into the imaging modules of smartphones, where they could replace the traditional refractive bulk lenses,” Chen pointed out. “They could also be used in devices such as drones, where the small size and lightweight camera would ensure imaging quality without compromising the drone’s mobility.”
Enhancing image quality
The camera used in the new work was previously developed by the researchers and uses a metalens with 1000-nm tall cylindrical silicon nitride nano-posts. The metalens focuses light directly onto a CMOS imaging sensor without requiring any other optical elements. Although this design created a very small camera the compact architecture limited the image quality. Thus, the researchers decided to see if machine learning could be used to improve the images.
Deep learning is a sort of machine learning that use multilayered artificial neural networks to automatically learn features from data and make difficult judgments or predictions. Using a convolution imaging model, the researchers generated a huge number of high- and low-quality image pairs. These image pairs were used to train a multi-scale convolutional neural network, which could recognize the properties of each type of image and use them to transform low-quality photographs into high-quality ones.
“A key part of this work was developing a way to generate the large amount of training data needed for the neural network learning process,” said Chen. “Once trained, a low-quality image can be sent from the device to into the neural network for processing, and high-quality imaging results are obtained immediately.”
Applying the neural network
To validate the new deep learning technique, the researchers applied it to 100 test photos. They looked at two common image processing metrics: the peak signal-to-noise ratio and the structure similarity index. They discovered that photos processed by the neural network showed considerable improvements in both metrics. They also demonstrated that the method could quickly provide high-quality imaging data that nearly matched what was collected directly through experimentation.
The researchers are currently constructing metalenses with complicated functionalities, such as color or wide-angle imaging, and developing neural network algorithms to improve the image quality of these advanced metalenses. To make this technology commercially viable, new assembly procedures for integrating metalenses into smartphone imaging modules would be required, as well as picture quality enhancing software created exclusively for mobile phones.
“Ultra-lightweight and ultra-thin metalenses represent a revolutionary technology for future imaging and detection,” said Chen. “Leveraging deep learning techniques to optimize metalens performance marks a pivotal developmental trajectory. We foresee machine learning as a vital trend in advancing photonics research.”
