About this article
- Submitted: Jan 12, 2023
- Final Revised: Jun 01, 2023
- Accepted: May 07, 2025
- Published: Jun 30, 2023
- Abstract view: 70
- PDF Download: 1
- Volumes: Vol. 1 No. 1 (2023) Pages 1-14
- DOI: 10.63208/21018-16
License
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Copyright (c) 2023 Author(s)
Cite this article
Abstract
Noise and structural distortion significantly affect the performance of Single Image Super-Resolution (SISR). While recent advancements leverage Generative Adversarial Networks (GANs) to produce photo-realistic images, handling noise and preserving structure remain challenging. This paper introduces a novel approach, termed SNS, which enhances SISR for noisy images by integrating a denoising preprocessing module and a structure-preserving gradient branch. The denoising module learns the noise distribution and employs residual-skip connections to effectively remove noise before super-resolution. Simultaneously, the gradient branch restores high-resolution gradient maps and incorporates gradient and spatial losses to guide optimization, thus enforcing structural fidelity. GAN-based mechanisms are retained to synthesize fine details. Experimental evaluations demonstrate that the proposed method achieves superior performance on noisy datasets, with improvements in Perceptual Index (PI) and Learned Perceptual Image Patch Similarity (LPIPS), while maintaining competitive Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) compared to existing approaches, including those combined with DNCNN. On the Urban100 dataset (noise level 25), SNS achieved 3.6976 (PI), 0.1124 (LPIPS), 24.652 (PSNR), and 0.9481 (SSIM), confirming its effectiveness.
References
Goodfellow, I.J.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; Bengio, Y. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27; Neural Information Processing Systems Foundation, Inc.: San Diego, CA, USA, 2014.
Zhang, K.; Zuo, W.M.; Chen, Y.J.; Meng, D.Y.; Zhang, L. Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising. IEEE Trans. Image Process. 2007, 26, 3142–3155.
Ma, C.; Rao, Y.; Cheng, Y.; Chen, C.; Lu, J.; Zhou, J. Structure-Preserving Super Resolution with Gradient Guidance. arXiv, 2020; arXiv:2003.13081.
Hwang, H.; Haddad, R.A. Adaptive median filters: New algorithms and results. IEEE Trans. Image Process. 1995, 4, 499–502.
Zhang, P.X.; Li, F. A New Adaptive Weighted Mean Filter for Removing Salt-and-Pepper Noise. IEEE Signal Process. Lett. 2014, 21, 1280–1283.
Liu, L.; Chen, C.P.; Zhou, Y.; You, X. A new weighted mean filter with a two-phase detector for removing impulse noise. Inf. Sci. 2015, 315, 1–16.
Kandemir, C.; Kalyoncu, C.; Toygar, Ö. A weighted mean filter with spatial-bias elimination for impulse noise removal. Dig. Signal Process. 2015, 46, 164–174.
Zhou, Z. Cognition and Removal of Impulse Noise With Uncertainty. IEEE Trans. Image Process. 2012, 21, 3157–3167.
Li, Z.; Tang, K.; Cheng, Y.; Chen, X.; Zhou, C. Modified Adaptive Gaussian Filter for Removal of Salt and Pepper Noise. KSII Trans. Internet Inf. Syst. 2015, 9, 2928–2947.
Nasri, M.; Saryazdi, S.; Nezamabadi-pour, H. A Fast Adaptive Salt and Pepper Noise Reduction Method in Images. Circ. Syst. Signal Pr. 2013, 32, 1839–1857.
Huang, Y.; Zhang, Y.; Li, N.; Chambers, J. A robust Gaussian approximate filter for nonlinear systems with heavy tailed measurement noises. In Proceedings of the 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Shanghai, China, 20–25 March 2016; pp. 4209–4213.
Jain, V.; Seung, H.S. Natural image denoising with convolutional networks. In Proceedings of the 21st International Conference on Neural Information Processing Systems, Vancouver, BC, Canada, 3–6 December 2007; pp. 769–776.
Burger, H.C.; Schuler, C.J.; Harmeling, S. Image denoising: Can plain neural networks compete with BM3D? In Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 16–21 June 2012; pp. 2392–2399.
He, K.; Zhang, X.; Ren, S.; Sun, J. Deep Residual Learning for Image Recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27–30 June 2016; pp. 770–778.
Lehtinen, J.; Munkberg, J.; Hasselgren, J.; Laine, S.; Karras, T.; Aittala, M.; Aila, T. Noise2Noise: Learning Image Restoration without Clean Data. arXiv, 2018; arXiv:1803.04189.
Chen, J.W.; Chen, J.W.; Chao, H.Y.; Yang, M. Image Blind Denoising With Generative Adversarial Network Based Noise Modeling. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3155–3164.
Chowdhuri, D.; Sendhil, K.K.S.; Babu, M.R.; Reddy, C.P.J.I.J.o.C.S.; Technologies, I. Very Low Resolution Face Recognition in Parallel Environment. Int. J. Comput. Sci. Inf. Technol. 2012, 3, 4408–4410.
Yang, Q.; Yang, R.; Davis, J.; Nister, D. Spatial-Depth Super Resolution for Range Images. In Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition, Minneapolis, MN, USA, 17–22 June 2007; pp. 1–8.
Nasrollahi, K.; Moeslund, T.B. Super-resolution: A comprehensive survey. Mach. Vis. Appl. 2014, 25, 1423–1468.
Ni, K.S.; Nguyen, T.Q. An Adaptable k-Nearest Neighbors Algorithm for MMSE Image Interpolation. IEEE Trans. Image Process. 2009, 18, 1976–1987.
Gribbon, K.T.; Bailey, D.G. A novel approach to real-time bilinear interpolation. In Proceedings of the DELTA 2004 Second IEEE International Workshop on Electronic Design, Test and Applications, Perth, Australia, 28–30 January 2004; pp. 126–131.
Fritsch, F.; Carlson, R. Monotone Piecewise Cubic Interpolation. SIAM J. Numer. Anal. 1980, 17, 238–246.
Flowerdew, R.; Green, M.; Kehris, E. Using areal interpolation methods in geographic information systems. Pap. Reg. Sci. 1991, 70, 303–315.
Dong, C.; Loy, C.C.; Tang, X.O. Accelerating the Super-Resolution Convolutional Neural Network. In Proceedings of the Computer Vision—ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, 11–14 October 2016; pp. 391–407.
Shi, W.Z.; Caballero, J.; Huszar, F.; Totz, J.; Aitken, A.P.; Bishop, R.; Rueckert, D.; Wang, Z.H. Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network. In Proceedings of the 2016 Ieee Conference on Computer Vision and Pattern Recognition (Cvpr) 2016, Las Vegas, NV, USA, 27–30 June 2016; pp. 1874–1883.
Kim, J.; Lee, J.K.; Lee, K.M. Accurate Image Super-Resolution Using Very Deep Convolutional Networks. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (Cvpr) 2016, Las Vegas, NV, USA, 27–30 June 2016; pp. 1646–1654.
Kim, J.; Lee, J.K.; Lee, K.M. Deeply-Recursive Convolutional Network for Image Super-Resolution. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (Cvpr) 2016, Las Vegas, NV, USA, 27–30 June 2016; pp. 1637–1645.
Tai, Y.; Yang, J.; Liu, X.M. Image Super-Resolution via Deep Recursive Residual Network. In Proceedings of the 30th IEEE Conference on Computer Vision and Pattern Recognition (Cvpr 2017), Honolulu, HI, USA, 21–26 July 2017; pp. 2790–2798.
Tong, T.; Li, G.; Liu, X.J.; Gao, Q.Q. Image Super-Resolution Using Dense Skip Connections. In Proceedings of the 2017 IEEE International Conference on Computer Vision (ICCV), Venice, Italy, 22–29 October 2017; pp. 4809–4817.
Wang, X.; Yu, K.; Wu, S.; Gu, J.; Liu, Y.; Dong, C.; Change Loy, C.; Qiao, Y.; Tang, X. ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks. arXiv, 2018; arXiv:1809.00219.
Hu, X.; Mu, H.; Zhang, X.; Wang, Z.; Tan, T.; Sun, J. Meta-SR: A Magnification-Arbitrary Network for Super-Resolution. arXiv, 2019; arXiv:1903.00875.
Wang, W.; Guo, R.; Tian, Y.; Yang, W. CFSNet: Toward a Controllable Feature Space for Image Restoration. In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea, 27 October–2 November 2019; pp. 4139–4148.
Martin, D.; Fowlkes, C.; Tal, D.; Malik, J. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings of the Eighth IEEE International Conference on Computer Vision, Vancouver, BC, Canada, 7–14 July 2001; pp. 416–423.
Singh, A.; Porikli, F.; Ahuja, N. Super-resolving Noisy Images. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA, 23–28 June 2014; pp. 2846–2853.
Laghrib, A.; Ezzaki, M.; El Rhabi, M.; Hakim, A.; Monasse, P.; Raghay, S. Simultaneous deconvolution and denoising using a second order variational approach applied to image super resolution. Comput. Vis. Image Underst. 2018, 168, 50–63.
Hu, J.; Wu, X.; Zhou, J.L. Noise robust single image super-resolution using a multiscale image pyramid. Signal Process 2018, 148, 157–171.
Chen, L.; Dan, W.; Cao, L.J.; Wang, C.; Li, J. Joint Denoising and Super-Resolution via Generative Adversarial Training. In Proceedings of the 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China, 20–24 August 2018; pp. 2753–2758.
Anoosheh, A.; Sattler, T.; Timofte, R.; Pollefeys, M.; Van Gool, L. Night-to-Day Image Translation for Retrieval-based Localization. In Proceedings of the 2019 International Conference on Robotics and Automation (ICRA), Montreal, QC, Canada, 20–24 May 2019; pp. 5958–5964.
Luan, F.J.; Paris, S.; Shechtman, E.; Bala, K. Deep Photo Style Transfer. In Proceedings of the 30th Ieee Conference on Computer Vision and Pattern Recognition (Cvpr 2017), Honolulu, HI, USA, 21–26 July 2017; pp. 6997–7005.
Fattal, R. Image upsampling via imposed edge statistics. ACM Trans. Graph. 2007, 26.
Yan, Q.; Xu, Y.; Yang, X.K.; Nguyen, T.Q. Single Image Superresolution Based on Gradient Profile Sharpness. IEEE Trans. Image Process. 2015, 24.
Sun, J.; Sun, J.; Xu, Z.B.; Shum, H.Y. Gradient Profile Prior and Its Applications in Image Super-Resolution and Enhancement. IEEE Trans. Image Process. 2011, 20, 1529–1542.
Liu, Y.; Zheng, C.W.; Zheng, Q.; Yuan, H.L. Removing Monte Carlo noise using a Sobel operator and a guided image filter. Vis. Comput. 2018, 34, 589–601.
Yang, W.H.; Feng, J.S.; Yang, J.C.; Zhao, F.; Liu, J.Y.; Guo, Z.M.; Yan, S.C. Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution. IEEE Trans. Image Process. 2017, 26, 5895–5907.
Bevilacqua, M.; Roumy, A.; Guillemot, C.; Morel, M.L.A. Low-Complexity Single-Image Super-Resolution based on Nonnegative Neighbor Embedding. In Proceedings of the British Machine Vision Conference, Surrey, UK, 3–7 September 2012.
Hui, Z.; Gao, X.B.; Yang, Y.C.; Wang, X.M. Lightweight Image Super-Resolution with Information Multi-distillation Network. In Proceedings of the 27th Acm International Conference on Multimedia (Mm’19), Nice, France, 21–25 October 2019; pp. 2024–2032.
Zhang, K.; Zuo, W.; Zhang, L. Learning a Single Convolutional Super-Resolution Network for Multiple Degradations. In Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June 2018; pp. 3262–3271.
Zeyde, R.; Elad, M.; Protter, M. On Single Image Scale-Up Using Sparse-Representations. In Proceedings of the 7th international conference on Curves and Surfaces, Avignon, France, 24–30 June 2010; pp. 711–730.
Huang, J.B.; Singh, A.; Ahuja, N. Single Image Super-resolution from Transformed Self-Exemplars. In Proceedings of the 2015 Ieee Conference on Computer Vision and Pattern Recognition (Cvpr), Boston, MA, USA, 7–12 June 2015; pp. 5197–5206.
Blau, Y.; Mechrez, R.; Timofte, R.; Michaeli, T.; Zelnik-Manor, L. The 2018 PIRM Challenge on Perceptual Image Super-resolution. arXiv, 2018; arXiv:1809.07517.
Zhang, R.; Isola, P.; Efros, A.A.; Shechtman, E.; Wang, O. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018; pp. 586–595.
Wang, Z.; Bovik, A.C.; Sheikh, H.R.; Simoncelli, E.P. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 2004, 13, 600–612.
Wang, X.T.; Yu, K.; Dong, C.; Loy, C.C. Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 18–23 June 2018; pp. 606–615.