Hostname: page-component-848d4c4894-tn8tq Total loading time: 0 Render date: 2024-07-01T08:16:01.312Z Has data issue: false hasContentIssue false
  • English
  • Français

O-Net: A Fast and Precise Deep-Learning Architecture for Computational Super-Resolved Phase-Modulated Optical Microscopy

Published online by Cambridge University Press:  15 June 2022

Shiraz S. Kaderuppan Open the ORCID record for Shiraz S. Kaderuppan [Opens in a new window] ,
Wai Leong Eugene Wong ,
Anurag Sharma  and
Wai Lok Woo
Shiraz S. Kaderuppan*
Affiliation:
Newcastle Research and Innovation Institute Pte Ltd, (NewRIIS), Devan Nair Institute for Employment and Employability, Singapore 609607 Faculty of Science, Agriculture and Engineering, Newcastle University, Newcastle upon Tyne NE1 7RU, UK
Wai Leong Eugene Wong
Affiliation:
Newcastle Research and Innovation Institute Pte Ltd, (NewRIIS), Devan Nair Institute for Employment and Employability, Singapore 609607 Faculty of Science, Agriculture and Engineering, Newcastle University, Newcastle upon Tyne NE1 7RU, UK Newcastle University in Singapore, SIT Building @ Nanyang Polytechnic, Singapore 567739
Anurag Sharma
Affiliation:
Newcastle Research and Innovation Institute Pte Ltd, (NewRIIS), Devan Nair Institute for Employment and Employability, Singapore 609607 Faculty of Science, Agriculture and Engineering, Newcastle University, Newcastle upon Tyne NE1 7RU, UK Newcastle University in Singapore, SIT Building @ Nanyang Polytechnic, Singapore 567739
Wai Lok Woo
Affiliation:
Faculty of Science, Agriculture and Engineering, Newcastle University, Newcastle upon Tyne NE1 7RU, UK Computer and Information Sciences, Northumbria University, Newcastle upon Tyne NE1 8ST, UK
*
*Corresponding author: Shiraz S. Kaderuppan, E-mail: s.s.o.kaderuppan@newcastle.ac.uk
Get access

Abstract

We present a fast and precise deep-learning architecture, which we term O-Net, for obtaining super-resolved images from conventional phase-modulated optical microscopical techniques, such as phase-contrast microscopy and differential interference contrast microscopy. O-Net represents a novel deep convolutional neural network that can be trained on both simulated and experimental data, the latter of which is being demonstrated in the present context. The present study demonstrates the ability of the proposed method to achieve super-resolved images even under poor signal-to-noise ratios and does not require prior information on the point spread function or optical character of the system. Moreover, unlike previous state-of-the-art deep neural networks (such as U-Nets), the O-Net architecture seemingly demonstrates an immunity to network hallucination, a commonly cited issue caused by network overfitting when U-Nets are employed. Models derived from the proposed O-Net architecture are validated through empirical comparison with a similar sample imaged via scanning electron microscopy (SEM) and are found to generate ultra-resolved images which came close to that of the actual SEM micrograph.

Keywords

optical microscopydeep learningcomputational nanoscopyphase-modulated microscopysuper-resolution microscopy
Type
Software and Instrumentation
Information
Microscopy and Microanalysis , Volume 28 , Issue 5 , October 2022 , pp. 1584 - 1598
Copyright
Copyright © The Author(s), 2022. Published by Cambridge University Press on behalf of the Microscopy Society of America

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Abramowitz, M & Davidson, MW (n.d. a). Hoffman Modulation Contrast Basics. EVIDENT|OLYMPUS CORPORATION. Available at https://www.olympus-lifescience.com/en/microscope-resource/primer/techniques/hoffman/ (retrieved July 4, 2021).Google Scholar
Abramowitz, M & Davidson, MW (n.d. b). Introduction to Phase Contrast. EVIDENT|OLYMPUS CORPORATION. Available at https://www.olympus-lifescience.com/en/microscope-resource/primer/techniques/phasecontrast/phase/ (retrieved April 19, 2021).Google Scholar
AIVIA (n.d.). Leica Microsystems. Available at https://www.aivia-software.com/aivia (retrieved July 15, 2021).Google Scholar
Akst, J (2018). AI Networks Generate Super-resolution from Basic Microscopy. Available at https://www.the-scientist.com/news-opinion/ai-networks-generate-super-resolution-from-basic-microscopy-65219 (retrieved October 14, 2021).Google Scholar
AMD Radeon™ RX Graphics Cards (n.d.). Advanced Micro Devices, Inc. Available at https://www.amd.com/en/graphics/radeon-rx-graphics (retrieved July 4, 2021).Google Scholar
Artificial Intelligence for 3D Visualization and Analysis Software (n.d.). Thermo Fisher Scientific Inc. Available at https://www.thermofisher.com/us/en/home/electron-microscopy/products/software-em-3d-vis/3d-visualization-analysis-software/artificial-intelligence.html (retrieved July 15, 2021).Google Scholar
AWS Deep Learning AMIs (n.d.). Amazon Web Services, Inc. Available at https://aws.amazon.com/machine-learning/amis/ (retrieved July 4, 2021).Google Scholar
Bagnell, CR Jr. (2012). Chapter 11 – Differential Interference Contrast Microscopy. Available at https://www.med.unc.edu/microscopy/files/2018/06/lm-ch-11-dic.pdf (retrieved November 2, 2019).Google Scholar
Barone, S (n.d.). Diatom Shop. Diatom Lab. Available at http://www.diatomshop.com/#:~:text=Diatom%20Test%20Slide%20of%20Micromanipulated%20Pinnularia%20dactylus%20var (retrieved July 4, 2021).Google Scholar
Bayraci, S & Susuz, O (2019). A deep neural network (DNN) based classification model in application to loan default prediction. Theor Appl Econ 4(621), 75–84.Google Scholar
Belthangady, C & Royer, LA (2019). Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat Methods 16(12), 1215–1225.CrossRefGoogle ScholarPubMed
Betzig, E, Patterson, GH, Sougrat, R, Lindwasser, OW, Olenych, S, Bonifacino, JS, Davidson, MW, Lippincott-Schwartz, J & Hess, HF (2006). Imaging intracellular fluorescent proteins at nanometer resolution. Science 313(5793), 1642–1645.CrossRefGoogle ScholarPubMed
Brownlee, J (2019). How to Evaluate Generative Adversarial Networks. Machine Learning Mastery Pty. Ltd. Available at https://machinelearningmastery.com/how-to-evaluate-generative-adversarial-networks/ (retrieved July 2, 2022).Google Scholar
Chambers, W, Fellers, TJ & Davidson, MW (n.d. a). Darkfield illumination. Nikon Instruments Inc. Available at https://www.microscopyu.com/techniques/stereomicroscopy/darkfield-illumination (retrieved July 4, 2021).Google Scholar
Chambers, W, Fellers, TJ & Davidson, MW (n.d. b). Oblique Illumination. Nikon Instruments Inc. Available at https://www.microscopyu.com/techniques/stereomicroscopy/oblique-illumination (retrieved July 4, 2021).Google Scholar
Cohn, C (2015). A Beginner's Guide to Upselling and Cross-selling. Forbes. Available at https://www.forbes.com/sites/chuckcohn/2015/05/15/a-beginners-guide-to-upselling-and-cross-selling/?sh=1a725d182912 (retrieved July 4, 2021).Google Scholar
GeForce RTX 30 Series (n.d.). NVIDIA Corporation. Available at https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/ (retrieved July 4, 2021).Google Scholar
Girshick, R, Donahue, J, Darrell, T & Malik, J (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, pp. 580–587.CrossRefGoogle Scholar
Godoy, D (2018). Understanding Binary Cross-Entropy/log Loss: A Visual Explanation. Retrieved from Towards Data Science: https://towardsdatascience.com/understanding-binary-cross-entropy-log-loss-a-visual-explanation-a3ac6025181a.Google Scholar
He, K, Zhang, X, Ren, S & Sun, J (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 770–778.CrossRefGoogle Scholar
Hendrycks, D & Gimpel, K (2020). Gaussian Error Linear Units (GELUs). arXiv:1606.08415.Google Scholar
Hoffman, DP, Slavitt, I & Fitzpatrick, CA (2021). The promise and peril of deep learning in microscopy. Nat Methods 18, 131–132.CrossRefGoogle ScholarPubMed
IBM Watson products (n.d.). IBM. Available at https://www.ibm.com/watson/products-services (retrieved July 4, 2021).Google Scholar
Introduction to Modulation Transfer Function (n.d.). Edmund Optics Inc. Available at https://www.edmundoptics.com/knowledge-center/application-notes/optics/introduction-to-modulation-transfer-function/ (retrieved July 4, 2021).Google Scholar
Isola, P, Zhu, J -Y, Zhu, T & Efros, AA (2017). Image-to-Image translation with conditional adversarial networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, pp. 5967–5976.CrossRefGoogle Scholar
Kaderuppan, SS, Wong, EWL, Sharma, A & Woo, WL (2020). Smart nanoscopy: A review of computational approaches to achieve super-resolved optical microscopy. IEEE Access 8, 214801–214831.CrossRefGoogle Scholar
Kaderuppan, SS, Wong, EWL, Sharma, A & Woo, WL (2022). Impact analysis of deep neural network training methodology on computational nanoscopy. In Focus on Microscopy (FOM) 2022 – Online. Available at https://www.focusonmicroscopy.org/2022-program-online/?event_id=27 (retrieved April 19, 2022).Google Scholar
Kızrak, A (2019). Deep Learning for Image Segmentation: U-Net Architecture (Medium). Available at https://heartbeat.fritz.ai/deep-learning-for-image-segmentation-u-net-architecture-ff17f6e4c1cf (retrieved July 4, 2021).Google Scholar
Krizhevsky, A, Sutskever, I & Hinton, G E (2012). Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems (NIPS) 2012, Lake Tahoe, NV, USA, Vol. 25, pp. 1097–1105.Google Scholar
LeCun, Y, Boser, B, Denker, JS, Henderson, D, Howard, RE, Hubbard, W & Jackel, LD (1989). Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4), 541–551.CrossRefGoogle Scholar
Murphy, DB, Oldfield, R, Schwartz, S & Davidson, MW (n.d.). Introduction to Phase Contrast Microscopy. Nikon – MicroscopyU. Available at https://www.microscopyu.com/techniques/phase-contrast/introduction-to-phase-contrast-microscopy (retrieved April 24, 2019).Google Scholar
Murphy, K (2012). Machine Learning: A Probabilistic Perspective. Cambridge, MA/London, UK: The MIT Press.Google Scholar
Nehme, E, Weiss, LE, Michaeli, T & Shechtman, Y (2018). Deep-STORM: Super-resolution single-molecule microscopy by deep learning. Optica 5(4), 458–464.CrossRefGoogle Scholar
Ouyang, W, Aristov, A, Lelek, M, Hao, X & Zimmer, C (2018). Deep learning massively accelerates super-resolution localization microscopy. Nat Biotechnol 36(5), 460–468.CrossRefGoogle ScholarPubMed
pix2pix: Image-to-image translation with a conditional GAN (2021). (TensorFlow) June 19. Available at https://www.tensorflow.org/tutorials/generative/pix2pix (retrieved July 4, 2021).Google Scholar
Ramachandran, P, Zoph, B & Le, QV (2017). Swish: A self-gated activation function. arXiv:1710.05941v1.Google Scholar
Ren, R, Guo, Z, Jia, Z, Yang, J, Kasabov, NK & Li, C (2019). Speckle noise removal in image-based detection of refractive index changes in porous silicon microarrays. Sci Rep 9, 15001.CrossRefGoogle ScholarPubMed
Rivenson, Y, Göröcs, Z, Günaydin, H, Zhang, Y, Wang, H & Ozcan, A (2017). Deep learning microscopy. Optica 4(11), 1437–1443.CrossRefGoogle Scholar
Robinson, PC & Davidson, MW (n.d.). Polarized Light Microscopy. Nikon Instruments Inc. Available at https://www.microscopyu.com/techniques/polarized-light/polarized-light-microscopy (retrieved July 4, 2021).Google Scholar
Ronneberger, O, Fischer, P & Brox, T (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. Freiburg: Computer Vision Group. Available at https://lmb.informatik.uni-freiburg.de/Publications/2015/RFB15a/ (retrieved 30 April, 2019).Google Scholar
Scientific Volume Imaging (n.d.). Huygens PSF distiller. Scientific Volume Imaging B.V. Available at https://svi.nl/Huygens-PSF-Distiller (retrieved July 4, 2021).Google Scholar
Sze, V, Chen, Y-H, Yang, T-J & Elmer, JS (2017). Efficient processing of deep neural networks: A tutorial and survey. Proc IEEE 105(12), 2295–2329.CrossRefGoogle Scholar
Wang, H, Rivenson, Y, Jin, Y, Wei, Z, Gao, R, Günaydın, H, Bentolila, LA, Kural, C & Ozcan, A (2019). Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat Methods 16, 103–110.CrossRefGoogle ScholarPubMed
Welcome To Colaboratory (n.d.). Google Research. Available at https://colab.research.google.com/notebooks/intro.ipynb?utm_source=scs-index (retrieved July 4, 2021).Google Scholar
Yang, S & Lee, B-U (2015). Poisson-Gaussian noise reduction using the hidden Markov model in contourlet domain for fluorescence microscopy images. PLoS One 10(9), e0136964.CrossRefGoogle ScholarPubMed
Zawacki-Richter, O, Marín, V, Bond, M & Gouverneur, F (2019). Systematic review of research on artificial intelligence applications in higher education – where are the educators? Int J Educ Technol High Educ 16, 39.CrossRefGoogle Scholar
Zhongyi, H, Benzheng, W, Yuanjie, Z, Yilong, Y, Kejian, L & Shuo, L (2017). Breast cancer multi-classification from histopathological images with structured deep learning model. Sci Rep 7, 4172.Google Scholar
Supplementary material: File

Kaderuppan et al. supplementary material

Kaderuppan et al. supplementary material 1

Download Kaderuppan et al. supplementary material(File)
File 94.8 MB
Supplementary material: File

Kaderuppan et al. supplementary material

Kaderuppan et al. supplementary material 2

Download Kaderuppan et al. supplementary material(File)
File 3.5 MB