Hostname: page-component-848d4c4894-xfwgj Total loading time: 0 Render date: 2024-07-07T02:48:14.089Z Has data issue: false hasContentIssue false
  • English
  • Français

Sparse Scanning Electron Microscopy Data Acquisition and Deep Neural Networks for Automated Segmentation in Connectomics

Published online by Cambridge University Press:  07 April 2020

Pavel Potocek ,
Patrick Trampert ,
Maurice Peemen ,
Remco Schoenmakers  and
Tim Dahmen
Pavel Potocek
Affiliation:
Materials and Structural Analysis Thermo Fisher Scientific, Eindhoven, The Netherlands
Patrick Trampert
Affiliation:
German Research Center for Artificial Intelligence, DFKI, Saarbrücken, Germany Saarland University, Saarbrücken, Germany
Maurice Peemen
Affiliation:
Materials and Structural Analysis Thermo Fisher Scientific, Eindhoven, The Netherlands
Remco Schoenmakers
Affiliation:
Materials and Structural Analysis Thermo Fisher Scientific, Eindhoven, The Netherlands
Tim Dahmen*
Affiliation:
German Research Center for Artificial Intelligence, DFKI, Saarbrücken, Germany
*
*Author for correspondence: Tim Dahmen, E-mail: tim.dahmen@dfki.de
Get access

Abstract

With the growing importance of three-dimensional and very large field of view imaging, acquisition time becomes a serious bottleneck. Additionally, dose reduction is of importance when imaging material like biological tissue that is sensitive to electron radiation. Random sparse scanning can be used in the combination with image reconstruction techniques to reduce the acquisition time or electron dose in scanning electron microscopy. In this study, we demonstrate a workflow that includes data acquisition on a scanning electron microscope, followed by a sparse image reconstruction based on compressive sensing or alternatively using neural networks. Neuron structures are automatically segmented from the reconstructed images using deep learning techniques. We show that the average dwell time per pixel can be reduced by a factor of 2–3, thereby providing a real-life confirmation of previous results on simulated data in one of the key segmentation applications in connectomics and thus demonstrating the feasibility and benefit of random sparse scanning techniques for a specific real-world scenario.

Keywords

connectomicsdeep learningscanning electron microscopysegmentationsparse scanning
Type
Software and Instrumentation
Information
Microscopy and Microanalysis , Volume 26 , Issue 3 , June 2020 , pp. 403 - 412
Copyright
Copyright © Microscopy Society of America 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Al-najjar, YY & Soong, DC (2012). Comparison of image quality assessment: PSNR, HVS, SSIM, UIQI. Int J Sci Eng Res 3, 1–5.Google Scholar
Anderson, HS, Ilic-Helms, J, Rohrer, B, Wheeler, J & Larson, K (2013). Sparse imaging for fast electron microscopy. In Computational Imaging, Bouman, CA, Pollak, I & Wolfe, PJ (Eds.), pp. 86570C. Burlingame, CA, USA: IS&T/SPIE Electronic Imaging.Google Scholar
Arganda-Carreras, I, Turaga, SC, Berger, DR, Cireşan, D, Giusti, A, Gambardella, LM, Schmidhuber, J, Laptev, D, Dwivedi, S, Buhmann, JM, Liu, T, Seyedhosseini, M, Tasdizen, T, Kamentsky, L, Burget, R, Uher, V, Tan, X, Sun, C, Pham, TD, Bas, E, Uzunbas, MG, Cardona, A, Schindelin, J & Seung, HS (2015). Crowdsourcing the creation of image segmentation algorithms for connectomics. Front Neuroanat 9.CrossRefGoogle ScholarPubMed
Bota, M, Sporns, O & Swanson, LW (2015). Architecture of the cerebral cortical association connectome underlying cognition. Proc Natl Acad Sci 112, E2093–E2101.CrossRefGoogle ScholarPubMed
Botha, BPC (2013). Visual Computing for Medicine, 2nd ed., The Morgan Kaufmann Series in Computer Graphics. Elsevier Science.Google Scholar
Boughorbel, F, Potoček, P, Hovorka, M, Strakoš, L, Mitchels, J, Vystavěl, T, Trampert, P, Ben Lich, A & Dahmen, T (2017). High throughput large volume SEM workflow using sparse scanning and in-painting algorithms inspired by compressive sensing. Microsc Microanal 23(Suppl S1), 150–151.CrossRefGoogle Scholar
Carlson, DB & Evans, JE (2012). Low-dose imaging techniques for transmission electron microscopy. In The Transmission Electron Microscope, Maaz, K (Ed.), pp. 85–98. InTech.Google Scholar
Chen, L-C, Papandreou, G, Kokkinos, I, Murphy, K & Yuille, AL (2018). DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans Pattern Anal Mach Intell 40, 834–848.CrossRefGoogle ScholarPubMed
Dahmen, T, Engstler, M, Pauly, C, Trampert, P, De Jonge, N, Mücklich, F & Slusallek, P (2016). Feature adaptive sampling for scanning electron microscopy. Sci Rep 6, 25350.CrossRefGoogle ScholarPubMed
Dahmen, T, Potocek, P, Trampert, P, Peemen, M & Schoenmakers, R (2019). Sparse scanning electron microscopy and deep learning for imaging and segmentation of neuron structures. Microsc Microanal 25, 196–197.CrossRefGoogle Scholar
Deerinck, TJ, Bushong, E, Thor, A & Ellisman, M (2010). NCMIR methods for 3D EM: A new protocol for preparation of biological specimens for serial block face scanning electron microscopy. Nat Center Microsc Image Res 6–8.Google Scholar
Dong, C, Loy, CC, He, K & Tang, X (2014). Learning a deep convolutional network for image super-resolution. In Computer Vision – ECCV 2014, Zurich, Switzerland, pp. 184–199.CrossRefGoogle Scholar
Donoho, DL (2006). Compressed sensing. IEEE Trans Inform Theory 52, 1289–1306.CrossRefGoogle Scholar
Dusek, J & Roubik, K (2003). Testing of new models of the human visual system for image quality evaluation. In Seventh International Symposium on Signal Processing and Its Applications, 2003. Proceedings, vol. 2, pp. 621–622. Paris, France: IEEE.CrossRefGoogle Scholar
Hawe, S, Kleinsteuber, M & Diepold, K (2013). Analysis operator learning and its application to image reconstruction. IEEE Trans Image Process 22, 2138–2150.CrossRefGoogle ScholarPubMed
He, K, Zhang, X, Ren, S & Sun, J (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. 4, pp. 770–778. Las Vegas, NV, USA: IEEE.CrossRefGoogle Scholar
Hilbert, D (1891). Über die stetige Abbildung einer Linie auf ein Flächenstück. Math Ann 38, 459–460.CrossRefGoogle Scholar
Hildebrand, DGC, Cicconet, M, Torres, RM, Choi, W, Quan, TM, Moon, J, Wetzel, AW, Scott Champion, A, Graham, BJ, Randlett, O, Plummer, GS, Portugues, R, Bianco, IH, Saalfeld, S, Baden, AD, Lillaney, K, Burns, R, Vogelstein, JT, Schier, AF, Lee, W-CA, Jeong, W-K, Lichtman, JW & Engert, F (2017). Whole-brain serial-section electron microscopy in larval zebrafish. Nature 545, 345–349.CrossRefGoogle ScholarPubMed
Keller, AL, Zeidler, D & Kemen, T (2014). High throughput data acquisition with a multi-beam SEM. In Proc. SPIE 9236, Postek MT, Newbury DE, Platek SF & Maugel TK (Eds.), p. 92360B. Las Vegas, NV, USA: IEEE.Google Scholar
Kim, J, Lee, JK & Lee, KM (2016). Accurate image super-resolution using very deep convolutional networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1646–1654. Las Vegas, NV, USA: IEEE.CrossRefGoogle Scholar
Lehtinen, J, Munkberg, J, Hasselgren, J, Laine, S, Karras, T, Aittala, M & Aila, T (2018). Noise2Noise: Learning Image Restoration without Clean Data. In Proceedings of the 35th International Conference on Machine Learning, in PMLR 80, pp. 2965–2974.Google Scholar
Lichtman, JW, Pfister, H & Shavit, N (2014). The big data challenges of connectomics. Nat Neurosci 17, 1448–1454.CrossRefGoogle ScholarPubMed
Martin, D, Fowlkes, C, Tal, D & Malik, J (2001). A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. Proc IEEE Int Conf Comput Vision 2, 416–423.CrossRefGoogle Scholar
Mennes, M, Biswal, BB, Castellanos, FX & Milham, MP (2013). Making data sharing work: The FCP/INDI experience. NeuroImage 82, 683–691.CrossRefGoogle ScholarPubMed
Mousavi, A & Baraniuk, RG (2017). Learning to invert: Signal recovery via deep convolutional networks. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2272–2276. New Orleans, LA, USA: IEEE.CrossRefGoogle Scholar
Natarajan, BK (1995). Sparse approximate solutions to linear systems. SIAM J Comput 24, 227–234.CrossRefGoogle Scholar
Nowozin, S (2014). Optimal decisions from probabilistic models: The intersection-over-union case. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 548–555. Columbus, Ohio, USA: IEEE.CrossRefGoogle Scholar
Oho, E, Sugawara, T & Suzuki, K (2005). An improved scanning method based on characteristics of the human visual system for scanning electron microscopy. Scanning 27, 170–175.CrossRefGoogle ScholarPubMed
Pereira, AF, Hageman, DJ, Garbowski, T, Riedesel, C, Knothe, U, Zeidler, D & Knothe Tate, ML (2016). Creating high-resolution multiscale maps of human tissue using multi-beam SEM. PLoS Comput Biol 12, e1005217.CrossRefGoogle ScholarPubMed
Potocek, P, Schoenmakers, R, Trampert, P, Dahmen, T & Peemen, M (2018). Sparse scanning electron microscopy for imaging and segmentation in connectomics. In IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2018, Madrid, Spain, December 3–6, 2018, pp. 2461–2465.CrossRefGoogle Scholar
Quan, TM, Hildebrand, DGC & Jeong, W-K (2016). FusionNet: A deep fully residual convolutional neural network for image segmentation in connectomics. arXiv 1612.05360.Google Scholar
Rani, M, Dhok, SB & Deshmukh, RB (2018). A systematic review of compressive sensing: Concepts, implementations and applications. IEEE Access 6, 4875–4894.CrossRefGoogle Scholar
Ronneberger, O, Fischer, P & Brox, T (2015). U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention (MICCAI), vol. LNCS 9351, LNCS, pp. 234–241. Springer.CrossRefGoogle Scholar
Sanders, T & Dwyer, C (2020). Inpainting versus denoising for dose reduction in scanning-beam microscopies. IEEE Trans Image Process 29, 351–359.CrossRefGoogle Scholar
Sheikh, HR & Bovik, AC (2006). Image information and visual quality. IEEE Trans Image Process 15, 430–444.CrossRefGoogle ScholarPubMed
Swanson, LW & Lichtman, JW (2016). From cajal to connectome and beyond. Ann Rev Neurosci 39, 197–216.CrossRefGoogle ScholarPubMed
Timischl, F (2014). A dynamic scanning method based on signal-statistics for scanning electron microscopy. Scanning 36, 317–326.CrossRefGoogle ScholarPubMed
Trampert, P, Bourghorbel, F, Potocek, P, Peemen, M, Schlinkmann, C, Dahmen, T & Slusallek, P (2018 a). How should a fixed budget of dwell time be spent in scanning electron microscopy to optimize image quality? Ultramicroscopy 191, 11–17.CrossRefGoogle ScholarPubMed
Trampert, P, Schlabach, S, Dahmen, T & Slusallek, P (2018 b). Exemplar-based inpainting based on dictionary learning for sparse scanning electron microscopy. Microsc Microanal 24, 700–701.CrossRefGoogle Scholar
Trampert, P, Schlabach, S, Dahmen, T & Slusallek, P (2019). Deep learning for sparse scanning electron microscopy. Microsc Microanal 25, 158–159.CrossRefGoogle Scholar
Trépout, S (2019). Tomographic collection of block-based sparse STEM images: Practical implementation and impact on the quality of the 3D reconstructed volume. Materials 12, 2281.CrossRefGoogle ScholarPubMed
Wang, C, Qian, X, Yan, Y, Dong, F & Wang, H (2008). An evaluation method for reconstructed images in electrical tomography. In IEEE Instrumentation and Measurement Technology Conference, Victoria, BC, pp. 692–696.CrossRefGoogle Scholar
Wang, Z, Bovik, AC, Sheikh, HR & Simoncelli, EP (2004). Image quality assessment: From error visibility to structural similarity. IEEE Trans Image Process 13, 600–612.CrossRefGoogle ScholarPubMed
Xie, W, Feng, Q, Srinivasan, R, Stevens, A & Browning, ND (2017). Acquisition of STEM images by adaptive compressive sensing. Microsc Microanal 23, 96–97.CrossRefGoogle Scholar
Zhang, K, Zuo, W, Chen, Y, Meng, D & Zhang, L (2017). Beyond a gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Trans Image Process 26, 3142–3155.CrossRefGoogle ScholarPubMed
Zhou, M, Chen, H, Paisley, J, Ren, L, Li, L, Xing, Z, Dunson, D, Sapiro, G & Carin, L (2012). Nonparametric bayesian dictionary learning for analysis of noisy and incomplete images. IEEE Trans Image Process 21, 130–144.CrossRefGoogle ScholarPubMed
Supplementary material: File

Potocek et al. supplementary material

Potocek et al. supplementary material 1

Download Potocek et al. supplementary material(File)
File 20 MB
Supplementary material: File

Potocek et al. supplementary material

Potocek et al. supplementary material 2

Download Potocek et al. supplementary material(File)
File 1.3 MB
Supplementary material: File

Potocek et al. supplementary material

Potocek et al. supplementary material 3

Download Potocek et al. supplementary material(File)
File 55.2 MB
Supplementary material: File

Potocek et al. supplementary material

Potocek et al. supplementary material 4

Download Potocek et al. supplementary material(File)
File 546.9 KB
Supplementary material: File

Potocek et al. supplementary material

Potocek et al. supplementary material 5

Download Potocek et al. supplementary material(File)
File 70.5 KB