Skip to main content Accessibility help

Automating material image analysis for material discovery

  • Chiwoo Park (a1) and Yu Ding (a2)


Advancements in temporal and spatial resolutions of microscopes promise to expand the frontiers of understanding in materials science. Imaging techniques produce images at a high-frame rate, streaming out a tremendous amount of data. Analysis of all these images is time-consuming and labor intensive, creating a bottleneck in material discovery that needs to be overcome. This paper summarizes recent progresses in machine learning and data science for expediting and automating material image analysis. The discussion covers both static image and dynamic image analyses, followed by remarks concerning ongoing efforts and future needs in automated image analysis that accelerates material discovery.


Corresponding author

Address all correspondence to Yu Ding at


Hide All
1.Basic Research Needs for Innovation and Discovery of Transformative Experimental Tools; available at, 2017.
2.Crewe, A.V.: Scanning transmission electron microscopy. J. Microsc. 100, 247259 (1974).
3.Salapaka, S.M. and Salapaka, M.V.: Scanning probe microscopy. IEEE Control Syst. 28, 6583 (2008).
4.Abellan, P., Mehdi, B.L., Parent, L.R., Gu, M., Park, C., Xu, W., Zhang, Y., Arslan, I., Zhang, J.-G., and Wang, C.-M.: Probing the degradation mechanisms in electrolyte solutions for Li-ion batteries by in situ transmission electron microscopy. Nano Lett. 14, 12931299 (2014).
5.Chien, M.P., Thompson, M.P., Barback, C.V., Ku, T.H., Hall, D.J., and Gianneschi, N.C.: Enzyme-directed assembly of a nanoparticle probe in tumor tissue. Adv. Mater. 25, 35993604 (2013).
6.Evans, J.E., Jungjohann, K.L., Browning, N.D., and Arslan, I.: Controlled growth of nanoparticles from solution with in situ liquid transmission electron microscopy. Nano Lett. 11, 28092813 (2011).
7.Kim, J.S., LaGrange, T., Reed, B.W., Taheri, M.L., Armstrong, M.R., King, W.E., Browning, N.D., and Campbell, G.H.: Imaging of transient structures using nanosecond in situ TEM. Science 321, 14721475 (2008).
8.LaGrange, T., Campbell, G.H., Reed, B., Taheri, M., Pesavento, J.B., Kim, J.S., and Browning, N.D.: Nanosecond time-resolved investigations using the in situ of dynamic transmission electron microscope (DTEM). Ultramicroscopy 108, 14411449 (2008).
9.Woehl, T.J., Evans, J.E., Arslan, I., Ristenpart, W.D., and Browning, N.D.: Direct in situ determination of the mechanisms controlling nanoparticle nucleation and growth. ACS Nano 6, 85998610 (2012).
10.Woehl, T.J., Park, C., Evans, J.E., Arslan, I., Ristenpart, W.D., and Browning, N.D.: Direct observation of aggregative nanoparticle growth: kinetic modeling of the size distribution and growth rate. Nano Lett. 14, 373378 (2013).
11.Patterson, J. P., Abellan, P., Denny, M. S. Jr., Park, C., Browning, N. D., Cohen, S. M., Evans, J. E., and Gianneschi, N. C.: Observing the growth of metal–organic frameworks by in situ liquid cell transmission electron microscopy. J. Am. Chem. Soc. 137, 73227328 (2015).
12.Jesse, S. and Kalinin, S.V.: Band excitation in scanning probe microscopy: sines of change. J. Phys. D: Appl. Phys. 44, 464006 (2011).
13.Rodriguez, B.J., Callahan, C., Kalinin, S.V., and Proksch, R.: Dual-frequency resonance-tracking atomic force microscopy. Nanotechnology 18, 475504 (2007).
14.Kalinin, S.V., Strelcov, E., Belianinov, A., Somnath, S., Vasudevan, R.K., Lingerfelt, E.J., Archibald, R.K., Chen, C., Proksch, R., Laanait, N., and Jesse, S.: Big, deep, and smart data in scanning probe microscopy. ACS Nano 10, 90689086 (2016).
15.Roco, M.C.: The Long View of Nanotechnology Development: The National Nanotechnology Initiative at 10 Years. Journal of Nanoparticles 13, 427445 (2011).
16.Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 679698 (1986).
17.Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9, 6266 (1979).
18.Jiang, X. and Mojon, D.: Adaptive local thresholding by verification-based multithreshold probing with application to vessel detection in retinal images. IEEE Trans. Pattern Anal. Mach. Intell. 25, 131137 (2003).
19.Vo, G. and Park, C.: Robust regression for image binarization under heavy noises and nonuniform background. Pattern Recognit. 81, 224239 (2018).
20.Park, C., Huang, J.Z., Huitink, D., Kundu, S., Mallick, B.K., Liang, H., and Ding, Y.: A multistage, semi-automated procedure for analyzing the morphology of nanoparticles. IIE Trans. 44, 507522 (2012).
21.Park, C., Huang, J.Z., Ji, J.X., and Ding, Y.: Segmentation, inference and classification of partially overlapping nanoparticles. IEEE Trans. Pattern Anal. Mach. Intell. 35, 669681 (2013).
22.Beucher, S. and Meyer, F.: The morphological approach to segmentation: the watershed transformation. Optical Engineering 34, 433433 (1992).
23.Qian, Y., Huang, J.Z., Li, X., and Ding, Y.: Robust nanoparticles detection from noisy background by fusing complementary image information. IEEE Trans. Image Process. 25, 57135726 (2016).
24.Zafari, S., Eerola, T., Sampo, J., Kälviäinen, H., and Haario, H.: Segmentation of overlapping elliptical objects in silhouette images. IEEE Trans. Image Process. 24, 59425952 (2015).
25.Meng, X.-L., and Rubin, D.B.: Maximum likelihood estimation via the ECM algorithm: a general framework. Biometrika 80, 267278 (1993).
26.Bubenik, P.: Statistical topological data analysis using persistence landscapes. J. Mach. Learn. Res. 16, 77102 (2015).
27.Konomi, B.A., Dhavala, S.S., Huang, J.Z., Kundu, S., Huitink, D., Liang, H., Ding, Y., and Mallick, B.K.: Bayesian object classification of gold nanoparticles. Ann. Appl. Stat. 7, 640668 (2013).
28.Frank, J.: Electron Tomography: Three-Dimensional Imaging with the Transmission Electron Microscope (New York, NY: Springer Science & Business Media, 2013).
29.Mu, C. and Park, C.: Optimal filtered backprojection for fast and accurate tomography reconstruction. Pattern Recognition Submitted (2019).
30.Li, X., Belianinov, A., Dyck, O., Jesse, S., and Park, C.: Two-level structural sparsity regularization for identifying lattices and defects in noisy images. Ann. Appl. Stat. 12, 348377 (2018).
31.Dong, L., Li, X., Qian, Y., Yu, D., Zhang, H., Zhang, Z., and Ding, Y.: Quantifying nanoparticle mixing state to account for both location and size effects. Technometrics 59, 391403 (2017).
32.Belianinov, A., He, Q., Kravchenko, M., Jesse, S., Borisevich, A., and Kalinin, S.V.: Identification of phases, symmetries and defects through local crystallography. Nat. Commun. 6, 7801 (2015).
33.Bright, D.S. and Steel, E.B.: Two-dimensional top hat filter for extracting spots and spheres from digital images. J. Microsc. 146, 191200 (1987).
34.Sage, D., Neumann, F.R., Hediger, F., Gasser, S.M., and Unser, M.: Automatic tracking of individual fluorescence particles: application to the study of chromosome dynamics. IEEE Trans. Image Process. 14, 13721383 (2005).
35.Hughes, J., Fricks, J., and Hancock, W.: Likelihood inference for particle location in fluorescence microscopy. Ann. Appl. Stat. 4, 830848 (2010).
36.Laanait, N., Ziatdinov, M., He, Q., and Borisevich, A.: Identifying local structural states in atomic imaging by computer vision. Adv. Struct. Chem. Imaging 2, 14 (2017).
37.Ripley, B.D.: The second-order analysis of stationary point processes. J. Appl. Probab. 13, 255266 (1976).
38.Li, X., Zhang, H., Jin, J., Huang, D., Qi, X., Zhang, Z., and Yu, D.: Quantifying dispersion of nanoparticles in polymer nanocomposites through transmission electron microscopy micrographs. J. Micro Nano-Manufacturing 2, 021008 (2014).
39.De Jonge, N. and Ross, F.M.: Electron microscopy of specimens in liquid. Nat. Nanotechnol. 6, 695 (2011).
40.Kalinin, S.V., Sumpter, B.G., and Archibald, R.K.: Big-deep-smart data in imaging for guiding materials design. Nat. Mater. 14, 973 (2015).
41.Zheng, H., Meng, Y.S., and Zhu, Y.: Frontiers of in situ electron microscopy. MRS Bull. 40, 1218 (2015).
42.Grzelczak, M., Vermant, J., Furst, E.M., and Liz-Marzán, L.M.: Directed self-assembly of nanoparticles. ACS Nano 4, 35913605 (2010).
43.Mikhailov, A. and Gundersen, G.: Relationship between microtubule dynamics and lamellipodium formation revealed by direct imaging of microtubules in cells treated with nocodazole or taxol. Cytoskeleton 41, 325340 (1998).
44.Bergen, L.G. and Borisy, G.G.: Head-to-tail polymerization of microtubules in vitro. Electron microscope analysis of seeded assembly. J. Cell Biol. 84, 141150 (1980).
45.Park, C.: Estimating multiple pathways of object growth using nonlongitudinal image data. Technometrics. 56, 186199 (2014).
46.Park, C. and Shrivastava, A.K.: Multimode geometric-profile monitoring with correlated image data and its application to nanoparticle self-assembly processes. J. Qual. Technol. 46, 216233 (2014).
47.Park, C., Woehl, T.J., Evans, J.E., and Browning, N.D.: Minimum cost multi-way data association for optimizing multitarget tracking of interacting objects. IEEE Trans. Pattern Anal. Mach. Intell. 37, 611624 (2015).
48.Qian, Y., Huang, J. Z.; Park, C., and Ding, Y.: Fast dynamic nonparametric distribution tracking in electron microscopic data. Ann. Appl. Stat. in press (2019).
49.Qian, Y., Huang, J.Z., and Ding, Y.: Identifying multi-stage nanocrystal growth using in situ TEM video data. IISE Trans. 49, 532543 (2017).
50.Zheng, H., Smith, R.K., Jun, Y.-W., Kisielowski, C., Dahmen, U., and Alivisatos, A.P.: Observation of single colloidal platinum nanocrystal growth trajectories. Science 324, 13091312 (2009).
51.Rodriguez, A. and Ter Horst, E.: Bayesian dynamic density estimation. Bayesian Anal. 3, 339365 (2008).
52.Mena, R.H. and Ruggiero, M.: Dynamic density estimation with diffusive Dirichlet mixtures. Bernoulli. (Andover) 22, 901926 (2016).
53.Jiang, H., Fels, S., and Little, J. J.: In A linear programming approach for multiple object tracking, 2007 IEEE Conference on Computer Vision and Pattern Recognition, IEEE: 2007; pp. 18.
54.Okuma, K., Taleghani, A., De Freitas, N., Little, J. J., and Lowe, D. G.: In A boosted particle filter: Multitarget detection and tracking, 2004 European Conference on Computer Vision, Springer, 2004; pp. 2839.
55.Pirsiavash, H., Ramanan, D., and Fowlkes, C. C.: In Globally-optimal greedy algorithms for tracking a variable number of objects, 2011 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 2011; pp. 12011208.
56.Jaqaman, K., Loerke, D., Mettlen, M., Kuwata, H., Grinstein, S., Schmid, S.L., and Danuser, G.: Robust single-particle tracking in live-cell time-lapse sequences. Nat. Methods 5, 695 (2008).
57.Henriques, J. F., Caseiro, R., and Batista, J.: In Globally optimal solution to multi-object tracking with merged measurements, 2011 IEEE International Conference on Computer Vision, IEEE, 2011; pp. 24702477.
58.Welch, D.A., Woehl, T.J., Park, C., Faller, R., Evans, J.E., and Browning, N.D.: Understanding the role of solvation forces on the preferential attachment of nanoparticles in liquid. ACS Nano 10, 181187 (2015).
59.Esmaieeli Sikaroudi, A., Welch, D.A., Woehl, T.J., Faller, R., Evans, J.E., Browning, N.D., and Park, C.: Directional statistics of preferential orientations of two shapes in their aggregate and Its application to nanoparticle aggregation. Technometrics 60, 332344 (2018).
60.Mehdi, B.L., Qian, J., Nasybulin, E., Park, C., Welch, D.A., Faller, R., Mehta, H., Henderson, W.A., Xu, W., and Wang, C.M.: Observation and quantification of nanoscale processes in lithium batteries by operando electrochemical (S) TEM. Nano Lett. 15, 21682173 (2015).
61.Touve, M.A., Figg, C.A., Wright, D.B., Park, C., Cantlon, J., Sumerlin, B.S., and Gianneschi, N.C.: Polymerization-induced self-assembly of micelles observed by liquid cell transmission electron microscopy. ACS Cent. Sci. 4, 543547 (2018).
62.Stevens, A., Luzi, L., Yang, H., Kovarik, L., Mehdi, B., Liyu, A., Gehm, M., and Browning, N.: A sub-sampled approach to extremely low-dose STEM. Appl. Phys. Lett. 112, 043104 (2018).
63.Kovarik, L., Stevens, A., Liyu, A., and Browning, N.D.: Implementing an accurate and rapid sparse sampling approach for low-dose atomic resolution STEM imaging. Appl. Phys. Lett. 109, 164102 (2016).
64.Stevens, A., Kovarik, L., Abellan, P., Yuan, X., Carin, L., and Browning, N.D.: Applying compressive sensing to TEM video: a substantial frame rate increase on any camera. Adv. Struct. Chem. Imaging 1, 10 (2015).
65.Castro, R., Haupt, J., and Nowak, R.: In Compressed sensing vs. active learning, 2006 IEEE International Conference on Acoustics, Speech and Signal Processing, IEEE, 2006; p. III.
66.Edgeworth, R. and Wilhelm, R.G.: Adaptive sampling for coordinate metrology. Prec. Eng. 23, 144154 (1999).
67.Park, C. and Qiu, P.: Sequential Adaptive Design for Jump Regression Estimation. Submitted (IEEE Transactions on Pattern Analysis and Machine Intelligence 2019). Also available at
68.Zewail, A. H. and Thomas, J. M.: 4D Electron Microscopy: Imaging in Space and Time. (Imperial College Press: London, 2009).
69.Sreehari, S., Venkatakrishnan, S., Bouman, K. L., Simmons, J. P., Drummy, L. F., and Bouman, C. A.: In Multi-resolution data fusion for super-resolution electron microscopy, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017; pp. 10841092.
70.Xia, H., Ding, Y., and Mallick, B.K.: Bayesian hierarchical model for combining misaligned two-resolution metrology data. IIE Trans. 43, 242258 (2011).
71.Ezzat, A.A., Pourhabib, A., and Ding, Y.: Sequential design for functional calibration of computer models. Technometrics. 60, 286296 (2018).
72.Pourhabib, A., Huang, J.Z., Wang, K., Zhang, C., Wang, B., and Ding, Y.: Modulus prediction of buckypaper based on multi-fidelity analysis involving latent variables. IIE Trans. 47, 141152 (2015).
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

MRS Communications
  • ISSN: 2159-6859
  • EISSN: 2159-6867
  • URL: /core/journals/mrs-communications
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed