Skip to main content Accessibility help
Hostname: page-component-cf9d5c678-9z9qw Total loading time: 0.907 Render date: 2021-07-31T02:25:13.976Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": true, "newCiteModal": false, "newCitedByModal": true, "newEcommerce": true, "newUsageEvents": true }

A self-driving microscope and the Atomic Forge

Published online by Cambridge University Press:  05 September 2019

Ondrej Dyck
Center for Nanophase Materials Sciences, Oak Ridge National Laboratory, USA
Stephen Jesse
Center for Nanophase Materials Sciences, Oak Ridge National Laboratory, USA
Sergei V. Kalinin
Center for Nanophase Materials Sciences, Oak Ridge National Laboratory, USA


Material Matters
Copyright © Materials Research Society 2019 

The electron microscope predates the transistor and the charge-coupled device (CCD). The opportunity to integrate these advancements into the electron microscope was seized and revolutionized in the modern (scanning) transmission electron microscope (STEM). Real-time analysis became possible and is now routine. Consider the efforts one would have to undertake to perform a Fourier transform in the days when images were acquired by exposing photographic plates. Indeed, it was real-time data analysis that enabled the measurement and correction of aberrations in modern instrumentation.Reference Pennycook and Nellist1 What are the next opportunities on the horizon for the STEM?

With the impressive progress made in the fields of deep learning, computer vision, and automation, we posit that the next revolution in microscopy will stem from the integration of these tool sets: a self-driving microscope. Such a machine will “understand” what it is looking at and automatically document features of interest. The microscopist will have high-level tools to tell the microscope to “look for distortions at that interface” or “obtain a tomographic reconstruction of this structure.” The microscope will know what various features look like by referencing databases, or it can be shown examples on the fly. Consider the trainable Weka segmentation plugin for ImageJ.Reference Arganda-Carreras, Kaynig, Rueden, Eliceiri, Schindelin, Cardona and Seung2,Reference Schneider, Rasband and Eliceiri3 This plugin allows the user to highlight regions of an image to inform the computer which features belong in which categories. As examples are added the computer becomes increasingly accurate at classifying the rest of the image automatically.

Using the Atomic Forge, Oak Ridge National Laboratory (ORNL) researchers brought two, three, and four silicon atoms together to build clusters (green) and make them rotate within a layer of graphene (blue). Photo credit: ORNL.

Advances in deep learning to interpret atomically resolved images are already beginning. For example, Ziatdinov et al. applied deep convolutional neural networks to automate the detection of molecular orientation, defect identification, and classification.Reference Ziatdinov, Dyck, Maksov, Hudak, Lupini, Song, Snijders, Vasudevan, Jesse and Kalinin4,Reference Ziatdinov, Dyck, Maksov, Li, Sang, Xiao, Unocic, Vasudevan, Jesse and Kalinin5 Maksov et al. illustrated the untangling of lattice dynamics involving the interaction of thousands of defects using deep learning and unsupervised unmixing strategies,Reference Maksov, Dyck, Wang, Xiao, Geohegan, Sumpter, Vasudevan, Jesse, Kalinin and Ziatdinov6 and Vasudevan et al. showed strategies for automated classification of Bravais lattice symmetries.Reference Vasudevan, Laanait, Ferragut, Wang, Geohegan, Xiao, Ziatdinov, Jesse, Dyck and Kalinin7 There is no fundamental impediment for implementing tools such as these in the microscope to tell the computer what to pay attention to and where to gather data. As these fields progress and as the techniques are refined and proven, integration into the microscope itself becomes increasingly attractive and powerful.

A parallel development is the recognition that the STEM can be used to tailor materials at the atomic level, termed the Atomic Forge.Reference Kalinin, Borisevich and Jesse8 The electron beam can alter materials but historically, steps have been taken to avoid beam alterations, as they typically preclude characterization of a pristine sample by introducing defects. Recently, however, these beam-induced modifications have been leveraged to produce remarkable demonstrations of materials manipulation at the nanometer and atomic scales. Jesse et al. demonstrated the controlled layer-by-layer growth from an amorphous precursor in strontium titanateReference Jesse, He, Lupini, Leonard, Oxley, Ovchinnikov, Unocic, Tselev, Fuentes-Cabrera, Sumpter, Pennycook, Kalinin and Borisevich9 and Si.Reference Jesse, Hudak, Zarkadoula, Song, Maksov, Fuentes-Cabrera, Ganesh, Kravchenko, Snijders, Lupini, Borisevich and Kalinin10 Hudak et al. showed how individual Bi atoms can be moved with the electron beam through a crystalline Si lattice to form patterned structures with atomic column precision.Reference Hudak, Song, Sims, Troparevsky, Humble, Pantelides, Snijders and Lupini11 Dyck et al. illustrated the introduction of single dopant Si atoms in graphene with near lattice site precision.Reference Dyck, Kim, Kalinin and Jesse12 Susi et al. laid the groundwork for predicting and demonstrating controlled Si dopant motion through a graphene lattice.Reference Susi, Kotakoski, Kepaptsoglou, Mangler, Lovejoy, Krivanek, Zan, Bangert, Ayala, Meyer and Ramasse13,Reference Susi, Meyer and Kotakoski14 And Dyck et al. demonstrated the assembly of primitive structures with Si dopants in graphene.Reference Dyck, Kim, Jimenez-Izal, Alexandrova, Kalinin and Jesse15

These demonstrations of such exquisite precision harken back to the initial atomic motion demonstrated in a scanning tunneling microscope by Eigler,Reference Crommie, Lutz and Eigler16Reference Eigler and Schweizer19 which ignited the field of nanotechnology. However, the range of beam-driven materials transformations observed in STEM covers the gamut, including particle fragmentation, precipitation, nanotube/nanowire growth, catalytic etching and growth, phase changes, deposition, sculpting, and welding.Reference Dyck, Ziatdinov, Lingerfelt, Unocic, Hudak, Lupini, Jesse and Kalinin20 Most of these are not performed with single atom precision, but the examples mentioned above show that this level of precision is certainly possible. What remains is to establish the techniques and tools that will enable such modifications to be performed routinely for the construction of atomically precise extended structures. This possibility is further enhanced by the idea of a self-driving microscope. If a microscope can drive itself, knows what it is looking at, and can respond “intelligently” to this information, perhaps it can also build structures at the atomic scale.

Both the self-driving microscope and the Atomic Forge are in the early stages of development, with the concepts themselves just beginning to crystalize. Many questions and challenges remain, and these ideas could easily be dismissed as unattainable. Aberration correction itself was repeatedly dismissed after multiple failed attempts;Reference Pennycook and Nellist1 nevertheless, we now enjoy the fruits of those who persisted in the face of doubt, and have almost achieved Feynman’s vision of a microscope with 100 times higher resolution than his time.Reference Pennycook21,Reference Feynman22 Similarly, the self-driving microscope and the Atomic Forge will fulfill another Feynman prophecy: “What would happen if we could arrange the atoms one by one the way we want them?”Reference Feynman22


This material is based upon work supported by the US Department of Energy, Office of Science, Division of Materials Science and Engineering, Basic Energy Sciences, and was performed at the Oak Ridge National Laboratory’s Center for Nanophase Materials Sciences, a US Department of Energy, Office of Science User Facility.

We welcome comments and feedback on this article via email


Pennycook, S.J., Nellist, P.D., Scanning Transmission Electron Microscopy: Imaging and Analysis (Springer Science & Business Media, Berlin, Germany, 2011).CrossRefGoogle Scholar
Arganda-Carreras, I., Kaynig, V., Rueden, C., Eliceiri, K.W., Schindelin, J., Cardona, A., Seung, H.S., Bioinformatics 33, 2424 (2017).CrossRefGoogle Scholar
Schneider, C.A., Rasband, W.S., Eliceiri, K.W., Nat. Methods 9, 671 (2012).CrossRefGoogle Scholar
Ziatdinov, M., Dyck, O., Maksov, A., Hudak, B.M., Lupini, A.R., Song, J., Snijders, P.C., Vasudevan, R.K., Jesse, S.,Kalinin, S.V., “Deep Analytics of Atomically-Resolved Images: Manifest and Latent Features,” preprint, arXiv:1801.05133 (2018).Google Scholar
Ziatdinov, M., Dyck, O., Maksov, A., Li, X., Sang, X., Xiao, K., Unocic, R.R., Vasudevan, R., Jesse, S., Kalinin, S.V., ACS Nano 11, 12742 (2017).CrossRefGoogle Scholar
Maksov, A., Dyck, O., Wang, K., Xiao, K., Geohegan, D.B., Sumpter, B.G., Vasudevan, R.K., Jesse, S., Kalinin, S.V., Ziatdinov, M., “Deep Learning Analysis of Defect and Phase Evolution during Electron Beam Induced Transformations in WS2,” preprint, arXiv:1803.05381 (2018).CrossRefGoogle Scholar
Vasudevan, R.K., Laanait, N., Ferragut, E.M., Wang, K., Geohegan, D.B., Xiao, K., Ziatdinov, M.A., Jesse, S., Dyck, O.E., Kalinin, S.V., “Mapping Mesoscopic Phase Evolution during E-beam Induced Transformations via Deep Learning of Atomically Resolved Images,” preprint, arXiv:1802.10518 (2018).CrossRefGoogle Scholar
Kalinin, S.V., Borisevich, A., Jesse, S., Nature 539, 485 (2016).CrossRefGoogle Scholar
Jesse, S., He, Q., Lupini, A.R., Leonard, D.N., Oxley, M.P., Ovchinnikov, O., Unocic, R.R., Tselev, A., Fuentes-Cabrera, M., Sumpter, B.G., Pennycook, S.J., Kalinin, S.V., Borisevich, A.Y., Small 11, 5895 (2015).CrossRefGoogle Scholar
Jesse, S., Hudak, B.M., Zarkadoula, E., Song, J., Maksov, A., Fuentes-Cabrera, M., Ganesh, P., Kravchenko, I., Snijders, P.C., Lupini, A.R., Borisevich, A., Kalinin, S.V., “Direct Atomic Fabrication and Dopant Positioning in Si Using Electron Beams with Active Real Time Image-Based Feedback,” preprint, arXiv:1711.05810 (2017).CrossRefGoogle Scholar
Hudak, B.M., Song, J., Sims, H., Troparevsky, M.C., Humble, T.S., Pantelides, S.T., Snijders, P.C., Lupini, A.R., ACS Nano 12 (6), 5873 (2018).CrossRefGoogle Scholar
Dyck, O., Kim, S., Kalinin, S.V., Jesse, S., Appl. Phys. Lett. 111, 113104 (2017).CrossRefGoogle Scholar
Susi, T., Kotakoski, J., Kepaptsoglou, D., Mangler, C., Lovejoy, T.C., Krivanek, O.L., Zan, R., Bangert, U., Ayala, P., Meyer, J.C., Ramasse, Q., Phys. Rev. Lett. 113, 115501 (2014).CrossRefGoogle Scholar
Susi, T., Meyer, J.C., Kotakoski, J., Ultramicroscopy 180, 163 (2017).CrossRefGoogle Scholar
Dyck, O., Kim, S., Jimenez-Izal, E., Alexandrova, A.N., Kalinin, S.V., Jesse, S., “Assembling Di- and Multiatomic Si Clusters in Graphene via Electron Beam Manipulation,” preprint, arXiv:1710.09416 (2018).Google Scholar
Crommie, M.F., Lutz, C.P., Eigler, D.M., Science 262, 218 (1993).CrossRefGoogle Scholar
Crommie, M.F., Lutz, C.P., Eigler, D.M., Nature 363, 524 (1993).CrossRefGoogle Scholar
Eigler, D.M., Lutz, C.P., Rudge, W.E., Nature 352, 600 (1991).CrossRefGoogle Scholar
Eigler, D.M., Schweizer, E.K., Nature 344, 524 (1990).CrossRefGoogle Scholar
Dyck, O., Ziatdinov, M., Lingerfelt, D.B., Unocic, R.R., Hudak, B.M., Lupini, A.R., Jesse, S., Kalinin, S.V., Nat. Rev. Mater. (2019), doi:10.1038/s41578-019-0118-z.Google Scholar
Pennycook, S.J., MRS Bull . 40 (1), 71 (2015).CrossRefGoogle Scholar
Feynman, R.P., Eng. Sci. 23 (5), 22 (1960).Google Scholar
You have Access

Send article to Kindle

To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

A self-driving microscope and the Atomic Forge
Available formats

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

A self-driving microscope and the Atomic Forge
Available formats

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

A self-driving microscope and the Atomic Forge
Available formats

Reply to: Submit a response

Please enter your response.

Your details

Please enter a valid email address.

Conflicting interests

Do you have any conflicting interests? *