Skip to main content Accessibility help
×
Home
Hostname: page-component-78dcdb465f-8p2q5 Total loading time: 0.408 Render date: 2021-04-17T23:31:34.012Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": false, "newCiteModal": false, "newCitedByModal": true }

Implementation of 2D Domain Decomposition in the UCAN Gyrokinetic Particle-in-Cell Code and Resulting Performance of UCAN2

Published online by Cambridge University Press:  15 January 2016

Jean-Noel G. Leboeuf
Affiliation:
Department of Physics, University of Alaska, Fairbanks, AK 99775-5920, USA
Viktor K. Decyk
Affiliation:
Department of Physics and Astronomy, and Institute for Digital Research and Education (IDRE), University of California, Los Angeles, CA 90095-1547, USA
David E. Newman
Affiliation:
Department of Physics, University of Alaska, Fairbanks, AK 99775-5920, USA
Raul Sanchez
Affiliation:
Departamento de Fsica, Universidad Carlos III, Leganes 28911, Madrid, Spain
Get access

Abstract

The massively parallel, nonlinear, three-dimensional (3D), toroidal, electrostatic, gyrokinetic, particle-in-cell (PIC), Cartesian geometry UCAN code, with particle ions and adiabatic electrons, has been successfully exercised to identify non-diffusive transport characteristics in present day tokamak discharges. The limitation in applying UCAN to larger scale discharges is the 1D domain decomposition in the toroidal (or z-) direction for massively parallel implementation using MPI which has restricted the calculations to a few hundred ion Larmor radii or gyroradii per plasma minor radius. To exceed these sizes, we have implemented 2D domain decomposition in UCAN with the addition of the y-direction to the processor mix. This has been facilitated by use of relevant components in the P2LIB library of field and particle management routines developed for UCLA's UPIC Framework of conventional PIC codes. The gyro-averaging specific to gyrokinetic codes is simplified by the use of replicated arrays for efficient charge accumulation and force deposition. The 2D domain-decomposed UCAN2 code reproduces the original 1D domain nonlinear results within round-off. Benchmarks of UCAN2 on the Cray XC30 Edison at NERSC demonstrate ideal scaling when problem size is increased along with processor number up to the largest power of 2 available, namely 131,072 processors. These particle weak scaling benchmarks also indicate that the 1 nanosecond per particle per time step and 1 TFlops barriers are easily broken by UCAN2 with 1 billion particles or more and 2000 or more processors.

Type
Research Article
Copyright
Copyright © Global-Science Press 2016 

Access options

Get access to the full version of this content by using one of the access options below.

References

[1]Brizard, A. J., and Hahm, T. S., Foundations of nonlinear gyrokinetic theory, Reviews of Modern Physics, 79 (2007), 421468.CrossRefGoogle Scholar
[2]Dimits, A. M., Bateman, G., Beer, M. A., Cohen, B. I., Dorland, W., Hammett, G. W., Kim, C., Kinsey, J. E., Kotschenreuther, M., Kritz, A. H., Lao, L. L., Mandrekas, J., Nevins, W. M., Parker, S. E., Redd, A. J., Shumaker, D. E., Sydora, R., and Weiland, J., Comparisons and physics basis of tokamak transport models and turbulence simulations, Physics of Plasmas, 7 (2000), 969983.CrossRefGoogle Scholar
[3]Parker, S. E., Lee, W. W., and Santoro, R. A., Gyrokinetic simulation of ion temperature gradient driven turbulence in 3D toroidal geometry, Physical Review Letters, 71 (1993), 20422045.CrossRefGoogle ScholarPubMed
[4]Sydora, R. D., Toroidal gyrokinetic particle simulations of core fluctuations and transport, Physica Scripta, 52 (1995), 474480.CrossRefGoogle Scholar
[5]Sydora, R. D., Decyk, V. K., and Dawson, J. M., Fluctuation-induced heat transport results from a large global 3D toroidal particle simulation model, Plasma Physics and Controlled Fusion, 38 (1996), A281A294.CrossRefGoogle Scholar
[6]Lin, Z., Hahm, T. S., Lee, W. W., Tang, W. M., and White, R. B., Turbulent transport reduction by zonal flows: Massively parallel simulations, Science, 281 (1998), 18351837.CrossRefGoogle ScholarPubMed
[7]Idomura, Y., Tokuda, S., and Kishimoto, Y., Global gyrokinetic simulation of ion temperature gradient driven turbulence in plasmas using a canonical Maxwellian distribution, Nuclear Fusion, 43 (2003), 234243.CrossRefGoogle Scholar
[8]Jolliet, S., Bottino, A., Angelino, P., Hatzky, R., Tran, T. M., Mcmillan, B. F., Sauter, O., Ap-pert, K., Idomura, Y., and Villard, L., A global collisionless PIC code in magnetic coordinates, Computer Physics Communications, 177 (2007), 409425.CrossRefGoogle Scholar
[9]Parker, S. E., and Lee, W. W., A fully nonlinear characteristic method for gyrokinetic simulation, Physics of Fluids B, 5 (1993), 7786.CrossRefGoogle Scholar
[10]Dawson, J. M., Particle simulation of plasmas, Reviews of Modern Physics, 55 (1983), 403448.CrossRefGoogle Scholar
[11]Kniep, J. C., Gyrokinetic Particle-in-Cell Simulations of Strong Poloidal Flow Radial Gradient and Parallel Nonlinearity Effects on Toroidal Ion Temperature Gradient Driven Turbulence, Ph. D. Thesis, University of California, Los Angeles, CA, 2006.Google Scholar
[12]Norton, C. D., Decyk, V. K., Szymanski, B. K., and Gardner, H., The transition and adoption of modern programming concepts for scientific computing in Fortran, Scientific Programming, 15 (2007), 2744.CrossRefGoogle Scholar
[13]Decyk, V. K., and Norton, C. D., UCLA parallel PIC framework, Computer Physics Communications, 164 (2004), 8085.CrossRefGoogle Scholar
[14]Decyk, V. K., UPIC: A framework for massively parallel particle-in-cell codes, Computer Physics Communications, 177 (2007), 9597.CrossRefGoogle Scholar
[15]Lee, W. W., Gyrokinetic particle simulation model, Journal of Computational Physics, 72 (1987), 243269.CrossRefGoogle Scholar
[16]Leboeuf, J. N., Dawson, J. M., Decyk, V. K., Kissick, M. W., Rhodes, T. L., and Sydora, R. D., Effect of externally imposed and self-generated flows on turbulence and magnetohydrodynamic activity in tokamak plasmas, Physics of Plasmas, 7 (2000), 17951801.CrossRefGoogle Scholar
[17]Rettig, C. L., Staebler, G. M., Rhodes, T. L., Leboeuf, J. N., Peebles, W. A., Doyle, E. J., Burrell, K. H., and Moyer, R. A., Search for the ion temperature gradient mode in a tokamak plasma and comparison with theoretical predictions, Physics of Plasmas, 8 (2001), 22322237.CrossRefGoogle Scholar
[18]Rhodes, T. L., Leboeuf, J.-N., Sydora, R. D., Groebner, R. J., Doyle, E. J., McKee, G. R., Peebles, W. A., Rettig, C. L., Zeng, L., and Wang, G., Comparison of turbulence measurements from DIII-D low-mode and high-performance plasmas to turbulence simulations and models, Physics of Plasmas, 9 (2002), 21412148.CrossRefGoogle Scholar
[19]Kniep, J. C., Leboeuf, J.-N., and Decyk, V. K., Gyrokinetic particle-in-cell calculations of ion temperature gradient driven turbulence with parallel nonlinearity and strong flow corrections, Computer Physics Communications, 164 (2004), 98102.CrossRefGoogle Scholar
[20]Sanchez, R., Newman, D. E., Leboeuf, J.-N., Decyk, V. K., and Carreras, B. A., Nature of transport across sheared zonal flows in electrostatic ion-temperature-gradient gyrokinetic plasma turbulence, Physical Review Letters, 101 (2008), 205002.CrossRefGoogle ScholarPubMed
[21]Sanchez, R., Newman, D. E., Leboeuf, J.-N., Carreras, B. A., and V. K. Decyk, ., On the nature of radial transport across sheared zonal flows in ion-temperature-gradient gyrokinetic tokamak plasma turbulence, Physics of Plasmas, 16 (2009), 055905.CrossRefGoogle Scholar
[22]Sanchez, R., Newman, D. E., Leboeuf, J.-N., and Decyk, V. K., On the nature of turbulent transport across sheared zonal flows: insights from gyro-kinetic simulations, Plasma Physics and Controlled Fusion, 53 (2011), 074018.CrossRefGoogle Scholar
[23]Hahm, T. S., Nonlinear gyrokinetic equations for turbulence in core transport barriers, Physics of Plasmas, 3 (1996), 4658-4664.CrossRefGoogle Scholar
[24]Naitou, H., Sonoda, T., Tokuda, S., and Decyk, V. K., Parallelization of gyrokinetic particle code and its application to internal kink mode simulation, Journal of Plasma and Fusion Research, 72 (1996), 259269.Google Scholar
[25]Kim, C. C. and Parker, S. E., Massively parallel three-dimensional toroidal gyrokinetic flux-tube turbulence simulation, Journal of Computational Physics, 161 (2000), 589604.CrossRefGoogle Scholar
[26]Hatzky, R., Domain cloning for a particle-in-cell (PIC) code on a cluster of symmetric-multiprocessor (SMP) computers, Parallel Computing, 32 (2006), 325330.CrossRefGoogle Scholar
[27]Madduri, K., Williams, S., Ethier, S., Oliker, L., Shalf, J., Strohmaier, E., Yelick, K., Memory-efficient optimization of gyrokinetic particle-to-grid interpolation for multicore processors, Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis (SC ’09), IEEE Computer Society, Los Alamitos, CA, 2009, pp. 112. http://www.computer.org/csdl/proceedings/sc/2009/8744/00/87440048-abs.htmlGoogle Scholar
[28]Adams, M. F., Ethier, S., and Wichmann, N., Performance of particle in cell methods on highly concurrent computational architectures, Journal of Physics: Conference Series, 78 (2007), 012001.Google Scholar
[29]Wang, B., Ethier, S., Tang, W., Williams, T., Ibrahim, K., Madduri, K., Williams, S., and Oliker, L., Kinetic turbulence simulations at extreme scale on leadership-class systems, Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis (SC ’13), Association for Computing Machinery, New York, NY, USA, 2013, Article No. 82. http://dl.acm.org/citation.cfm?doid=2503210.2503258Google Scholar
[30]Oliker, L., Canning, A., Carter, J., Shalf, J., and Ethier, S., Scientific application performance on leading scalar and vector supercomputering platforms, International Journal of High Performance Computing Applications, 22 (2008), 520.CrossRefGoogle Scholar
[31]Decyk, V. K., and Singh, T. V., Particle-in-cell algorithms for emerging computer architectures, Computer Physics Communications, 185 (2014), 708719.CrossRefGoogle Scholar

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 0
Total number of PDF views: 28 *
View data table for this chart

* Views captured on Cambridge Core between September 2016 - 17th April 2021. This data will be updated every 24 hours.

Send article to Kindle

To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Implementation of 2D Domain Decomposition in the UCAN Gyrokinetic Particle-in-Cell Code and Resulting Performance of UCAN2
Available formats
×

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Implementation of 2D Domain Decomposition in the UCAN Gyrokinetic Particle-in-Cell Code and Resulting Performance of UCAN2
Available formats
×

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Implementation of 2D Domain Decomposition in the UCAN Gyrokinetic Particle-in-Cell Code and Resulting Performance of UCAN2
Available formats
×
×

Reply to: Submit a response


Your details


Conflicting interests

Do you have any conflicting interests? *