Skip to main content Accessibility help

Toward Cost-Effective Reservoir Simulation Solvers on GPUs

  • Zheng Li (a1), Shuhong Wu (a2), Jinchao Xu (a3) and Chensong Zhang (a4)


In this paper, we focus on graphical processing unit (GPU) and discuss how its architecture affects the choice of algorithm and implementation of fully-implicit petroleum reservoir simulation. In order to obtain satisfactory performance on new many-core architectures such as GPUs, the simulator developers must know a great deal on the specific hardware and spend a lot of time on fine tuning the code. Porting a large petroleum reservoir simulator to emerging hardware architectures is expensive and risky. We analyze major components of an in-house reservoir simulator and investigate how to port them to GPUs in a cost-effective way. Preliminary numerical experiments show that our GPU-based simulator is robust and effective. More importantly, these numerical results clearly identify the main bottlenecks to obtain ideal speedup on GPUs and possibly other many-core architectures.


Corresponding author

*Corresponding author. Email: (Z. Li), (S.Wu), (J. Xu), (C. Zhang)


Hide All
[1] Appleyard, J. R., Appleyard, J. D., Wakefield, M. A. and Desitter, A. L., Accelerating reservoir simulators using GPU technology, SPE Reservoir Simulation Symposium, 2011.
[2] Baker, A. H., Falgout, R. D., Kolev, T. V. and Yang, U. M., Multigrid smoothers for ultraparallel computing, SIAM J. Sci. Comput., 33(5) (2011), pp. 28642887.
[3] Bank, R. E., Chan, T. F., Coughran, W. M. Jr and Smith, R. K., The Alternate-Block-Factorization procedure for systems of partial differential equations, BIT Numer. Math., 29(4) (1989), pp. 938954.
[4] Bell, N., Dalton, S. and Olson, L. N., Exposing fine-grained parallelism in algebraic multigrid methods, SIAM J. Sci. Comput., 34(4) (2012), pp. C123C152.
[5] Bell, N. and Garland, M., Efficient sparse matrix-vector multiplication on CUDA, Technical report, Nvidia Technical Report NVR-2008-004, Nvidia Corporation, 2008.
[6] Braess, D., Towards algebraic multigrid for elliptic problems of second order, Computing, 55(4) (1995), pp. 379393.
[7] Brandt, A., Algebraic multigrid theory: the symmetric case, Appl. Math. Comput., 19(1) (1986), pp. 2356.
[8] Brandt, A., McCormick, S. and Ruge, J., Algebraic multigrid (amg) for automatic multigrid solutions with application to geodetic computations, Report, Inst. for Computational Studies, Fort Collins, Colo, 1982.
[9] Brandt, A., McCoruick, S. and Ruge, J., Algebraic multigrid (amg) for sparse matrix equations, Sparsity Appl., (1985), pp. 257284.
[10] Brannick, J., Chen, Y., Hu, X. and Zikatanov, L., Parallel unsmoothed aggregation algebraic multigrid algorithms on gpus, Numerical Solution of Partial Differential Equations: Theory, Algorithms and Their Applications, pages 81102, Springer, 2013.
[11] Byun, J.-H., Lin, R., Yelick, K. A. and Demmel, J., Autotuning sparse matrix-vector multiplication for multicore, Technical Report UCB/EECS-2012-215, EECS Department, University of California, Berkeley, November 2012.
[12] Cao, H., Tchelepi, H., Wallis, J. and Yardumian, H., Parallel scalable unstructured CPR-type linear solver for reservoir simulation, Paper SPE 96809 presented at the SPE Annual Technical Conference and Exhibition, Dallas, Texas, 9-12 October, 2005.
[13] Chen, Z., Huan, G. and Ma, Y., Computational Methods for Multiphase Flows in Porous Media, Volume 2, SIAM, 2006.
[14] Choi, J. W., Singh, A. and Vuduc, R. W., Model-driven autotuning of sparse matrix-vector multiply on GPUs, ACM SIGPLAN Notices, 45(5) (2010), pp. 115.
[15] Christie, M. and Blunt, M., Tenth SPE comparative solution project: A comparison of upscaling techniques, SPE Reservoir Evaluation & Engineering, 4(04) (2001), pp. 308317.
[16] Coats, K. H. et al., A note on IMPES and some IMPES-based simulation models, SPE J., 5(03) (2000), pp. 245251.
[18] Dang, H. V. and Schmidt, B., CUDA-enabled sparse matrix-vector multiplication on GPUs using atomic operations, Parallel Comput., 39(11) (2013), pp. 737750.
[19] Dogru, A. H., Fung, L. S. and Middya, U. et al., A next-generation parallel reservoir simulator for giant reservoirs, SPE/EAGE Reservoir Characterization & Simulation Conference, 2009.
[20] Douglas, J. Jr, Peaceman, D. and Rachford, H. Jr et al., A method for calculating multi-dimensional immiscible displacement, Trans. Amer. Inst. Min. Metallurgical Petroleum Eng., pages 297306, 1959.
[21] Esler, K., Mukundakrishnan, K., Natoli, V., Shumway, J., Zhang, Y. and Gilman, J., Realizing the potential of GPUs for reservoir simulation, ECMOR XIV-14th European Conference on the Mathematics of Oil Recovery, 2014.
[22] Falgout, R., An introduction to algebraic multigrid computing, Comput. Sci. Eng., 8(6) (2006).
[23] Feng, C., Multilevel Iterative Methods and Solvers for Reservoir Simulation on CPU-GPU Heterogenous Computers, PhD thesis, Xiangtan University, 2014.
[24] Fung, L. S., Sindi, M. O. and Dogru, A. H. et al., Multi-paradigm parallel acceleration for reservoir simulation, SPE Reservoir Simulation Symposium, 2013.
[25] Gandham, R., Esler, K. and Zhang, Y., A GPU accelerated aggregation algebraic multigrid method, Comput. Math. Appl., 68(10) (2014), pp. 11511160.
[26] Hayder, M. E. and Baddourah, M. et al., Challenges in high performance computing for reservoir simulation, Paper SPE, 152414 (2012), pp. 47.
[27] Hu, X., Vassilevski, P. S. and Xu, J., Comparative convergence analysis of nonlinear AMLI-cycle multigrid, SIAM J. Numer. Anal., 51(2) (2013), pp. 13491369.
[28] Kim, H., Xu, J. and Zikatanov, L., A multigrid method based on graph matching for convection–diffusion equations, Numer. Linear Algebra Appl., 10(1-2) (2003), pp. 181195.
[29] Klie, H. M., Sudan, H. H., Li, R. and Saad, Y. et al., Exploiting capabilities of many core platforms in reservoir simulation, SPE Reservoir Simulation Symposium, Society of Petroleum Engineers, 2011.
[30] Lacroix, S., Vassilevski, Y. V. and Wheeler, M. F., Decoupling preconditioners in the implicit parallel accurate reservoir simulator (IPARS), Numerical Linear Algebra Appl., 8(8) (2001), pp. 537549.
[31] Li, R. and Saad, Y., GPU-accelerated preconditioned iterative linear solvers, J. Supercomput., 63(2) (2013), pp. 443466.
[32] Liu, H., Yang, B. and Chen, Z., Accelerating algebraic multigrid solvers on NVIDIA GPUs, Comput. Math. Appl., 70(5) (2015), pp. 11621181.
[35] Napov, A. and Notay, Y., An algebraic multigrid method with guaranteed convergence rate, SIAM J. Sci. Comput., 34(2) (2012), pp. A1079A1109.
[36] Notay, Y., Flexible conjugate gradients, SIAM J. Sci. Comput., 22(4) (2000), pp. 14441460.
[37] Notay, Y., Aggregation-based algebraic multigrid for convection-diffusion equations, SIAM J. Sci. Comput., 2012.
[38] Pavlas, E. J. Jr et al., Fine-scale simulation of complex water encroachment in a large carbonate reservoir in saudi arabia, SPE Reservoir Evaluation & Engineering, 5(05) (2002), pp. 346354.
[39] Peaceman, D. W., Presentation of a horizontal well in numerical reservoir simulation, The 11th SPE Symposium on Reservoir Simulation, 1991.
[40] Saad, Y., Iterative methods for sparse linear systems, SIAM, 2003.
[41] Stüben, K., Algebraic Multigrid (AMG): an Introduction with Applications, GMD Forschungszentrum Informationstechnik, 1999.
[42] Sudan, H., Klie, H., Li, R. and Saad, Y., High performance manycore solvers for reservoir simulation, 12th European Conference on the Mathematics of Oil Recovery, 2010.
[44] Tchelepi, H. and Zhou, Y. et al., Multi-GPU parallelization of nested factorization for solving large linear systems, SPE Reservoir Simulation Symposium, Society of Petroleum Engineers, 2013.
[45] Trangenstein, J. A. and Bell, J. B., Mathematical structure of the black-oil model for petroleum reservoir simulation, SIAM J. Appl. Math., 49(3) (1989), pp. 749783.
[47] Vaněk, P., Brezina, M. and Mandel, J. et al., Convergence of algebraic multigrid based on smoothed aggregation, Numer. Math., 88(3) (2001), pp. 559579.
[48] Vaněk, P., Mandel, J. and Brezina, M., Algebraic multigrid by smoothed aggregation for second and fourth order elliptic problems, Computing, 196 (1996), pp. 179196.
[49] Wallis, J., Incomplete Gaussian elimination as a preconditioning for generalized conjugate gradient acceleration, Paper SPE 12265 presented at the SPE Reservoir Simulation Symposium, San Francisco, California, 15-18 November, 1983.
[50] Wallis, J., Kendall, R., Little, T. and Nolen, J., Constrained residual acceleration of conjugate residual methods, SPE, 13536 (1985), pp. 1013.
[51] Wang, L., Hu, X., Cohen, J. and Xu, J., A parallel auxiliary grid algebraic multigrid method for graphic processing units, SIAM J. Sci. Comput., 35(3) (2013), pp. C263C283.
[52] Wu, S., Feng, C., Zhang, C.-S., Li, Q. and Al, E., A multilevel preconditioner and its shared memory implementation for new generation reservoir simulator, Petroleum Science, (2014), pp. 118.
[53] Yu, S., Liu, H., Chen, Z. J., Hsieh, B. and Shao, L. et al., GPU-based parallel reservoir simulation for large-scale simulation problems, SPE Europec/EAGE Annual Conference, Society of Petroleum Engineers, 2012.


MSC classification

Related content

Powered by UNSILO

Toward Cost-Effective Reservoir Simulation Solvers on GPUs

  • Zheng Li (a1), Shuhong Wu (a2), Jinchao Xu (a3) and Chensong Zhang (a4)


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed.