Spotlight: Petrobras Speeds Up Linear Solvers for Reservoir Simulation Using NVIDIA Grace CPU

Arm-based processors such as the NVIDIA Grace CPU are gaining momentum in HPC applications, including in the energy industry. Petrobras, NVIDIA, and Brazil’s CESAR Innovation Center are partnering to port and benchmark SolverBR to the NVIDIA Grace CPU in an initiative to measure the main benefits of Arm-based CPUs.  

The NVIDIA Grace CPU has 72-core Arm Neoverse V2, connected by a high-bandwidth NVIDIA scalable coherency fabric and paired with high-bandwidth and low-power double data rate 5x (LPDDR5X) memory. 

Initial results demonstrated that NVIDIA Grace delivers best-in-class performance ratios across time-to-solution (TTS) and estimated energy-to-solution (ETS), compared to x86-based flagship processors available on-premises and in the cloud. 

The project focuses on maintaining a multiplatform build system, enabling a single codebase and compilation scripts to work seamlessly across various platforms. This ensures consistent testing and reliable performance comparisons when porting the x86 codebase to Arm with minimal required effort. The Arm ecosystem robust open-source compilers and debuggers have significantly facilitated this transition. 

GCC 12.3 was chosen as the test compiler due to its excellent performance. GCC 12.3 or above is recommended due to its tuning support for the Arm Neoverse V2 core. The porting to Arm process involved a few simple steps:  

1. Removal of all x86 architecture-specific flags, such as -maxv, -march, and -mtune, which have different meanings in Arm. 
2. Use the -O3 optimization level to trigger optimization steps such as function inlining, vectorization, loop unrolling, interchange, and fusion. For even more performance, consider -Ofast. 
3. Append the -mcpu=native to CFLAGS to ensure the compiler auto-detects the build system’s CPU.
4. Finally, use -flto as an optional flag for link-time optimization. Depending on the application, -fsigned-char or -funsigned-char may also be required. 

Minimal effort was taken to address compilation errors by replacing Intel Intrinsics functions with Arm-specific functions using the header-only library sse2neon. Runtime errors were fixed, including memory synchronization issues caused by specific compiler optimizations, which resulted in instruction reordering and subsequent floating-point precision divergencies. 

For this initial experiment, Petrobras used a fixed set of compilation flags for each architecture (x86_64 and aarch64) without implementing processor-specific tuning. The goal is to comprehend the performance behavior out-of-the-box. Table 1 illustrates the compilation flags employed. 

 Measuring performance and energy efficiency 

Singularity containers were employed to replicate the SolverBR computing stack across various platforms to ensure reproducibility. A single definition file was utilized to generate multiple runtime containers, resulting in unique .sif files, one for each CPU architecture. Table 2 specifies all the tested CPUs, both x86-based and Arm-based, on-premises and in the cloud. NVIDIA conducted the experiments on the Grace architecture, while Petrobras and CESAR executed the benchmarks on the remaining architectures. 

*Intel Xeon Gold 6248 is the main on-premises CPU cluster at Petrobras Research Center (CENPES) 
**NVIDIA Grace CPU data computed on a single NVIDIA Grace SoC using the GH200 Superchip platform

Testing the linear portion of the reservoir simulation pipeline shown in Figure 1 involved extracting the CMG-generated sparse matrices from pre-salt oil and gas field datasets (Búzios, Proxy 100, Proxy 200, Sapinhoá) and the SPE10 benchmark model. The example establishes how CMG simulators can be modularized and extended with third-party components or software. CMG engineers are currently exploring additional opportunities to port components and capabilities of their simulators, aiming to optimize performance across various platforms and hardware architectures.

Figure 2 displays the linear methods configurations: Adaptive Implicit Method (AIM), Krylov Subspace Projection Generalized Minimal Residual (KSPGMRES), Constrained Pressure Residual (CPR), Domain Decomposition (DD), Incomplete LU Factorization (ILU). Each system of equations was solved 50 times for all models across 3 years, resulting in thousands of executions. 

Figure 3 presents the speedup results for each model, utilizing the maximum available cores in single-socket processors. Petrobras currently uses the Intel Xeon Gold 6248 platform in its on-premises production CPU cluster, which serves as the reference point for results normalization. 

The NVIDIA Grace architecture demonstrates superior performance, achieving the highest performance ratios across all models, including: 

• Up to 4.5x speedup over Intel Xeon Gold 6248 (Petrobras baseline CPU) 
• Up to 2.9x speedup over Intel Xeon Platinum 8488C 
• Up to 1.9x speedup over AMD EPYC 9R14 

Figure 4 shows leading scalability for the NVIDIA Grace CPU when varying the total number of cores up to the maximum for each single-socket processor, up to 53% over the least scalable option. The NVIDIA Grace CPU demonstrates exceptional performance for this specific workload due to its unique characteristics, which include high effective memory bandwidth, an advanced CPU Scalable Coherent Fabric (NVIDIA SCF), and the adoption of server-class Arm Neoverse V2 cores. 

To estimate energy efficiency for the NVIDIA Grace CPU compared to on-premises CPUs, the maximum thermal design power (TDP) of each CPU was evaluated at full load with an estimated memory consumption based on processor capacity and technology generation. For the NVIDIA Grace CPU SoC, CPU memory consumption was approximately 250 W, and the speedups were reported using the AMD EPYC 9R14 as the baseline. 

NVIDIA Grace demonstrated the highest estimated energy efficiency across all linear solver tests at maximum load, with up to 4.3x higher energy efficiency (Figure 5).

Conclusion