Accelerating Spark Performance on AMD EPYC Processors for a US based Fortune 100 company

This case study explores how we successfully addressed Spark performance bottlenecks for a client using AMD EPYC processors. The client's Service Level Agreements (SLAs) were not being met, prompting the need for Spark acceleration.

Initial Approach:

• Guten-Velox Accelerators: We initially implemented Guten-Velox accelerators, which improved performance to some extent.

Challenge

• Underutilized AVX Registers: Analysis revealed that the AVX512 registers on the AMD EPYC processors were not being utilized by the Guten-Velox solution compiled with GCC. AVX512 is a set of instructions that can significantly improve performance on compatible processors.

Solution

• AOCC and Code Optimization: We recompiled the code using the AMD Optimized C Compiler (AOCC) to take advantage of the AVX512 instruction set. Additionally, we made code modifications to address incompatibilities and ensure proper AVX512 support.

• Benchmarking and Validation: After recompilation and code optimization, we conducted thorough performance benchmarking. The results showed significant performance improvement compared to the initial Guten-Velox solution compiled with GCC.

Result

By leveraging the AVX512 capabilities of AMD EPYC processors through AOCC and code optimization, we were able to achieve substantial performance gains for our client, enabling them to meet their SLAs.

Benefits/Impact

• Optimizing Spark for specific processor architectures can yield significant performance improvements.
• AVX512 instructions on AMD EPYC processors offer substantial performance benefits when utilized effectively.
• Using an optimized compiler like AOCC can unlock the potential of these processors.

This case study serves as a valuable example for organizations seeking to optimize Spark performance on AMD EPYC-based platforms.