Characterizing Effective Compute, Memory, and Communication Performance on Cerebras WSE-2

Abstract

The WSE-2 chip, integrated into the Cerebras CS-2 system, is drawing attention for its potential in scientific applications. The WSE-2 has distinctive features that set it apart from other computers, including a large number of small processing elements (PEs), a low-latency two-dimensional (2-D) mesh topology, and a unique distributed memory architecture. By understanding this architecture and optimizing applications for it, scientific applications can be accelerated significantly. However, its optimization opportunities, effective performance, and characteristics are still not fully understood. In this study, we characterize the effective performance of compute, memory, and communication using microbenchmarks. Our approach consists of three main aspects. First, we present effective computational and memory performance, as well as the roofline model. Second, we measure the performance of inter-PE communication including PE-to-PE communication latency, bisection bandwidth, and the latency of four collective communication operations. Third, we develop an explainable performance model based on the benchmark results. Experimental results show that the single-precision floating-point computation performance is 1.06TFlops/s, the maximum memory bandwidth is 8.80PB/s, and the average PE-to-PE communication latency is 1.2 cycles. Our characterization provides deeper insights into WSE-2 and offers guidance for optimizing applications on this platform.