Exact Extract — Huawei Training Material, Pages 151 and 152
"Absolute xPU Idle Time Increases with Cluster Scale as Over Half of Al Training Is Spent on Dataset Loading, CKPT Read/Write, and Fault Recovery"
"Training sets are massive small files and CKPTs are large files, requiring the storage system to deliver both high OPS and bandwidth."
"Low cluster utilization due to idle xPUs and ineffective training"
"Effective training time with GPUs < 50% of the training process"
"Periodic saving of TB- to PB-level checkpoint files challenges the write bandwidth of the storage system for large I/Os."
Technical Deep Dive:
Large-model training depends on continuous coordination between computing resources, storage systems, and high-speed networks.
Even when a computing cluster contains thousands of powerful NPUs or GPUs, effective computing utilization can remain low if accelerators spend significant time waiting for data.
Option B is correct in the training scenario described. Loading massive datasets and performing checkpoint read/write operations can create substantial I/O bottlenecks.
Training datasets may contain billions of small files, while model checkpoints can contain terabytes or even petabytes of data in large distributed training environments.
If storage bandwidth or I/O processing performance is insufficient, computing devices cannot maintain continuous execution of training operations.
Similarly, slow checkpoint saving interrupts useful computation, and prolonged checkpoint loading delays recovery after failures.
Huawei addresses these bottlenecks through high-performance AI storage, parallel data access, data-path optimization, and checkpoint acceleration.
OceanStor A Series storage provides high-bandwidth and high-IOPS capabilities intended to minimize the time spent loading training data and restoring checkpoints.
Option A is not the intended answer because insufficient peak hardware performance does not by itself explain low utilization.
Option C can also cause low utilization , especially when distributed training is limited by collective communication performance.
However, the specific training example emphasizes storage-related inefficiencies.
Option D is not a general primary cause of low accelerator utilization.
Therefore, B is the intended answer. The word primary is context-dependent and should not be interpreted as a universal diagnosis for every AI cluster.