Over-temperature at HPCF
Incident Report for CU Boulder RC
Resolved
A drain in the HPCF cooling system was discovered incorrectly "stuck" open, which allowed the evaporative cooling tower to drain, impeding cooling efficiency. The problem has been corrected, and we have resumed normal operation on Summit compute and Blanca HPC resources.
Posted Sep 01, 2019 - 19:52 MDT
Investigating
We have been advised of higher than expected temperatures at the HPCF, which includes Summit and Blanca HPC nodes. We have configured those compute nodes to drain until we receive word that the HVAC system has been inspected and the issue resolved.

We are not killing any running jobs at this time, and we have not observed any thermal alerts from the nodes themselves yet.
Posted Sep 01, 2019 - 16:29 MDT
This incident affected: RMACC Summit and Blanca.