There were arcs seen on the electrical switch gear that required a shutdown. The power outage at Bates started on Monday morning (11\/03\/2025) at around 10:30 a.m. All worker nodes went down, the UPS row including our CephFS storage and the networking switch stayed online. At 1:30 p.m. the problem was resolved by the Middleton electric company and we received the all clear and brought all services back online.<\/p>\n\n\n\n
The HDFS storage, which is based on the worker nodes, has many corrupted files at the moment because 16 disk nodes did not come back online. We are working on recovering the struggling disk nodes.<\/p>\n","protected":false},"excerpt":{"rendered":"
There were arcs seen on the electrical switch gear that required a shutdown. The power outage at Bates started on Monday morning (11\/03\/2025) at around 10:30 a.m. All worker nodes went down, the UPS row including our CephFS storage and the networking switch stayed online. At 1:30 p.m. the problem was resolved by the Middleton … Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":649,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[11,6],"tags":[],"class_list":["post-622","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-outage","category-report"],"_links":{"self":[{"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/posts\/622","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=622"}],"version-history":[{"count":8,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/posts\/622\/revisions"}],"predecessor-version":[{"id":657,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/posts\/622\/revisions\/657"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/media\/649"}],"wp:attachment":[{"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=622"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=622"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=622"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}