{"id":264,"date":"2025-05-07T12:59:35","date_gmt":"2025-05-07T12:59:35","guid":{"rendered":"https:\/\/ppc.mit.edu\/t2\/?p=264"},"modified":"2025-11-03T20:20:04","modified_gmt":"2025-11-03T20:20:04","slug":"mit-tier2-report-may-7-2025","status":"publish","type":"post","link":"https:\/\/ppc.mit.edu\/t2\/?p=264","title":{"rendered":"MIT Tier-2 Report – May 7 2025"},"content":{"rendered":"\n

SiteReadiness: 57.1%; SAM tests: 74.4%<\/p>\n\n\n\n

Last week we tried to optimize our new CephFS file system. This ended up in an unexpected CephFS behavior which resulted the file system to put itself in read-only mode.<\/p>\n\n\n\n

We tried For several days to understand how to fix the problem and repair CephFS. At the end of this week we came to the conclusion that the problem is probably not fixable. We attempt one suggestion that we found, however, as expected, it resulted in all metadata lost. <\/p>\n\n\n\n

At that moment on Tuesday I wiped out old system and bootstrapped a new CephFS installation, and started to re-transfer all data into it (at the moment \/store\/mc).<\/p>\n\n\n\n

So far I pushed ~150 TB in one day. If CephFS remains stable I will increase the rate. My estimate is that I should be able to push ~400 TB in one day.<\/p>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n\n
\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n\n
\n
\"\"<\/figure>\n<\/figure>\n","protected":false},"excerpt":{"rendered":"

SiteReadiness: 57.1%; SAM tests: 74.4% Last week we tried to optimize our new CephFS file system. This ended up in an unexpected CephFS behavior which resulted the file system to put itself in read-only mode. We tried For several days to understand how to fix the problem and repair CephFS. At the end of this … Read more<\/a><\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6],"tags":[12],"class_list":["post-264","post","type-post","status-publish","format-standard","hentry","category-report","tag-weekly"],"_links":{"self":[{"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/posts\/264","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=264"}],"version-history":[{"count":3,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/posts\/264\/revisions"}],"predecessor-version":[{"id":646,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=\/wp\/v2\/posts\/264\/revisions\/646"}],"wp:attachment":[{"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=264"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=264"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ppc.mit.edu\/t2\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=264"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}