Weekly

SiteReadiness: 100% SAM tests: 92.2% Mostly smooth operations over the last week. Almost finished with pushing data (what can easily be copied) into CephFS. However, there will be a couple of hundred thousand files missing, mostly in /store/mc, because I suspect those files are only available on tape and my tools are very inefficient transferring … Read more

SiteReadiness: 100% SAM tests: 95.4% We had smooth operations over the last week. We wiped out previous CephFS deployment and bootstrapped a new cluster last Tuesday. While doing it, SAM tests turned red for several hours while new file system was unavailable; as a result SiteRediness for that day was in red, and our site … Read more

SiteReadiness: 57.1%; SAM tests: 74.4% Last week we tried to optimize our new CephFS file system. This ended up in an unexpected CephFS behavior which resulted the file system to put itself in read-only mode. We tried For several days to understand how to fix the problem and repair CephFS. At the end of this … Read more

SiteReadiness: 71.4% SAM tests: 91.8% We were adding for the first time more disks into one of the storage nodes. As a result of inexperience with CepFS internals one of the disks reached full state, and CephFS put itself into read only state. This affected our availability for two days. I asked Rucio team to … Read more

SiteReadiness: 100% SAM tests: 97.3% Over the last week we had a problems with some tests. The largest inefficiency comes from the HammerCloud tests. On Tuesday we restarted xrootd after adjusting read permissions for general CMS users. We had a report of some files in CephFS corrupted. At the moment we think that the corruption … Read more

No outstanding issues over the last week. SiteReadiness: 100%SAM tests: 99.3%