SiteReadiness: 85.7% SAM tests: 91.9% Over the last week we started to have problems with the worker nodes (WNs). Some WNs loose reliable connectivity to the outside world (ping starts showing packet loss), some go offline, and some need to be rebooted. I am not sure what is exactly the reason behind it – CMS … Read more

SiteReadiness: 100.0% SAM tests: 98.7% No significant issues over last week. I switched CRAB area (/store/temp) from HDFS to CephFS, preparing to switch /store/unmerged, and with that all CMS areas will be in CephFS.

SiteReadiness: 100.0% SAM tests: 99.0% No significant issues over last week. It looks like the storage is synchronized with Rucio. There were no FTS logs indicating that we miss some files or the files have incorrect checksum. Also no such new reports from users came. CephFS storage is sitting at ~74%, I am considering switching … Read more

SiteReadiness: 71.4% SAM tests: 100.0% The only issue over the last week, like in the previous week, was FTS errors. Together with DM team we took care of all files that were lost, and of all files that had incorrect check sum. I am monitoring FTS logs to catch cases when a transfer fails because … Read more

SiteReadiness: 71.4% SAM tests: 99.9% The only issue over the last week is FTS errors. Data Management team transferred all missing files. The remaining item was to take care of files that only existed at our site and were lost. They should have been invalidated globally, but instead DM team invalidated them only partially, as … Read more

SiteReadiness: 57.1% SAM tests: 90.3% I switched our site configuration on May 22. However, changes were propagated to /cvmfs only on May 27. For several days there was a difference between global and local site configuration. I am not sure if this is the reason behind our site in red for two days in site … Read more

SiteReadiness: 100% SAM tests: 92.2% Mostly smooth operations over the last week. Almost finished with pushing data (what can easily be copied) into CephFS. However, there will be a couple of hundred thousand files missing, mostly in /store/mc, because I suspect those files are only available on tape and my tools are very inefficient transferring … Read more

SiteReadiness: 100% SAM tests: 95.4% We had smooth operations over the last week. We wiped out previous CephFS deployment and bootstrapped a new cluster last Tuesday. While doing it, SAM tests turned red for several hours while new file system was unavailable; as a result SiteRediness for that day was in red, and our site … Read more

SiteReadiness: 57.1%; SAM tests: 74.4% Last week we tried to optimize our new CephFS file system. This ended up in an unexpected CephFS behavior which resulted the file system to put itself in read-only mode. We tried For several days to understand how to fix the problem and repair CephFS. At the end of this … Read more

SiteReadiness: 71.4% SAM tests: 91.8% We were adding for the first time more disks into one of the storage nodes. As a result of inexperience with CepFS internals one of the disks reached full state, and CephFS put itself into read only state. This affected our availability for two days. I asked Rucio team to … Read more