Maxim Goncharov

SiteReadiness: 100.0% SAM tests: 98.8% No significant problems over last week. Our focus was on repairing worker nodes and XRootD servers.

SiteReadiness: 100.0% SAM tests: 99.6% No significant problems over last week. While investigating XRootD/network problems last week we discovered that our disk end point is bombarded by calls that one would see for the tape end point, like requests to stage files.

We had problems with XRootD SAM probes sporadically failing over the period of 4 days. At the same time we observed significantly increased incoming network traffic. We do not fully understand the root cause of a problem, investigating.

SiteReadiness: 100.0% SAM tests: 91.5% We upgraded one of our CEs to Alma9, and in the process of upgrading the second CE. Condor for some time remained in unbalanced state that affected average number of jobs running, as well as SAM test efficiency.

SiteReadiness: 100.0% SAM tests: 92.6% We continue upgrading worker nodes to Alma9. Condor seems to be unbalanced at the moment; jobs sometimes go into held state; SAM jobs periodically do not get scheduled promptly leading to the SAM test inefficiency.

SiteReadiness: 100.0% SAM tests: 93.8% Quiet last seven days. One of our CEs is down as we are upgrading it to Alma9 and modern Condor version.

SiteReadiness: 85.7% SAM tests: 89.8% We had a downtime on Tuesday July 14. Host facility was upgrading electrical infrastructure. The cluster was brought offline at 6 am (ET), including storage and administrative servers. All services were fully back online at around 9 pm.

SiteReadiness: 100.0% SAM tests: 99.3% No significant problems over last week. We continue working with CMS submission infrastructure team in order to keep our cores fully busy. For various reasons we continue missing ~2.5K cores, some are down because of power outages, some are in the process of Alma9 installation, and some need OS disk … Read more

SiteReadiness: 100.0% SAM tests: 100.0% No significant problems over last week. We reverted back to individual host certificates, as a result CMS restriction on certain types of production at our site was lifted. CERN factory started to require token support, as a result one of our CEs lost capability to schedule pilots from CERN; we … Read more

SiteReadiness: 100.0% SAM tests: 98.5% No significant problems over last week. For various reasons we are temporarily missing ~2.5K cores, some are down because of power outages, some are in the process of Alma9 installation, and some have problematic Alma9 setup. Over the last week we see CMS under-utilizing our site by 2K-3K cores.