DAQ – Storage and Transfer System

The CMS experiment performs its online event selection using two trigger levels: the level-one trigger, implemented in custom electronics, which selects approximately 100 kHz of events; and the high-level trigger (HLT), running on a farm of commercial computer nodes. The HLT processes fully assembled events, applying algorithms similar to those used in offline reconstruction, and selects a few kHz of events for storage on disk.

The Storage and Transfer System (STS) is the last step in the CMS data acquisition system. STS collects the events selected by the HLT algorithms and writes them either to a distributed file system, that acts as a buffer between the online DAQ system and the offline world where the data is processed, or to fast local storage. The data is sent to the EOS storage facility at CERN for long-term storage and processing by Tier-0 by the transfers service, designed and implemented by our team. Two constraints of the system are to allow data taking to continue even if the connectivity to Tier-0 at CERN is temporarily lost or degraded, and to be able to temporarily write data at rates larger than the available bandwidth to Tier-0.

One of the major software tasks and development work is related to the managing of a very large number of files, and merging them to a single file per luminosity section and stream. Fully merged files, residing in the cluster file system, are handled and transferred to their offline destination by the transfer service. 

The current implementation for the last years of Run-3 is running two independent and concurrently operating data-flow paths. 
The 1st one is based on the Lustre global file system, installed on hardware provided by a commercial company, DDN. It has been operational since 2021, and the available space on the global file system for Run 3 is around 1.3 PB.
The 2nd one is based on 16 servers, each outfitted with 28 TB of usable local SSD space (2 × 14 TB disks configured in RAID0). They were added in 2024-2025 to meet the continuously increasing data-taking bandwidth requirements of Run 3, and the software was significantly updated to be able to operate both Lustre and SSD data-flow paths.

Our MIT team has designed all the software related to the file merger and transfer systems. We have also dealt with defining the characteristics and choosing the hardware that corresponds to the needs of the experiment. We operate and maintain that software and the storage hardware, and we provide 24 x 7 on-call support for these services. Our main work has been understanding, optimizing, maintaining, and continuously evaluating it.
Based on anticipated trigger rates and real-time system conditions, such as file system occupancy, the team dynamically selects and combines the Lustre and SSD-based data paths to optimize the balance between performance, robustness, and data integrity. During the 2025 heavy-ion data-taking period, the system routinely sustained rates of 37 GB/s, thereby enabling the CMS physics program to record the full dataset permitted by trigger constraints and physics objectives.

In the coming Long Shutdown 3, our team will be focused on designing and updating STS for the challenges that CMS is expected to face during the HL-LHC data-taking.