At the end of March (18-22), CMS held a major internal workshop (Computing and Offline Week) dedicated to issues related to data processing, distribution, and analysis. This workshop occurs twice a year and brings together experts from various laboratories, computing centers, and universities to discuss issues that directly affect the physics results produced by the collaboration. With tens of petabytes of data collected each year and a comparable amount of data simulated, it is not possible to produce any physics results without reliable computing.
Among the various important topics discussed during the week, one received special attention: analysis facilities and infrastructures focused on creating an optimal environment for physicists producing final physics results for publications worldwide. MIT is one of the most active players in developing an analysis facility that should help physicists from different experiments analyze their data and in particular to be able to deal with the much increased computing requirements during the high luminosity phase of the LHC (HL-LHC). The amount of data is expected to grow by about a factor of ten in volume but also in complexity. To succeed, one needs a strong and experienced team of physicists familiar with the most recent developments in the areas of analysis frameworks and software algorithms, as well as computing hardware and its operations.
SubMIT is one of the four US-CMS pilot facilities that is already available to all members of the MIT Physics Department. The project is led by M. D’Alfonso, J. Bendavid, and C. Paus. subMIT allows its users to access resources directly via SSH as well as using JupyterHub. It provides about a thousand CPU slots and more than 50 GPUs accessible interactively and through Slurm, with optional access to OSG, CMS Tier-3 and Tier-2, LQCD Cluster, and EAPS computing resources.
At the meeting, multiple ideas were discussed on how to provide the most user-friendly access to resources, ranging from interactive resource provisioning to access to data and distributed computing processing. It was clearly established that CMS needs analysis facilities that provide capabilities beyond the general-purpose workload management system provided by the CMS GlideinWMS system based on HTCondor. Among different technologies, JupyterHub, Dask, and Kubernetes look the most promising to become the core tools for future analysis facilities. Although it is still at an early stage, which will require more research and development, the outline of the future analysis facilities is emerging from the many options.