The SubMIT project team has recently undertaken a significant project by migrating from a Gluster-based to a Ceph-based file system. This move marks a critical step in ensuring that the SubMIt system can handle the increasing demands of research and data processing in the coming years. Previously, the system had data from over 800 SubMIT users stored in Gluster, a file system known for its straightforward approach to managing data across multiple servers. However, as the number of SubMIT users and the volume of data storage has grown, the team recognized the need for a more scalable and resilient solution. Enter Ceph, a cutting-edge storage platform, capable of managing vast amounts of data while ensuring high availability and fault tolerance.
The migration was no small feat and required months of planning and coordination. Over several phases, over 300 TB of data has been transferred from Gluster to Ceph, testing each step to ensure that all files were securely moved and that the new system was optimized for performance. This transition will provide the research community with a more robust and flexible storage foundation, supporting ongoing and future projects. Thanks to the team’s efforts, the SubMIT project is now better equipped to meet the evolving needs of its users, paving the way for more physics projects at MIT.