The MIT Tier-2 was established in 2006 and after some moving about it landed in 2009 at the newly inaugurated High Performance Research Computing Facility (HPRCF) at Bates, Middleton, MA. Its fundamental design was based on a massive Cisco switch with each worker node connected at 1 Gb/sec and each worker node provided compute and storage. The storage technology was based on Hadoop (HDFS) and it worked very well for a long time. Every year 10-30 servers were added making sure there were enough SATA disk slots to fill with the most cost effective hard drives. Servers were not optimized for density so the center grew substantially with time. It should be added that while the racks are water cooled and reasonably efficient, they only support about 12 kW of power drawn which intrinsically limits the servers that can be installed. Eventually the Tier-2 consisted of 740 servers which occupied 44 of the total 70 racks that the HPRCF provides.
In recent years the Tier-2 center became more susceptible to the frequent power issues at Bates—recovery took substantially longer in large part due to the implementation of erasure coding and the limited bandwidth per machine—and the size and power consumption of the facility has come under scrutiny. Therefore we started a major project to redesign the Tier-2 such that it would be more resilient to power failures, have a smaller footprint and use substantially less power, while providing the same computing resources to the various experiments involved. This set the stage for a detailed project with the MIT Department of Facilities brokered through the ORCD office which is summarized in the following.
The ORCD office was a strong partner supporting the Tier-2 community which is hosted in the Bates Engineering center of the Laboratory for Nuclear Science
The goal of this pilot project is to demonstrate that replacing outdated computing servers with a smaller number of modern, energy-efficient servers can be revenue neutral, meaning that the capital investment for new servers will be offset by energy cost savings.
Project Overview
- The pilot aimed to replace 13 racks (270 oldest servers) with 11 modern servers—4 compute servers and 7 for data storage.
- This consolidation is expected to save 85 kW of average power used, with the project becoming revenue neutral within 1.5 years after implementation.
- The estimated annual carbon savings are 284 metric tons, equivalent to about 21,300 eight-pound bags of charcoal.
Data and Estimation
- Power usage data for the Bates HPCRF data center, where the CMS cluster is housed, was somewhat limited but the Power Usage Effectiveness (PUE)—the ratio of total facility power to computing power—is estimated to be about 1.8 due to incomplete historical and per-server data.
- Despite these limitations, the analysis strongly supports the financial and environmental case for modernization.
Key Outcome
As of the end of 2024, the entire new hardware was installed, the data from the obsolete hardware transferred, and the old servers were decommissioned and disposed of. As shown in the picture rack space at the HPRCF, in particular eight racks in row one and five more racks in row two, has just been released for other use cases.
For the power consumption we used the electrical reading of the servers. The new servers were added on 14 Dec 2024 increasing the average power used by ≈5 kW from 200 kW to 205 kW total. In July 2025, all relevant data had been transferred and the obsolete hardware was switched off and decommissioned. After some testing and reconfiguration we reached a new stable operating condition with a reduced average power consumption of 140 kW. This means that the power usage of the computing servers went down by 60 kW (=30%) from 200 kW, while providing the same amount of storage and computing resources. Depending on the PUE the effect on the total power usage of the computing center will have to be adjusted. Assuming a PUE of 1.8 the saved power would be 108 kW ( 1.8 * 60 kW) well beyond the planned savings of 85 kW. Any PUE larger than 1.4 provides at least the planned power savings.
This energy-saving renovation pilot serves as a test case for sustainable, cost-effective computing upgrades within MIT’s research infrastructure and it worked well within the expected parameters. We recommend other heavy computing resource users to adapt such hardware review to optimize their use of real estate and electrical power.