Death, taxes and rack power density

Insights AI

While not as certain as death and taxes, there are signs that high-density racks will finally become more commonplace thanks to AI and other compute intensive workloads.

It feels like we are living in increasingly uncertain times. Recent political, social and economic upheaval seems to be proof that the old adage that ‘nothing is certain but death and taxes’ still rings true. But if we just consider the world of data centers, at one time there was also a good argument for including a third certainty: rack power densities would continue to rise.

There has been a long-held belief that data center rack power densities would increase steeply over time. Moore’s law, insatiable demands for compute and the general march of progress towards bigger and more powerful systems would all contribute to more power hungry racks of IT.

Industry insiders believed that average rack densities would increase from the 2 to 3kW per rack seen five to ten years ago to 20kW or even 50kW by now. But that isn’t actually the case. Only 4% of respondents to Uptime Institute’s 2017 industry survey reported an average rack power density greater than 12 kW per rack. The majority – around 67% of respondents – reported average power densities below 6kW per rack.

Speaking on a recent webcast, Uptime’s senior director of content & publications Matt Stansberry said that the long-expected wholesale uptake in density has failed to materialise. “For a number of years Uptime Institute conducted a very intensive study of server power densities. We were responding to industry fears that we were going to be looking at 25kW per rack, that people needed to be planning for it but our surveys put that number below 3kW,” he said. “Obviously there are going to be pockets of high-density but for the most part it is just not there and we see that still today. The majority of respondents are below 6kW per rack.”

There are a number of reasons for the lower than expected average rack power density:

  • The industry’s relentless focus on efficiency over the last decade is one. Crudely speaking this has had the effect of increasing the average compute per watt and keeping the power consumed per rack in check.
  • A conservative approach to data center design has also been a factor. Operators have historically kept to relatively low rack power designs, and over-specified cooling, to avoid hot spots or other risks to availability. It’s better to waste floor space than risk cooling problems or a lack of power.
  • Another factor is virtualisation that has the potential - at least in theory but not always in practice - to increase utilisation. As research group IHS Markit has pointed out, “Server virtualisation allows users to run servers at higher capacity, for example 80% instead of 20%. Instead of adding additional servers, users are now running current servers closer to full capacity which generally results in less power draw than adding new servers to perform the same amount of computing.”
  • Hyperscale operators have also had an impact. Large cloud services providers have been responsible for a significant proportion of new capacity over the last decade but many have built low-density but large square footage sites often in areas with low real-estate costs.
  • Data center technology suppliers such as Schneider Electric also argue that while lower densities are inefficient, higher-densities may actually be less cost-effective past 8 to 11 kW/rack. “Deploying the IT equipment at lower densities is much more costly per watt, and deploying higher provides minimal return for the added design complexities injected,” the company states.

But while the wholesale uptake in densities hasn’t happened as predicted, that doesn’t mean that there aren’t more high-density sites out there...

On the contrary, the anecdotal evidence is that densities will increase but only where it makes sense in terms of supporting specific intensive applications and workloads. Or as IHS describes it: “While average rack densities still remain below 5kW worldwide, densities of up to almost 50 kW have been implemented in applications such as HPC and supercomputing. It’s no longer uncommon to see rack densities of 20-30 kW in some applications.”

As we have described in previous blogs, the proliferation of machine learning and other AI workloads will likely lead to high-density racks based around GPUs and other relevant hardware.

Industry experts such as AMD’s senior VP and General Manager Enterprise, embedded, and semi-custom business group Forrest Norrod recently told me that he also expects to see increasing demand for higher-density systems. “You think about a 1U slice with a 1P EPYC (AMD’s new server processor), up to 2TB of RAM if you need it, and four accelerator cards…you can cram over 90 GPUs into a rack along with 40 EPYC processors, you can easily get over 50kVA, ” he said in a recent interview.

So rather than a wholesale move to high-density predicted a decade ago, it looks increasingly likely that power density will vary based on workload and location. In regions where real estate is cheap – such as rural hyperscale sites - it will still make sense to build out less dense deployments. But the need for increasing compute capacity at the edge – perhaps in urban locations - will drive more power dense configurations along with AI and other compute intensive applications.

We are also likely to see a mixture of power densities within the same site, or data hall, that could necessitate the use of different forms of cooling such as direct liquid cooling. “DLC makes it easier to have one set of racks at 5kW and another set at 60kW. It’s very hard to have your air conditioning go up and down to support that. Rear-door heat exchangers might help with some of that but there’s a limit to that too,” Steve Harrington, CTO at DLC supplier Chilldyne told me recently.

So it seems then that power dense racks will become more ubiquitous over time, that increase won’t be evenly deployed across the data center population. Sadly, in an increasingly unequal and polarised world, the same also seems to hold true for death and taxes.

Related blogs

AI Icelandic Style – Deep Learning in Reykjavik

Last week I was privileged to be part of our AI and HPC Field Trip to Iceland. The goal of the trip was to share insight and observations around the evolution of AI deep neural network (DNN) training and to tour our HPC-optimised data center. The attendees spanned DNN veterans like Eric Sigler of OpenAI, large enterprise data science leaders like Pardeep Bassi of Liverpool Victoria Insurance (LVE) and start-up pioneers like Max Gerer of e-bot7.

Read more

HPC and AI collusion – Rumours from the trade show floor

Recently I’ve garnered much of my blog inspiration from industry events. February has been no exception and I benefited from a fascinating day earlier in the month at the HPC and Big Data conference in London, just a stone’s throw from the Houses of Parliament. Here are some of my observations from my discussions there...

Read more

Talking Ai with London Cab Drivers

Being a daytime Londoner one of my favourite pastimes is to take an occasional ride in a proper London black cab and have a natter along the way. That is often a pretty cathartic experience - probably more so for the driver than the passenger, but I find it entertaining being the listening post, and London cab drivers are a good measure of the mood of the city. Recently I've had some great conversations about Ai and it's impact on society.

Read more

We use cookies to ensure we give you the best experience on our website, to analyse our website traffic, and to understand where our visitors are coming from. By browsing our website, you consent to our use of cookies and other tracking technologies. Read our Privacy Policy for more information.