The x86 rack server space is very crowded, but is still possible to rise above the crowd. Quite a few data centers have many "gaping holes" in the racks as they have exceeded the power or cooling capacity of the data center and it is no longer possible to add servers. One way to distinguish your server from the masses is to create a very low power server. The x86 rack server market is also very cost sensitive, so any innovation that seriously cuts the costs of buying and managing the server will draw some attention. This low power, cost sensitive part of the market does not get nearly the attention it deserves compared to the high performance servers, but it is a huge market. According to AMD, sales of their low power (HE and EE) Opterons account for up to 25% of their total server CPU sales, while the performance oriented SE parts only amount to 5% or less. Granted, AMD's presence in the performance oriented market is not that strong right now, but it is a fact that low power servers are getting more popular by the day.The low power market is very diverse. The people in the "cloudy" data centers are - with good reason - completely power obsessed as increasing the size of a data center is a very costly affair, to be avoided at all costs. These people tend to almost automatically buy servers with low power CPUs. Then there is the large group of people, probably working in the Small and Medium Enterprise businesses (SMEs) who know they have many applications where performance is not the first priority. These people want to fill their hired rack space without paying a premium to the hosting provider for extra current. It used to be rather simple: give heavy applications the (high performance) server they need and go for the simplest, smallest, cheapest, and lowest power server for applications that peak at 15% CPU like fileservers and domain controllers. Virtualization made the server choices a lot more interesting: more performance per server does not necessarily go to waste; it can result in having to buy fewer servers, so prepare to face some interesting choices.
Do you go for a server that tries to minimize power and rack space by using low power CPUs and a server form-factor such as the Twin or Blade server? In that case, you may end up with more servers than originally planned. Alternatively, do you use standard 1U/2U servers with standard CPUs? In that case, you may miss the chance to lower your monthly collocation and/or energy bill. While we won't be able to give you a perfectly tailored answer, we'll give you some of the fundamental information that you need to make an educated decision. In this article we will measure how much power low power CPUs, such as the Opteron 2377 EE and the Xeon L5520, save compared to their "normal" siblings. We will use our own virtualization benchmarks to make it a bit more realistic than SPECpowerjbb. Maybe the most important question: is the performance/watt ratio of a more expensive, low power server CPU really better?
Focusing on the CPUs alone would be a missed chance. Whether you need 10 or 8 servers to consolidate your applications does not depend solely on the CPU power, but also on the amount of memory you can place inside your server, the amount of expansion slots, and so on. In addition, power consumption does not depend solely on the CPU but also on how clever the server engineers design the server chassis and power supply. We assembled four different servers from three different manufacturers. Every server represents a different take on how to reduce power and CAPEX costs. Let us see how much sense it makes to invest in low power servers with low power CPUs in a virtualized environment.
Does that mean this article is only for server administrators and CIOs? Well, we feel that the hardware enthusiasts will find some interesting info too. We will test seven different CPUs, so this article will complement our six-core Opteron "Istanbul" and quad-core Xeon "Nehalem" reviews. How do lower end Intel "Nehalem" Xeons compare with the high end quad-core Opterons? What's the difference between a lower clocked six-core and a highly clocked quad-core? How much processing power do you have to trade when moving from a 95W TDP Xeon to a 60W TDP chip? What happens when moving from a 75W ACP (105W TDP) six-core Opteron to a 40W ACP (55W TDP) quad-core Opteron? These questions are not the ultimate goal of this article, but it should shed some light on these topics for the interested.
Making Sense of AMD ACP/TDP and Intel TDP
The only thing clear about the TDP numbers of AMD and Intel is that they are confusing and not comparable. It is clear that no power consumption or thermal design point number is going to make much sense to the server buyer unless the method of determining power consumption (or dissipation) is precisely defined by an independent third party. From that point of view, AMD Average CPU Power (ACP) only blurs the picture, even though it offers interesting information to those who are well informed about its purpose.
The idea behind ACP is good: the TDP number is for system builders and the people who design heatsinks, so you need another metric for the data center people. ACP should "reflect the CPU power consumption running typical data center workloads" according to AMD. That sounds like a reasonable goal, but we are less enthusiastic about the methodology. ACP is the geometric mean of what a CPU consumes running TPC-C, SPECcpu2006, SPECjbb2005, and Stream. Stream hardly stresses the CPU at all (it focuses on stressing memory bandwidth) and a geometric mean is always lower than an arithmetic mean. It seems to us that when it comes to power consumption it is better to use a slightly higher estimate than to introduce clever tricks to lower the number. Moreover, if you are running floating-point applications you should expect much higher power draw numbers than ACP suggests.
TDP or Thermal Design Power is defined by AMD as (from AMD Family 10h Server and Workstation Processor Power and Thermal Data Sheet, Rev 3.04 - June 2009):
"The maximum power a processor draws for a thermally significant period while running commercially useful software. The constraining conditions for TDP are specified in the notes in the thermal and power tables."
That seems very similar to Intel's TDP, but in practice AMD's TDP is (almost) equal to the maximum electrical power a CPU can draw (current times voltage). Therefore, AMD's own definition is not accurate for the published TDP results. The irony is that the published Intel TDP numbers are more accurately described by AMD's definition. Intel's engineers measure the power draw of hundreds of commercially available software packages and ignore the "not thermally significant" peaks. All those power measurements are averaged and a small percentage (a buffer) is added. Thus, Intel's TDP is lower than the maximum power draw.
In a nutshell:
- AMD's ACP uses a "round down" average of power measurements performed with industry standard benchmarks (usually running at 100% CPU load, with the exception of Stream).
- AMD's TDP is close to the electrical maximum a CPU can draw (when it is operating at its maximum voltage).
- Intel's TDP is a "round up" average of power measurements of processor intensive benchmarks.
If AMD would apply the methodology of Intel to determine TDP they would end up somewhere between ACP and the current "AMD TDP". "There is no substitute for your own power measurements" is a correct but an incredibly lame and unrealistic conclusion. Few IT professionals are going to perform power measurements on all the potentially interesting CPUs systematically. It would be nice if the SPEC committee would step up and determine a realistic power measurement for both integer and floating point loads based upon several industry standard benchmarks (with only basic tuning allowed) and not a flawed SPECjbb based one. Until then, this report should offer you a second opinion. It is worth noting that SPEC has launched their new SPECweb2009, which includes a SPECpower methodology. This is certainly a step in the right direction as SPECweb2009 is a lot more realistic than SPECjbb.
After so much theory, it is time to go hands-on again. Let's meet the server hardware we gathered for this test.
ASUS RS700D-E6/PS8
With the ASUS RS700 Duo, ASUS goes after Supermicro's popular Twin Server. Just like the Twin, ASUS has placed two servers in a 1U rack server. Unlike the Twin, each server has its own power supply. The advantage is that a PSU failure only takes down one server as opposed to two servers in the Supermicro Twin. However, Supermicro's Twin design with only one central power supply also has the advantage that the system never works at a very low load, which improves efficiency as power supplies are not very efficient at the lowest loads. In addition, in HPC clusters it does not matter whether one or two servers go down for a short period, as calculations are spread over a relatively large cluster. ASUS also feels virtualization is a key market, but we would like to have a fully redundant 1+1 PSU setup that feeds both server nodes for that kind of system.
We would add the ASUS server to our short list for HPC and load balanced web server applications. Virtualization, database, and other backend applications will require a redundant PSU.
With the exception of the non-redundant power supplies, ASUS made some very good choices. Just like the single node version, the ASUS RS700-E6 that we used for our Xeon Nehalem review, the server comes with an excellent 6-phase power CPU regulation and 3-phase power regulation for the DIMMs. We say excellent as ASUS uses high quality (5+ years at 86°C) solid capacitors. According to ASUS, this 6+3 phase power design has no trouble feeding 130W TDP CPU (the Xeon W5580 for example) and is thus more than capable of powering our 95W TDP CPUs.
ASUS also went for the 24D instead of the 36D version of the Tylersburg chipset, also known as the 5500 chipset. This should shave a few watts off the total power consumption. ASUS' experience in designing notebooks is also applied in their servers; notice the clever heatpipe design to cool the chipset and other smaller chips.
Overall, the ASUS Duo server is promising, but we miss the very clean and cableless design of the single node ASUS RS700-E6/RS4 server.
Chenbro/Intel Willowbrook
Intel sent us the Chenbro RM13204 server, based on their "Willowbrook" or "S5500WB" motherboard. This is a server board tuned for low power. Power consumption is optimized in several ways:
- Optimized voltage regulator design for "Nehalem" Xeons up to 80W TDP. Intel claims that 95W Xeons should work too, but the server would not even start with our Xeon X5570.
- "Spread core design": CPUs and memory are all placed in parallel to enable lower fan speeds. Typically, memory is placed after the CPU and system fans must blow harder to cool the DIMMs.
- A special low power Intel 82576EB replaces the slightly older "Intel 82575 EB" dual Gigabit Ethernet controller
- Only four DIMM slots per CPU instead of six or nine slots
That last power "saver" seems like a bad idea for the virtualization market, as it means that a dual CPU configuration is limited to 32GB. With expensive 8GB DIMMs, you can still reach 64GB. However, it seems like the "spread core" design leaves no space for extra DIMMs.
The Chenbro RM13204 is equipped with an 80+ Silver 450W Power Supply and four hot swappable 3.5" SATA drive bays. The Power supply is 90 to 91% efficient according to this report.
Lower Powered AMD Server: the A+ Server 1021M-UR
AMD does not make boards for commercial use and thus turns wisely towards a third party, in this case Supermicro. The H8DMU+, still based on the NVIDIA's MCP55Pro, is hardly optimized for low power. It is clear that the AMD platform really needs a refresh with some new modern HT3 (HyperTransport 3) capable low power chipsets. In a few weeks, the Fiorano platform should launch, but that does not mean the AMD based low power server does not have a chance. For example, the board offers no less than 16 DIMM slots and the server supports two redundant power supplies. Those two features alone make this server a lot more attractive to the virtualization crowd than the low power Chenbro + Willowbrook combination.
The server also allows one low profile and two full height expansion slots, which is rather exceptional for a 1U server. One of those two full height slots is Supermicro's proprietary UIO slot. The server is fed by highly efficient 650W power supplies in a redundant 1+1 configuration.
The Supermicro Twin2
This is the most innovative server of this review. Supermicro places four servers in a 2U chassis and feeds them with two redundant 1200W PSUs. The engineers at Supermicro have thus been able to combine very high density with redundancy - no easy feat. Older Twin servers were only attractive to the HPC world were computing density and affordable prices were the primary criteria. Thanks to the PSU redundancy, the Twin2 should provide better serviceability and appeal to people looking for a web, database, or virtualization server.
Most versions of this chassis support hot swappable server nodes, which makes the Twin2 a sort of mini-blade. Sure, you don't have the integrated networking and KVM of a blade, but on the flip side this thing does not come with TCO increasing yearly software licenses and the obligatory expensive support contracts.
By powering four nodes with a 1+1 PSU, Supermicro is able to offer redundancy and at the same time can make sure that the PSU always runs at a decent load, thus providing better efficiency. According to Supermicro, the 1200W power supplies can reach up to 93% efficiency. This is confirmed by the fact that the power supply is certified by the Electric Power Research Institute as an "80+ Gold" PSU with a 92.4% power efficiency at 50% load and 91.2% at 20% load. With four nodes powered, it is very likely that the PSU will normally run between these percentages. Power consumption is further reduced by using only four giant 80mm fans. Unfortunately, and this is a real oversight by Supermicro, as the fans are not easy to unplug and replace. We want hot swappable fans.
Supermicro managed to squeeze two CPUs and 12 DIMM slots on the tiny boards, which means that you can outfit each node with 48GB of relatively cheap 4GB DIMMs. Another board has a Mellanox Infiniband controller and connector onboard, and both QDR and DDR Infiniband are available. To top it off, Supermicro has chosen the Matrox G200W as a 2D card, which is good for those who still access their servers directly via KVM. Supermicro did make a few compromises: you cannot use Xeons with a TDP higher than 95W (who needs those 130W monsters anyway?), 8GB DIMMs seem to be supported only on a few SKUs right now, and there is only one low profile PCI-e x16 expansion slot.
The Twin2 chassis can be outfitted with boards that support Intel "Nehalem Xeons" as well as AMD "Istanbul Opterons". The "Istanbul version" came out while we were typing this and was thus not included in this review
Feature Comparison
To save you some time, we made a quick overview table of the features of each server in this test.
| Low Power Server Feature Comparison | ||||
| | Supermicro Twin2 | ASUS RS700D-E6/PS8 "Duo" | Chenbro RM130204 "Willowbrook" | Supermicro A+ Server 1021M-UR |
| Computing Nodes | 4 | 2 | 1 | 1 |
| Form Factor | 2U | 1U | 1U | 1U |
| CPU per Node | Dual Intel Xeon "Nehalem" | Dual Intel Xeon "Nehalem" | Dual Intel Xeon "Nehalem" | Dual AMD Shanghai or Istanbul |
| Highest TDP (CPU) | 95W per CPU | 130W per CPU | 80W per CPU | 137W per CPU |
| RAM per Node | 48/96GB | 48/96GB | 32/64GB | 64GB |
| PSU | 2x 1200W | 2x 770W | 450W | 650W |
| Redundant? | Yes | No | No | Yes |
| I/O Hub | Intel 5520 (36-D) | Intel 5500 (24-D) | Intel 5500 (24-D) | NVIDIA MCP55 Pro |
| Expansion Slots | One PCI-e x16 low profile | One PCI-e x16 low profile | One PCI-e x8 One PCI-e x4 | One PCI-e x8 full height One UIO (PCIe x8) |
| Storage per Node | 3x SATA | 4x SATA | 4x SATA | 4x SATA |
| HDD Form Factor | 3.5" | 2.5" | 3.5" | 3.5" |
| RAID Support | Intel ICH10R 0, 1, 5, 10 | LSI MegaRAID 0, 1, 10 | Intel ICH10R 0, 1, 10 | NVIDIA MCP55 0, 1, 10, 5 |
| Networking | Dual Intel 82576 Gigabit | Intel 82574L | Dual Intel 82576 Gigabit | NVIDIA MCP55 Dual Gigabit |
| Management | IPMI 2.0 KVM over LAN | Optional | Optional | SIMSO IPMI 2.0 |
| USB Front | none | 1 USB 2.0 | 1 USB 2.0 | none |
| USB Rear | 2x USB 2.0 per node | 2x USB 2.0 per node | 4x USB 2.0 | 2x USB 2.0 |
| Graphics | Matrox G200W | Aspeed AST2050 8MB | Server Engine LLC Pilot II Controller | ATI ES1000 16MB |
| Infiniband | DDR and QDR | N/A | Optional | N/A |
Notice that all these servers provide a rather modest storage system. In most applications, shared storage will be necessary to avoid a storage bottleneck and to provide the necessary redundancy.
Benchmark Configuration and Methodology
All our benchmarks are run on VMware ESX 4.0 build 164009. Database files are on one Intel X25-E SLC 32GB SSD, with log files on another Intel X25-E SLC 32GB. The AMD Opteron based machines had one extra DIMM (16GB instead of 12GB), but this did not influence the performance results. We also compensated in the power consumption numbers by subtracting the power consumption of one DIMM (5W at full load).
I'd also like to send out a big thanks to my colleague Tijl Deneut for his assistance with the power testing!
1U ASUS RS700D6-RS4 Duo
Dual Intel Xeon "Gainestown" X5570 2.93GHz, Dual Intel Xeon L5520 2.26GHz
Bios version June 15, 2009
12GB (3x4GB) 1066MHz Kingston KVR1066D3R4R7SK3
PSU: 2 times (one per node) 770W Delta Electronics DPS-770AB
1U Chenbro RM13204 Chassis
Dual Intel Xeon "Gainestown" X5570 2.93GHz, Dual Intel Xeon L5520 2.26GHz
Intel Willowbrook S5500WB motherboard, BIOS from June 3, 2009
12GB (3x4GB) 1066MHz Kingston KVR1066D3R4R7SK3
PSU: 450W YH-2451-0AA01R
2U Supermicro Twin2 (1U nodes)
Dual Intel Xeon "Gainestown" X5570 2.93GHz, Dual Intel Xeon L5520 2.26GHz
X8DDTT-IBXF, BIOS version R 1.0 c, June 15, 2009
12GB (3x4GB) 1066MHz Kingston KVR1066D3R4R7SK3
PSU: 2x1200W
Supermicro A+ Server 1021M-UR+V
Dual Opteron 2435 "Istanbul" 2.6GHz
Dual quad-core Opteron 2377 EE 2.3GHz
Dual Opteron 2384 2.7GHz
Dual Opteron 2389 2.9GHz
Supermicro H8DMU+, BIOS version June 18, 2009
16GB (4x4GB) 800MHz
PSU: 650W Cold Watt HE Power Solutions CWA2-0650-10-SM01-1
APUS and Oracle Calling Circle Client Configuration
Intel Core 2 Quad Q6600 2.4GHz
Foxconn P35AX-S
4GB (2x2GB) Kingston DDR2-667
NIC: Intel PRO/1000
Power was measured at the wall by two devices: the Extech 38081…
…and the Ingrasys iPoman II 1201. A big thanks to Roel De Frene of Triple S for letting us test this unit.
The Extech device allows us track power consumption each second, the iPoman only each minute. With the Supermicro Twin2, we wanted to measure four servers at once, and the iPoman device was handy for measuring power consumption of several server nodes at once. We double-checked power consumption readings with the Extech 38081.
Performance: Low Power CPUs Compared
The differences in performance are small, even insignificant, between the different servers. When it comes to comparing the servers, we'll focus on power consumption. When it comes to performance, we'll focus on the CPUs. We use our vApus Mark I benchmark as it has a mix of decision support (MS SQL Server), OLTP (Oracle), and web serving (PHP, IIS) on top of VMware's ESX 4.0. You can read a detailed description of our in-house developed benchmark here. We are working on an open source version to allow third parties to repeat our benchmarking.
We ran vApus mark I with one tile. Normally you'll see at least two tile (8 VMs) benchmarks in our server CPU tests unless we are limited by memory. As the amount of registered DDR3 memory available to us is at present 12 4GB DIMMs (48GB) and we had to fill a 4-server node Twin2 machine, we were limited to 12GB per node. Therefore, all testing is done with one tile or four VMs.
There's lots of interesting data here for the hardware loving professionals. It is the first time we have done such extensive testing on ESX 4.0, and we notice that while the Xeon X5570 and "Istanbul" Opteron 2435 are in line with our previous review, the quad-core "Shanghai" Opteron numbers are not (+6%). As in the previous review most of our attention was on the new AMD hex-core and its Xeon X5570 rival, so we did not double-check our Opteron 2384 numbers. If you compare them with our ESX 3.5 numbers, you'll see that quad-core Opteron is about 6% faster with ESX 4.0, while its six-core brother does not gain anything. As we use four vCPUs per VM, it is a lot harder for the ESX scheduler to do this efficiently on the six-core chip and that is most likely the reason why hex-core does not benefit from the ESX 3.5 to ESX 4.0 update.
Also interesting is that the Xeon "Nehalem" at 2.26GHz is capable of keeping up with the 2.9GHz Opteron "Shanghai". Remember, vAPUS Mark I is one of the benchmarks where AMD is at its best. In this best case for AMD, the Nehalem architecture is about 28% faster clock for clock. The Xeon E5504, the second slowest available Xeon "Nehalem", does not have Hyper-Threading (lose about 15%) or turbo boost (another ~5% loss) and only has 4MB of L3 cache. The result is that it is only 15% faster clock for clock than the Shanghai Opteron.
Also noteworthy is that one Xeon 2.93GHz performs as well as a dual Xeon at 2GHz and slightly better than a dual Opteron 2377 EE. If we tested with eight VMs, the dual machines would perform slightly better, while the single CPU system would stagnate. Still, that does not change the fact that a single socket server with a Xeon X5570 might be an interesting alternative to many dual socket servers if performance/watt matters.
Our performance conclusion is that the best Opteron EE is about as fast as the Xeon Nehalem at 2GHz. The L5520, Intel's lowest power "Nehalem" is as fast as the fastest 75W ACP Opteron at 2.9GHz. Next up are the power consumption numbers.
Power: Low Power CPUs Compared
First we'll start with the raw power consumption:
By dividing the performance scores on the previous page by the measured power consumption at full load, we get the performance per watt. We multiplied by 100 to make the results easier to read, so a score of 100 means that for every vApus Mark I performance point the system consumes one watt.
That the X5570 2.93GHz wins the performance/watt race came as a surprising… at first. However, remember that the "leakier" Nehalems go to the desktop as 130W TDP parts, while the server CPUs are the best parts (95W TDP). The top server Xeons are already binned for power consumption. In a way, the 2.93GHz Xeon is already a "lower power" CPU.
The "75W ACP Shanghai" based Opterons disappoint: the difference between them and the 95W TDP Xeon "Nehalem" is small. This is partly a result of the fact that the NVIDIA chipset based motherboards waste quite a bit more power than the Intel platform. Notice that a single Opteron 2435 consumes more than a Xeon X5570; once you add three DIMMs and a second CPU, the tables have turned. So adding a Xeon X5570 adds about 58W (248W - 175W - three DIMMs of 5W), while adding an Opteron 2435 2.6GHz adds about 47W (243 - 181 - three DIMMs of 5W). While the Opteron 2435 consumes less power than the X5570, an AMD based server still consumes slightly more than the Intel based server with one socket filled up. This clearly indicates that our AMD server platform is not as efficient as the Intel Server. This is something that AMD's upcoming high-efficiency Fiorano "Kroner" platform should address.
The Xeon X5570 2.93GHz only consumes a few watts more than the much slower quad-core Opterons. We have shown that in native situations, the Xeon X5570 is about 37% to 85% faster than a 2.7GHz quad-core Opteron or about 30% to 78% faster than a 2.9GHz Opteron. In our virtualization scenario, the X5570 is about 30% faster. The business case for the high-clocked quad-core Opteron looks very bleak. The six-core Opteron at 2.6GHz is stronger: 15% higher performance than the 2.9GHz quad-core and it consumes just as much. The Opteron EE performs well, although it fails to defeat the L5520 when it comes to "pure" performance/watt.
Power at full load is only part of the story of course. We are working on a throttled vApus workload, but a good look at the numbers from SPECpower_ssj2008 tells us that the power consumption at 60-90% is not radically different from 100%. That is not really surprising: 100% CPU load does not mean that the CPU is using twice the amount of decoding and executing resources than at 50%. Instead, 100% load means that the (ESX) scheduler never has to run an idle process. That happens 50% of the time when you are running at 50% load. Basically, you get the same resources stressed 50% or 100% of the time. You may expect that the power curve starting from the idle power value to the 100% load power value is more or less linear… if the clock speed and voltage is constant of course. By default, ESX does not change the clock speed (using SpeedStep or PowerNow!) as long as your load does not go under 60%. Whether we test at 60%, 80%, or 100%, the power consumption landscape should stay the same.
Some applications will always run at a relatively high CPU load, for example a web server that is read in Asia, Europe, and America. However, many applications are only used between 9AM and 6PM local time, so a large part of the time the machine might be running close to idle. Before we declare anybody the performance/watt king, we need to take the idle power numbers into account.
Idle Power in ESX: Challenging
Measuring idle power in ESX is easier said than done. ESX does not use SpeedStep or PowerNow! by default (similar to Windows and unlike Linux). You have to set the power.cpu policy to "dynamic" instead of the static default. That means that many virtualized data centers are consuming more than they need to as they are not using SpeedStep or PowerNow! Getting PowerNow! to work was pretty easy. SpeedStep was a different matter: only our Intel Willowbrook based server (Chenbro chassis) could enable it. We of course first enabled SpeedStep in the BIOS (EIST = enabled):
However, even after restarting the machine several times and setting power.cpupolicy to dynamic we ended up with this on both the ASUS and Supermicro machines:
Notice that the Power Management technology is not available. The Intel Willowbrook based server did not have this problem:
Both ASUS and Supermicro's engineers have started debugging this problem as we reported this, and you should see a solution for this problem very soon. The problem is described here by VMware and will be either be solved by an improved BIOS version or by ESX Update 1 which should available at the end of 2009.
* = SpeedStep was not supported.
Conventional wisdom from a few years back might make you expect that the lower clocked parts (like the Xeon E5504 80W) consume less power than the highest clocked parts (like Xeon X5570 at 2.93GHz), but that is far from the truth. In fact, the "basic Xeons" are the leakiest parts - the parts that Intel almost threw in the trash bin.
The idle numbers show that Intel's power gating techniques save quite a bit of energy. Notice how adding a second L5520 CPU and three DIMMs of DDR3-1066 to our Chenbro server only adds 9W. Add an Opteron EE to our AMD server and you add 22W. According to AMD's own numbers, the lowest idle power measured for the quad-core Opterons is about 13W, probably using a Windows Server version that is patched with the best power saving features available. A quick comparison shows that VMware's ESX is not that aggressive when it comes to saving power, so the idle CPU power is quite a bit higher. The result is that the best AMD platform consumes about 16W more than the best Intel platform when running idle.
Granted, a better AMD platform can lower that number, but it is clear that when it comes to idle power, the platform is not the only one to blame here. Intel's power gating is the decisive advantage here: it can turn the inactive cores completely off. Another indication is that the dual Opteron 2435 consumes about 156W when we turn off dynamic power management, which is higher than the Xeon X5570 (150W). On the flip side of the coin, AMD's "older" platform makes enabling PowerNow! easier in ESX 4.0.
Price Premiums for Low power CPUs
We know low power comes with a price premium, but how much? We compared the low power versions with a "normal power" model that should offer more or less the same performance. In the last column, we list which CPU you can get at the same price.
| AMD CPU Pricing | ||||
| AMD Opteron "Efficient" Model | Price | AMD 75W ACP Model | Price | Price Premium |
| 2377 EE 2.3 (40-55 W) | $698 | 2378 2.4 | $174 | $524 |
| 2373 EE 2.0 (40-55 W) | $377 | 2372 2.1 | $174 | $203 |
| 2425 HE 2.1 (six-core) (55-68W) | $523 | 2427 2.2 (six-core) | $455 | $68 |
| 2381 HE 2.5 (55-68W) | $455 | 2380 2.5 | $316 | $139 |
The HE parts are affordable; the price premium will be hard to notice in a fully configured server system. While we were writing this article, AMD introduced the new HE six-core Opteron parts. We'll update this article when we get the chance to test the low power hex-core CPUs.
On the other hand, the best power saving parts, the Extreme Efficiency parts, are quite expensive. For the price you have to pay for the best EE part (2.3GHz), you can get a six-core at 2.4GHz. In other words, for 35W less you have to sacrifice two cores and 100MHz clock speed. You can also trade in those 35W for about 26% more clock speed: the 2389 at 2.9GHz also costs $698. The Opteron EE is not a good deal for those looking for the best performance/watt per dollar. The reason for this high price is of course the fact that Intel does not have a comparable part:
| Intel CPU Pricing | ||||
| Intel Xeon Low Power Model | Price | Intel Xeon Model | Price | Price Premium |
| | | E5530 2.4 | $530 | |
| L5520 2.26 (60W) | $530 | E5520 2.26 | $373 | $157 |
| L5506 2.13 (60W) | $423 | E5506 2.13 | $266 | $157 |
Our measurements made it very clear that from a performance/watt point of view that the Xeon L5520 beats the Opteron EE. The point of the Opteron EE is of course the fact that the maximum power consumed is 16W lower. In the non-virtualized world, the Opteron EE may be an alternative to those on a quest for a very low power server who feel that Supermicro's Atom and Dell's VIA Nano based servers are too slow. It will probably also find a home in some custom made ultra dense server systems.
Personally, we would like to see these Opteron EEs become affordable for SMEs. Think of all the small business servers that hardly ever go beyond 10% load or the MySQL databases that are hit by lightly loaded websites and always hover between 10 and 50% load. Of course we are not blind to the business reasons why these parts are expensive: the typical "hyper scale" customer orders (ten) thousands of CPUs and are a very attractive market to cater for. The power obsessed SME market on the other hand is quite small.
Performance/Watt: Choosing the CPU
Let us make sense of the all the data we have seen so far. We start with the wretches and work our way up to the champions.
At the current pricing (July 2009), the Opteron 2384 through 2389 do not make any sense. You pay more than for a Xeon L5520 2.26GHz and you get - at best - the same performance while consuming 50W more running at full speed, and consuming 30W more in idle. AMD should really lower the pricing of the higher clocked Shanghais.
The "basic" range of Xeons (E5506, E5504, and E5502) does not make much sense either if you are looking for a good performance/watt ratio. These Xeons only support DDR3-800, QuickPath speed is lower (4.8GT/s instead of 5.86GT/s), and these CPUs have no support for Hyper-Threading and Turbo Boost. Combine that with the fact that these CPUs are "high leakage" parts and you'll understand that these CPUs should be avoided. Even Intel advises against buying these chips. Well, if you read between the lines….
Some like to portray the Xeon X55xx series as power guzzling CPUs. However, these are high performance CPUs that surprised us with modest power requirements. In virtualization environments, you will want to load up the machine with a lot more memory. The result is that the Xeon X5570 has the best performance/watt ratio of the industry, if you are looking for top performance and your application has little idle time (i.e. a 24/7 or worldwide server). The downside is that this is the most expensive two socket CPU. Is $1300 per CPU too much? Well, that depends on what you will be running on this CPU. Microsoft and Oracle licenses have this tendency to make even the most expensive CPUs look cheap by comparison.
The six-core Opterons make a lot more sense than their quad-core Opterons when you are looking for high performance and want to keep power in check. Just keep in mind that AMD's six-core is strong in two areas: decision support databases and virtualization. Our current virtualization test with 16 virtual CPUs even slows down the six-core a little; make sure the number of vCPUs can be divided by six and you will probably squeeze another 5% out of AMD's best Opteron. The six-core Opteron servers can also leverage the advantage of their cheaper infrastructure (DDR2, motherboard, already very mature platform) and are thus the choice of price sensitive buyers. The downside is that although the idle power consumption is excellent compared to the quad-core CPUs, it is still rather high compared to Intel platforms. We are looking forward to adding the six-core Opteron HE 2425HE 2.1GHz (55W ACP - 79 W TDP) to our tests, but maybe we should wait until AMD's low power platform comes available.
The Opteron EE is surely not for SMEs looking for a low power server: $700 per CPU is not what we would call a good deal. However, it performs decently and has a pretty low "full load" power consumption. If your application never runs idle and your data center is mostly power limited, the Opteron EE makes a strong case. In others words, the Opteron EE will mostly go to the "Facebooks, Netlogs, and Googles" of this world. They will get huge price reductions as they order in large volumes. The Achilles' heel of this low power CPU is the fact that it is not capable of shutting down cores completely and as a result consumes more than its archrival the L5520 when running idle.
If you are looking at the overall picture (price, performance/watt, and idle power), the Xeon L5520 2.26GHz is the best CPU of this test. Sure, it cannot beat its X5570 brother when it comes to "pure" performance/watt, but most applications run idle for long periods (nighttime for example). When running idle, the power consumption of the CPUs is as good as negligible and depends solely on the quality of your motherboard and PSU. Performance is also excellent, at (least at) the level of a quad-core Opteron at 2.9GHz. Only if performance is less important and your application is rarely running at idle might the slightly higher power consumption at full load sway the decision towards the Opteron EE.
Comparing the Servers
The interesting thing about the Supermicro Twin2 is that four servers share two PSUs. The PSUs are of course sized to run four nodes, so it would not be fair to compare power consumption with only one node running. Getting enough memory and CPUs was not easy, but we managed to gather 12 DIMMs and four Xeon X5570 chips. Our first test is with the Xeon L5520, so we did not include the Supermicro Twin2. The ASUS Duo server has one independent PSU for each node, so we only powered up one node.
The ASUS RS700D-E6/PS8 beats the platform that Intel uses as the low power reference design: kudos to the ASUS engineers are in order. The ASUS server is most likely one of the lowest consuming Xeon based servers on the planet. We compared this server to the Twin2 by doubling the power numbers we registered when running both nodes. We could not include the Chenbro server since it would not start with the Xeon X5570.
The ASUS is one of the most efficient servers we have ever measured, but the clever design of Supermicro gives the latter a small edge. Four servers running at full load consume only 171W per server node. We expect a fully loaded server with two X5570 and 24GB per node to consume about 259W (171W + 58W for the extra CPU + six DIMMs at 5W), which is still very low for such a high performance configuration. Let us check out idle power.
Having fewer PSUs helps of course. The Supermicro Twin2 is once again the winner with only 112W per server node.
Final Verdict: the Servers
Before we can draw conclusions, we need to look at the pricing of these barebones servers. We assume that hardware costs matter to you. In other words, we assume that you are not using expensive software like the enterprise versions of ESX and Oracle.
The European pricing is the end-user pricing as provided by Triple-S, a server and storage (BTO) vendor that is active in France, Belgium, and the Netherlands. The US pricing was gathered by looking at the pricewatch site. All prices exclude any taxes.
| Server Pricing Comparison | |||||
| Server Chassis | Comments | European pricing | US Price | ||
| Price (Euro) | Price per Server (Euro) | Price | Price per Server (US Dollar) | ||
| Supermicro Twin2 Nehalem | |||||
| SYS-6026TT-FF | No hot pluggable nodes, no Infiniband | € 2116 | € 529 | $2429 | $607 |
| SYS-6026TT-iBRF | Hot pluggable nodes, no Infiniband | € 2413 | € 603 | $2845 | $711 |
| SYS-6026TT-iBXF | No hot pluggable nodes, DDR Infiniband | € 3138 | € 785 | $3659 | $915 |
| SYS-6026TT-BIBXRF | Hot pluggable nodes, DDR Infiniband | € 3297 | € 824 | $4095 | $1024 |
| SYS-6026TT-BIBQF | Hot pluggable nodes, QDR Infiniband | € 3503 | € 876 | $4159 | $1040 |
| Supermicro Twin2 Istanbul | |||||
| PIO-2012TM-BTRF | | € 2117 | € 529 | N/A | |
| PIO-2012TM-BiBXRF | | € 3056 | € 764 | N/A | |
| ASUS | |||||
| ASUS RS700D6-E6/PS8 "Duo" | Dual node | € 1526 | € 763 | $1500 | $750 |
| ASUS RS700-E6/RS4 | Single node, cableless | € 1138 | € 1138 | $1150 | $1150 |
| Chenbro RM130204 "Willowbrook" | | € 700 | € 700 | $750 | $750 |
| Supermicro A+ Server 1021M-UR | | € 795 | € 795 | $998 | $998 |
The ASUS RS700D6-E6/PS8 or "ASUS Duo" offers very low power consumption and high quality power regulation. To really stand out, the ASUS Duo server should adopt the cableless design and the "easy to remove fans" of the ASUS RS700-E6/RS4. The latter is a bit more expensive but still one of the best x86 rack servers on the market. If you need more than one expansion slot and want optimum serviceability, the ASUS RS700-E6 is an excellent choice. The ASUS RS700D6-E6/PS8 is well suited for the HPC market but falls short in its other target market: virtualization. The two power supplies should be redundant and hot swappable. The Duo server should also be as easy to service as its single node brother.
The Chenbro RM130204 plus Intel Willowbrook comes with excellent ideas to lower power consumption, but the implementation can't compete with the best Supermicro and ASUS servers. Low power is nice, but as you can see in the results in this article, a clever design such as the dual PSUs in the Supermicro Twin2 that power four nodes can lower power even more. The Willowbrook based server is limited to only 32GB if you do not want to use expensive 8GB DIMMs and does not offer power supply redundancy. When it comes to virtualization, the Chenbro Willowbrook combination is rather mediocre. However, if you are looking to run a server version of Windows on this one, it is close to impossible to find a server that consumes so little power. Idle power can be as low as 70W if you use Windows 2003 SP2 and the hot patches that support the deep C-states of the latest Xeons. This server is almost inaudible when running under load and is whisper silent when running idle. The Chenbro RM130204 is a server you can afford to run outside the data center.
The Supermicro Twin2 offers very low power consumption and PSU redundancy at a very attractive price per node. This server is a serious alternative for all those who like the density of blade servers but don't want to pay the high prices and yearly software licenses. A typical blade server (with 10 blades) is about three times more expensive per server, and I/O modules (such as a gigabit Ethernet switches) easily cost twice as much as the stand-alone versions. The HPC people will be very pleased with the Infiniband solutions (the DDR version is able to come close to its maximum bandwidth). Supermicro reports that they already have several large-scale (2000+ servers) deployments.
The Twin2 is probably the best solution on the market if you are searching for a low cost, high density, and redundant server… but it is not perfect. Fans should be easier to remove and hot swappable. We would like to see support for 8GB DIMMs across all Twin2 versions, not just a few SKUs. We also want a version with 2.5" disks so we can use SSDs if necessary. Supermicro tells us that the Twin2 will be available with 2.5" disks later this year.
The Supermicro A+ Server 1021M-UR is AMD's current favorite for low power Opteron servers, but not for long. It is clear that the NVIDIA based platform is getting old, feature- and power-wise. We look forward to seeing the Opteron EE and especially the six-core HE in action on a "Fiorano Kroner" platform. Just like Intel's Willowbrook, the power regulation will be optimized for low power CPUs and will not support the more power hungry CPUs.
Conclusion
Intel's integrated power gating allows the newest Xeons to shutdown cores completely when running idle, and this works transparent to the software. Thanks to this technology and the generally excellent per clock performance, the Xeon L5520 is the best solution in most situations where performance/watt matters. The motherboard manufacturers are still struggling with the new platform as SpeedStep is not properly supported in the latest version of ESX. However, this is a temporary situation. Both ASUS and Supermicro will solve this problem in the coming weeks.
When it comes to low power, the AMD platform is getting a little old, but it is mature at the same time. We did not have any trouble enabling PowerNow! in ESX 4.0. AMD's CPUs lack the advanced power gating features that the newest Intel generation has and the result is that AMD CPUs consume up to 20% more running idle. That makes the AMD CPUs less attractive in applications where you run mostly in idle mode. AMD Opteron EE is the CPU that consumes the least running at full load, and this will make it interesting for power limited data centers. AMD's low power CPUs will become more attractive when the new power optimized Fiorano "Kroner" platform will arrive.
Posts
Thank you for sharing these remarkable info. I hope that people who are operating data centers are able to read your post to know how they can upgrade their system without blowing millions. Still, I believe that when costly equipment can give you the performance that you need, it is worth every penny. (Mac Pherson)
ReplyDelete