Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Matt]

We're evaluating a new build process for Media Center that's a little slower but possibly more error resistant.

My work machine has a JRMark of about 5100.  It's an overclocked i7 2600k.  Everything is on SSDs (except media).

Is there anything available that would be considerably faster, even a Xeon / server machine?  ECC memory would be a plus.  

Water cooling or hard core overclocking would be a little too much.

My understanding is that an Ivy Bridge would only be worth 10-15%.

Is there anything coming down the pipe?

Thanks for any advice.

[leezer3]

Depends on how much you want to spend and how parallelised you build process is TBQH. You're probably about right with the Ivybridge figure, and that'll apply with most desktop chips.
I don't know what your budget is, but I'd be seriously considering a dual or quad CPU Xenon machine, so giving you either 8 or 16 cores to play with
That'd also start having implications for what version of Windows you can run mind, I *think* the standard versions are only licenced for a single CPU socket, hence you'd have to invest in a server OS.

Basically then depends on your budget; Speccing a fully decked out dual socket Xenon server would cost you somewhere in the region of $5,000, but assuming a nicely parallel build process you'd probably get a 40-60% boost.
Almost sounds like you need to hire a pro with the appropriate equipment to find your bottlenecks- No use adding more cores if something bottlenecks and holds up the whole build process.

-Leezer-

[MrC]

Compilations of large programs are almost always CPU-bound.

I remember changing our old Unix builds system.  It used to take hours to compile the kernel.  With parallel compilation units, it went down to about 4 minutes.  Blazing.

[Matt]

I agree it's mostly CPU.  Although SSDs help a lot because of the huge amount of intermediate data that gets written to disk.

The build process parallelizes pretty well.

It compiles 8 projects at a time for the ~110 projects of a build.

However, I think it might get to a point where it's spending a lot of time on one big static library (the main part of MC) with nothing else running.  If we broke that library up or at least made sure it started first, it might help.

It might also be interesting to run the 32-bit and 64-bit legs of the compile at the same time.  Is there any easy way with a batch file to get two things running at once and still catch the return code and handle it?  Or is that a job for a custom program at that point?

[glynor]

Is there anything coming down the pipe?

Thanks for any advice.

Wait for a few months after Haswell ships (June) and they'll ship the Ivy-based Xeons (the current ones are based on Sandy Bridge-E).

Do that, and get a dual CPU workstation.  Assuming the build process is sufficiently parallel (I think they usually are), that might provide some nice returns.  For consumer-space boards?  No, absolutely wait for Haswell, and consider the "high-end" line of CPUs if memory bandwidth matters.

[Matt]

No, absolutely wait for Haswell, and consider the "high-end" line of CPUs if memory bandwidth matters.

I've read Haswell will be around 10% faster per clock and clocked about the same speeds.  Have you seen anything more encouraging?

Will it overclock better or offer 6 core versions or something?

The power consumption stuff the've done is awesome, but it's not exciting from a straight-line speed perspective.

[glynor]

Right, but you're on Sandy, and Haswell is likely to be another 10-15% bump over Ivy.  So, then you're in 20-25% range of IPC improvement, which is substantial (particularly look at the reviews when they come out for certain workloads and see if they apply).  Also, if you're interested in a Workstation dual-cpu system, it just wouldn't make sense to buy now, when they're still selling Sandys there (and you already have one).  They're better Sandys, but still Sandys.  Of course, they do have six and eight core versions.

But also...

The power consumption stuff the've done is awesome, but it's not exciting from a straight-line speed perspective.

Improved power consumption on the same process node == better overclocking.

Ivy is a process shrink, which gives efficiency gains just by it's very nature.  But, they're not good at the process yet (and keep in mind, of course, Intel does it first).  Haswell, like Sandy before it, is an "architecture node" in their tick/tock release cycle.  This means that they've now gotten good at a particular process node, and have already figured out the best way to optimize for yields and top binnings.

This means, better overclocking (usually, unless they hit issues with the new design, and Intel doesn't often screw up).

Very generally speaking, Sandy overclocks better than Ivy.  Ivy has some IPC advantages, and the GPU is way better (they seem to be alternating the GPU tick/tocks with the CPU cores), but it doesn't overclock very well.  Since they started this new cadence, the Tocks (new architecture) have tended to be the ones you want for overclocking, because they are the optimized releases on a particular node.

[Hendrik]

Compiling is multi-threaded, so you could greatly benefit from a multi-CPU setup, so grab Ivy Xeons once they are available.
I wish Intel would start offering 6 core CPUs in their mainline (ie. non-E lines) setup, i would get one even if its rather pricy.

[mykillk]

Compilation speed basically comes down to raw PC speed, how effectively your build environment takes advantage of parallel processing, and sometimes the hard disk speed

I have used SCONS before as my build environment and it did a really great job at splitting compilation into parallel tasks automatically. Other build tools also have similar parallel compilation support but my experience was mainly with SCONS (it's especially great if you already know Python).

It might be a fairly daunting task to recreate the MC18 build environment in SCONS or other build environment but it also will give you the benefit of helping to standardize the build environment across multiple operating systems.

[InflatableMouse]

Breathe .... just lost half an hour typing due to hitting the wrong key  

I'll be back later.

[Jong]

I've just run JRMark on my 16GB Ivy 3770k system running @4.5Ghz. It has a Vertex4 512GB system drive. It should be pretty speedy, yet it only scores around 4900 JRMark. So that's a pretty speedy Sandy Bridge system you have there!

Probably your best speedup now might be to go for a 6-core Sandy Bridge 3960X, which might be as simple as swapping the chip on your current system: http://www.anandtech.com/show/5091/intel-core-i7-3960x-sandy-bridge-e-review-keeping-the-high-end-alive/5

[Jong]

Might want to see this too: Building The Linux Kernel In 60 Seconds

[InflatableMouse]

As said above, compiling heavily depends on cpu power and because its so heavily multithreaded, the more cores the happier it gets.

You're coming from a Core i7 2600K, which is a 4-core processor supporting hyperthreading. I don't think HT adds much benefit to compiling unless the compiler is optimized to use HT. In some cases, disabling HT can even make certain tasks faster (and allows for higher overclocks and a cooler running cpu). The Core i7 is clocked at 3.4Ghz and although there are faster cpu's, even adding a single core would benefit you more than increasing clockspeeds (4 x 600 = 2.4Ghz, an extra core adds 3.8Ghz). So, based on that adding cores is your ticket to heaven.

You mentioned server hardware, but I can tell you the common server hardware from Dell or HP will be slower than what you can build yourself for less. That stuff is built to be reliable, not to be the fastest kid on the block. You say ECC would be a plus but I think you're wrong. Buffered ECC is quite a bit slower and although it sounds good, I can't remember the last time I had to have memory replaced in one of our servers. I remember it happened once and even that time it turned out to be the mainboard. Mind you I'm working in a datacenter with over a 1000 systems and I've been here since 2004. Some of that hardware is still running. Physical memory failures are rare and if there are issues due to memory, its because its incompatible with the mainboard, its incorrectly configured, incompatible sticks are stuck together or its running too hot. Solve those, run a 24-hour stresstest and you can very certain memory won't be an issue. Manufacturers can give lifetime warranty on their sticks - they don't do that if its even remotely likely to fail.

CPU's currently available that would add a significant speed bump
Intel Xeon E5-2687W, 8-core, HT, 3.1Ghz/3.8Ghz
Intel Xeon E5-2690, 8-core, HT, 2.9Ghz/3.8Ghz
Intel Core i7 3970X, 6-core, HT, 3.5Ghz/4Ghz

Whether those are an in-place upgrade to your current cpu I don't know, but we want to do things properly so:

Dual socket mainboards
Intel DBS2600CP4, dual socket 2011, 16x DDR3, C602 chipset
Asus Z9PE-D8 WS, dual socket 2011, 8x DDR3, C602 chipset
Supermicro X9DA7, dual socket 2011, 16x DDR3, C602 chipset

All three CPU's are Sandy's because we determined you need more cores and there are no 8-core or even 6-core Ivy's available. Intel has promised a 6-core i7 Ivy for the 2nd half of 2012, but until now we've seen none and no word from Intel either unless I missed something. Either way, they don't exist. They may not even come any time soon with Haswell scheduled for 3Q/4Q this year until Intel has something in the Haswell pipeline to match it.

Reasons not to wait for Haswell
Intel is promising a 10-15% speedbump from Haswell over Ivy. Ivy promised a similar speedbump over Sandy. But practically what we've seen is more around 5%. Expect something similar for Haswell. Realisticly we should expect around 10% over Sandy from Haswell. Intel's predictions have always been too optimistic and its basically just marketing bloat. Adding cores will provide a bigger speed increase, something that can be done with Sandy, but not with Ivy or Haswell.

Haswell isn't even here. Haswell may not even come may/june. It may be a big dissapointment at which point Sandy's may even rise in cost because of a higher demand. We can expect the same thing we've seen with every new cpu Intel introduced: ~5% speed increase and similar to the mainstream clockspeeds. There won't be 6 or 8-cores, there won't be extreme clockspeeds. Haswell won't beat the fastest Ivy's or even the fastest Sandy's in the first 6 months after release. Even if intel could deliver something like that right off the bat, they won't because its not the smart thing to do. They need something new in 6 to 12 months. They want people to buy at introduction and upgrade later, not buy now and not upgrade in 2 years. If you want to wait for a Haswell that can compete with 12 or 16 cores Sandy violence, you may need to wait a year and a half, maybe even two.

Another reason not to wait for Haswell could be the Ivy rumor mill. Rumors have it that Intel is planning a 10-core Ivy Xeon supporting HT, 30MB cache and 1866Mhz DDR3, compatible with current socket 2011 boards. Dare I say it even mentioned a 12-core in the pipeline? Imagine dual socket, 24-core @ 4Ghz. I can assure you no Haswell in the coming years will beat that.

Reasons to wait for Haswell
You could just upgrade to an i7 3970X now and wait for Haswell. It's most likely an in-place upgrade for your current cpu (but check your mainboard supports it!) and its relatively cheap compared to the other options available right now. It gives you 2 more cores and theoretically more than 50% speed bump (2 more cores and 600Mhz per core speed increase). Haswell will very likely bring much better power efficiency. If you wait, you get a new socket, new chipset, PCI-E 3.0 and-I-dont-know-what-else. Good stuff. Besides, the socket 2011 is not very future proof anymore. It will give you 12 or 16 cores in a dual socket setup and with that, a massive speed increase over what you have right now, but you only get to do it once. Well, maybe perhaps when/if the 10 or 12 core Ivy rumors are true and really fit in the 2011 boards I listed above you could do it again, but that is a major gamble because if they don't show up or don't fit on those boards, then you're stuck with it to replace it all again for a Haswell platform.

To sum it up

• Do nothing. Wait for Haswell. Yawn.
• Upgrade your current system with an Intel Core i7 3970X. Relatively cheap and provides about 50% speed increase, give or take. Wait for Haswell.
• Get a new dual socket mainboard and put 2 6-core Intel Core i7 3970X on it. Not so cheap anymore but will likely be more than twice as fast as your current system.
• Get a new dual socket mainboard and put 2 8-core Xeons on it. Rule the world hard and fast. The most expensive option, but like 4 times as fast as what you have now.

Note: The speed increases I estimate are linear because I assume compiling scales linearly with both cpu speed and extra cores. It may not work out that way because 1) the assumption is incorrect, 2) there are other unforeseen hardware limitations or 3) both 1 and 2 apply

Other things to consider
Although storage may not be a limiting factor in throughput, when you build a system around a dual socket with 12 or 16 cores (and possibly even 20 or 24) the sheer number of I/O requests and interrupts can become a bottleneck on another component like the controller if all disks are connected to it. Spreading disks over multiple controllers and separate buses would help prevent such issues. Things like looking at your workflow and optimizing your disk layout to match can help so that you're not reading and writing to the same disk, or so that your tempprary files during compilation are not on the same disk as the source or destination.

I bet the same can be said for memory as well. For normal situations in regular desktop work, having faster memory doesn't really pay off in terms of a noticable performance boost. But having so many cores running threads poking around in memory simultaneously may show bottlenecks that normally never show. What might help here is not just higher clocked RAM, but lower CAS latencies. Typically, the cheaper sticks that boost high clockspeeds have also very high CAS latencies but lower latencies might provide better performance in this case.

[Jong]

If going the 6-core route it seems sensible to compare 3960X and 3970X and buy the cheaper. Both are SB parts and the difference seems to be only the default clock speed. Any 3960X part will overclock way beyond the off the shelf 3970X speed and there are reports that some 3960X parts overclock better than some 3970X parts. Basically they are interchangeable and best to buy whichever is cheaper!

IM makes a good point about investing in plenty of good, fast, low latency RAM.

[Matt]

Thanks for all the advice everyone.

This thread got me looking into the concurrency of compiling a little more.  I noticed that Visual Studio can now use multiple cores to compile a single project:
http://msdn.microsoft.com/en-us/library/vstudio/bb385193.aspx

Currently we compile multiple projects at a time, but don't enable multi-processor compiling of a single project.  Hopefully turning that on for the bigger projects (or maybe all of them) will win us some additional performance.

[glynor]

Another reason not to wait for Haswell could be the Ivy rumor mill. Rumors have it that Intel is planning a 10-core Ivy Xeon supporting HT, 30MB cache and 1866Mhz DDR3, compatible with current socket 2011 boards. Dare I say it even mentioned a 12-core in the pipeline? Imagine dual socket, 24-core @ 4Ghz. I can assure you no Haswell in the coming years will beat that.tencies but lower latencies might provide better performance in this case.

That's a reason to wait for Haswell, not the reverse.

Intel's pattern over the past few years has been to introduce new architecture on the tock part of the cadence (second gen on the same process node), and at the same time (roughly), release the Xeon versions (and other high-end workstation chips) based on the previous-gen's (tick) architecture.  Right now, the high end chips are all based on Sandy, but we're likely to get Ivy-based Xeons sometime around June.  It is all about their cadence and systematic refinement.

Even if you don't buy a Haswell, I think there is a reason to wait for Haswell, assuming you already have a high-performing Sandy (which, he does).  In other words, rather than the current 2xxx-series Xeons, we'll have 3xxx series soon.

Also...

Intel is promising a 10-15% speedbump from Haswell over Ivy. Ivy promised a similar speedbump over Sandy. But practically what we've seen is more around 5%.

It is all workload dependent.  I've seen plenty of benchmarks where Ivy shows (even dramatic) results of its IPC gains.  As I explained above, Sandy (being a tock) is likely to be better for overclocking (it is).  But at identical clockspeeds, many applications do show speedups of between 5-15%.  The average is certainly not just "around 5%".  It just depends on what you do (though many games have tended to be more on the lower-end of the scale).

In any case, I don't think we're in the sweet spot for buying right now.  It also isn't the worst time possible (as April/May would be), but... If you can (and can still go through with it in June/July as easily), I'd wait a piece.

[Scolex]

I've just run JRMark on my 16GB Ivy 3770k system running @4.5Ghz. It has a Vertex4 256GB system drive. It should be pretty speedy, yet it only scores around 4900 JRMark. So that's a pretty speedy Sandy Bridge system you have there!

Probably your best speedup now might be to go for a 6-core Sandy Bridge 3960X, which might be as simple as swapping the chip on your current system: http://www.anandtech.com/show/5091/intel-core-i7-3960x-sandy-bridge-e-review-keeping-the-high-end-alive/5

I just ran the JRMark (18.0.124) on an Ivy i5 3570k @ 4.4 with 8GB of RAM and 2 Kingston HyperX 128GB SSDs in RAID 0 (striped) and got a 5362. This puzzles me a bit since a 3770k @ 4.5 should receive better numbers as I can't see my RAID SSDs effecting the result that much (could be wrong though).

Matt I would go with a multiple CPU setup if I wanted/needed more speed since IMHO no single socket setup out there is going to make a big enough difference to warrant the cost.

[Jong]

Yeah, I didn't spend a lot of time on it, just dashed up ran the test and posted. Maybe I should look into it a bit more.

Where does the test read/write data? Only to appData, or elsewhere? All my docs/pictures/music are on HDD but the main Users folder (inc. appData) is on SSD.

[glynor]

You mentioned server hardware, but I can tell you the common server hardware from Dell or HP will be slower than what you can build yourself for less. That stuff is built to be reliable, not to be the fastest kid on the block. You say ECC would be a plus but I think you're wrong. Buffered ECC is quite a bit slower and although it sounds good, I can't remember the last time I had to have memory replaced in one of our servers. I remember it happened once and even that time it turned out to be the mainboard. Mind you I'm working in a datacenter with over a 1000 systems and I've been here since 2004. Some of that hardware is still running. Physical memory failures are rare and if there are issues due to memory, its because its incompatible with the mainboard, its incorrectly configured, incompatible sticks are stuck together or its running too hot. Solve those, run a 24-hour stresstest and you can very certain memory won't be an issue. Manufacturers can give lifetime warranty on their sticks - they don't do that if its even remotely likely to fail.

While, I agree with you generally about ECC for this situation, preventing RAM "failure" is not what ECC is really all about.  ECC prevents transient errors from impacting individual transactions (which might not otherwise be detectable), to protect against potentially small, impossible to "reproduce", errors in results.  These can come from a variety of sources, but it can and does exist (and is getting worse as node sizes decrease).

You're right, ECC is slower.  However, memory latency has an overall very small effect on the performance of the system, once you reach the "sweet spot" of the architecture in question.  Anand has some very nice writeups on this, but it amounts to:  DDR-1600 is the sweet spot for Sandy/Ivy-based designs.  You do get more performance with going higher (dramatically in some memory bandwidth sensitive applications), but the rate of returns drops off sharply.  Since low-latency DDR's prices scale essentially "in the reverse", there is usually not a good ROI on going with high-speed or low-latency (same thing, different branding) RAM.

Still, I'd also agree, that ECC probably wouldn't help them much here.  It is essential for things like HPC for scientific applications.  Here... I don't know, you'd just rebuild again.

[Matt]

You say ECC would be a plus but I think you're wrong. Buffered ECC is quite a bit slower and although it sounds good, I can't remember the last time I had to have memory replaced in one of our servers.

I've had bad memory in my main home machine twice over the course of three machines.

It's a small sample size, but I just don't trust memory as a result.

When it goes south, it leaves you with a big mess: "Oh, any file copy I did in the last month might be corrupt."

[Jong]

Hmm. Updated my 3770K machine to .124 (it was on .117) and now get JRMark of 5281. No other changes, not even a reboot.

Code: [Select]

=== Running Benchmarks (please do not interrupt) ===

Running 'Math' benchmark...
    Single-threaded integer math... 3.015 seconds
    Single-threaded floating point math... 2.001 seconds
    Multi-threaded integer math... 0.853 seconds
    Multi-threaded mixed math... 0.530 seconds
Score: 2969

Running 'Image' benchmark...
    Image creation / destruction... 0.162 seconds
    Flood filling... 0.343 seconds
    Direct copying... 0.543 seconds
    Small renders... 0.851 seconds
    Bilinear rendering... 0.655 seconds
    Bicubic rendering... 0.620 seconds
Score: 6933

Running 'Database' benchmark...
    Create database... 0.297 seconds
    Populate database... 1.031 seconds
    Save database... 0.268 seconds
    Reload database... 0.054 seconds
    Search database... 0.747 seconds
    Sort database... 0.724 seconds
    Group database... 0.497 seconds
Score: 5942

JRMark (version 18.0.124): 5281
No idea if it is the upgrade that changed the result or just the time of day

One OT point, my HTPC (3550K @3.4Ghz, 8GB RAM, 128GB Vertex4 SSD) has a JRMark of 4640

Code: [Select]

=== Running Benchmarks (please do not interrupt) ===

Running 'Math' benchmark...
    Single-threaded integer math... 3.569 seconds
    Single-threaded floating point math... 2.373 seconds
    Multi-threaded integer math... 1.797 seconds
    Multi-threaded mixed math... 1.236 seconds
Score: 2117

Running 'Image' benchmark...
    Image creation / destruction... 0.186 seconds
    Flood filling... 0.375 seconds
    Direct copying... 0.426 seconds
    Small renders... 1.008 seconds
    Bilinear rendering... 0.651 seconds
    Bicubic rendering... 0.593 seconds
Score: 6792

Running 'Database' benchmark...
    Create database... 0.369 seconds
    Populate database... 1.186 seconds
    Save database... 0.267 seconds
    Reload database... 0.039 seconds
    Search database... 0.894 seconds
    Sort database... 0.914 seconds
    Group database... 0.622 seconds
Score: 5010

JRMark (version 18.0.124): 4640
In most ways my 3770K shows a big improvement, but the image benchmark not so much! What does the image benchmark test?

[InflatableMouse]

It is all workload dependent.  I've seen plenty of benchmarks where Ivy shows (even dramatic) results of its IPC gains.  As I explained above, Sandy (being a tock) is likely to be better for overclocking (it is).  But at identical clockspeeds, many applications do show speedups of between 5-15%.  The average is certainly not just "around 5%".  It just depends on what you do (though many games have tended to be more on the lower-end of the scale).

Workload dependent means its situational right? I've seen big dramatic benchmark results too, but how were they made? They are either very situational or made on 2 setups, different mobo's, maybe even different chipsets. The benchmark link I gave is particularly interesting if you ask me because its "just" an enthusiast, like you and me. He swapped the CPU in a system, he didn't compare 2 systems. He has nothing to gain by posting optimistic results like some big hardware sites reviewing the very stuff they advertise elsewhere on their site. If you look at his Pi and Prime benchmarks, the results are less than 5%.

Look at AnandTech's compilation results, we see an increase from 17.7 minutes to 18.6. Only just over 5% increase. I quote:

Compile Chromium Test

You guys asked for it and finally I have something I feel is a good software build test. Using Visual Studio 2008 I'm compiling Chromium. It's a pretty huge project that takes over forty minutes to compile from the command line on a Core i3 2100. But the results are repeatable and the compile process will stress all 12 threads at 100% for almost the entire time on a 980X so it works for me.

Ivy Bridge shows more traditional gains in our VS2008 benchmark—performance moves forward here by a few percent, but nothing significant. We are seeing a bit of a compressed dynamic range here for this particular compiler workload, it's quite possible that other bottlenecks are beginning to creep in as we get even faster microarchitectures.

Averaged out, I think I'm even on the high side with 5%.

[Matt]

Updated my 3770K machine to .124 (it was on .117) and now get JRMark of 5281.

JRMark scores often get better with newer builds.

JRMark is testing raw machine performance and also our code.  So when our code or compiler gets faster, the JRMark improves.

If you compare JRMark scores, it's important to use similar versions of the program.

At some point, I think we'll probably rebalance the tests (and maybe add a few things) because some of the performance improvements have changed the weighting quite a bit.

Here's an example:
http://yabb.jriver.com/interact/index.php?topic=70947.5

[InflatableMouse]

I've had bad memory in my main home machine twice over the course of three machines.

It's a small sample size, but I just don't trust memory as a result.

When it goes south, it leaves you with a big mess: "Oh, any file copy I did in the last month might be corrupt."

Glynor brings up some good points too which are very true. If you're worried about it then by all means go for it, if only for piece of mind. The speed tradeoff is a minor one.

[Jong]

There would certainly be no point upgrading to current IB chips. It might be worth waiting until there is an IB 6/8-core chip, whenever that might be. But the 3960/70X seems like it should offer a major boost quite simply.

[Jong]

JRMark scores often get better with newer builds.

JRMark is testing raw machine performance and also our code.  So when our code or compiler gets faster, the JRMark improves.

If you compare JRMark scores, it's important to use similar versions of the program.

At some point, I think we'll probably rebalance the tests (and maybe add a few things) because some of the performance improvements have changed the weighting quite a bit.

Here's an example:
http://yabb.jriver.com/interact/index.php?topic=70947.5

Thanks Matt, just interested that the image tests don't improve much with the major clock speed leap between the 2 systems (both on .124). Some sub-tests are even slower on the 3770K system. What bits of the system is it testing. When writing files (if it does that) where does it write them?

[Matt]

Thanks Matt, just interested that the image tests don't improve much with the major clock speed leap between the 2 systems (both on .124). Some sub-tests are even slower on the 3770K system. What bits of the system is it testing. When writing files (if it does that) where does it write them?

The image benchmark will test raw computing power and memory performance (images are big chunks of memory).  Both are important.

The only disk I/O is in the database test, and there's not much of it.

[InflatableMouse]

However, memory latency has an overall very small effect on the performance of the system, once you reach the "sweet spot" of the architecture in question.  Anand has some very nice writeups on this, but it amounts to:  DDR-1600 is the sweet spot for Sandy/Ivy-based designs.  You do get more performance with going higher (dramatically in some memory bandwidth sensitive applications), but the rate of returns drops off sharply.  Since low-latency DDR's prices scale essentially "in the reverse", there is usually not a good ROI on going with high-speed or low-latency (same thing, different branding) RAM.

Yeh, I know and you're right. But remember that's under normal circumstances. They *only* reason I bring that up is because I don't really know what happens when you start compiling on 16 or more cores. I don't think anyone has ever tested what effect low latency memory does in such a case (maybe none ). I do know that on heavily overclocked systems, lower latencies can have a dramatic effect on cpu heavy things like compression/decompression whereas on that same system not overclocked, it barely has an effect. So I was thinking under such loads, with so many threads poking around in memory, lower CAS latencies might speed things up as they do on a heavily overclocked system. But you know, its all guesswork and I honestly don't know.

[Jong]

The image benchmark will test raw computing power and memory performance (images are big chunks of memory).  Both are important.

The only disk I/O is in the database test, and there's not much of it.

I can only believe it is memory limited (at least on these systems). Both are running RAM at the same speed.

[Jong]

Rebooted and ran again (no other changes) and I get 5490! Image score much improved  ?

Code: [Select]

=== Running Benchmarks (please do not interrupt) ===

Running 'Math' benchmark...
    Single-threaded integer math... 3.019 seconds
    Single-threaded floating point math... 2.000 seconds
    Multi-threaded integer math... 0.891 seconds
    Multi-threaded mixed math... 0.613 seconds
Score: 2913

Running 'Image' benchmark...
    Image creation / destruction... 0.156 seconds
    Flood filling... 0.325 seconds
    Direct copying... 0.435 seconds
    Small renders... 0.852 seconds
    Bilinear rendering... 0.574 seconds
    Bicubic rendering... 0.527 seconds
Score: 7671

Running 'Database' benchmark...
    Create database... 0.316 seconds
    Populate database... 1.067 seconds
    Save database... 0.265 seconds
    Reload database... 0.048 seconds
    Search database... 0.748 seconds
    Sort database... 0.725 seconds
    Group database... 0.484 seconds
Score: 5886

JRMark (version 18.0.124): 5490
Enough now

[InflatableMouse]

E-peen alert! 

[Thumper]

Thanks for all the advice everyone.

This thread got me looking into the concurrency of compiling a little more.  I noticed that Visual Studio can now use multiple cores to compile a single project:
http://msdn.microsoft.com/en-us/library/vstudio/bb385193.aspx

Currently we compile multiple projects at a time, but don't enable multi-processor compiling of a single project.  Hopefully turning that on for the bigger projects (or maybe all of them) will win us some additional performance.

What about using a product like Incredibuild? It pushes builds out to idle PCs on the network. I've never had a project big enough to need it but there is a free trial;integrates with Visual Studio.  Supposedly allows parallelization without any source file or hardware changes. (I love advertising blurbs)

[glynor]

Look at AnandTech's compilation results, we see an increase from 17.7 minutes to 18.6. Only just over 5% increase.
Averaged out, I think I'm even on the high side with 5%.

Oh, I agree.  With many workloads the bump from Sandy to Ivy is minor (more minor than previous ticks in the cycle had been).  They focused on power efficiency improvements with the node change this time.

Though other applications can get substantial improvements... Photoshop, for one, sees some pretty nice gains.

To be clear:  I was never suggesting that Ivy would be a good upgrade from a Sandy system.  Quite the contrary!  I was just saying that when you go two "cycles" (two years on their current cadence), you get cumulative improvements.  Quite often, you'll see many of the underlying areas that didn't get dramatic improvements in the last cycle, gain additional focus in the next.  Therefore, it isn't usually the "worst examples" of bumps that you end up with over two+ cycles, but closer to the median.  It averages out higher than you were seeming to indicate, is what I'm saying, when you include multiple tick/tock cycles.  Iteration wins the race, in other words.

Waiting for Haswell to fully ship allows a few things:

1. You can see what has been improved in the new architecture, with shipping silicon.
2. If you decide to go with a "desktop" chip (Core i7), you'll probably have better overclocking potential with Haswell than Ivy (which will be similar to, but maybe a bit worse-than, Sandy).
3. More importantly, Ivy Bridge-E is coming in 2013 (probably within a few months of Haswell, just like before), and will be a drop-in upgrade for current X79 LGA2011 systems.  Ivy-E is expected to ship in 6-12 core versions.  They are also shipping improved versions of the existing Sandy Bridge-E chips in Q2 2013.

In summary:  There are new options in the LGA2011 space coming.  I'd wait for them, and compare them to the (then available) Haswell chips, and decide which has better bang-for-your-buck.

[InflatableMouse]

Waiting for Haswell to fully ship allows a few things:

Yeh I totally agree, if you can wait it can definitely be worth it.

Although I tried to give a balanced view I admit I focused more on what's available and less on whats coming; you laid that out quite nicely.

[mykillk]

Thanks for all the advice everyone.

This thread got me looking into the concurrency of compiling a little more.  I noticed that Visual Studio can now use multiple cores to compile a single project:
http://msdn.microsoft.com/en-us/library/vstudio/bb385193.aspx

Currently we compile multiple projects at a time, but don't enable multi-processor compiling of a single project.  Hopefully turning that on for the bigger projects (or maybe all of them) will win us some additional performance.

Yea, that's exactly the kind of feature I was talking about. It can make a HUGE difference! It only helps for release builds though according to the note at the bottom:

"The /Gm Compiler Option

By default, a project build enables the /Gm compiler option (incremental builds) for debug builds, and disables it for release builds. Therefore, the /MP compiler option is automatically disabled in debug builds because it conflicts with the default /Gm compiler option."

[mykillk]

What about using a product like Incredibuild? It pushes builds out to idle PCs on the network. I've never had a project big enough to need it but there is a free trial;integrates with Visual Studio.  Supposedly allows parallelization without any source file or hardware changes. (I love advertising blurbs)

A distributed build system like that isn't as appealing as it used to be because there's a limit to how much can be parallelized and today's CPUs are capable of delivering significant parallelism. Back when we only had single and dual core processors with a single hardware thread, something like Icredibuild was great.

But now we have quad, six, and 8-core CPUs with two hardware threads.

Put two Intel 6-core CPUs with hyper-threading and you have 24 parallel units of execution. That's some serious parallelism, and without the network latency.

[mykillk]

Matt do you use include guards or #pragma once in your header files (Assuming MC18 is written in C or C++)? Check this out:

http://www.bobarcher.org/software/include/index.html

[Hilton]

Hi Matt,

I think until ivy-e ships later this year your best bet is a 3970X or even a 3930K. (with a new MB and 1600Mhz Ram of course)

Changing over to a Xeon is only worth it IMHO if you plan on running dual CPU and can run highly parallel apps and compiling.
The top end Xeons are more than twice the price for about 75% of the Ghz of 3970X and only 2 more cores and this cant make up the deficit in Ghz.

In all the synthetic and workload tests Ive read the i7 3970X and 3930K both match or beat a single E5 2690 with just a mild overclock in most tests. Not even in highly multithreaded apps will the Xeon beat the 3930K or a moderately OC 2600K.

The i7 3930K will easily OC to 4.4Ghz and the Xeon 8 cores dont OC at all! Here's some benchmarks for you that someone ran over at tomshardware.
Even your current Sandy 2600K with a moderate overclock would be about 10mins faster than the Xeon 2670 in this test below. (Thats 20% FASTER)
You cant beat Ghz!

CS5.5
Video material - AVCHD 1080P 24 Frame Each Cut to 30 minutes of material
Export Codec - H264 HDTV 1080P 24 Preset Default
4 Effects per Layer - Fast Color Corrector, Brightness & Contrast, Video Limiter, Sharpen
Each Layer Scaled to 50% for 4 frame PinP view.

E5 2670 @ 2.6 GHz 8 CORE
32GB 1600
570GTX 2.5GB
4 1Tb Sata 32 Meg Cache 600 Drives in 2 Raid 0 arrays
CS5.5.2
3 Layer -
4 Layer - 40:41

X79 3.3 @ 3.8 GHz
32GB 1333
580GTX 3GB
4 1Tb Sata 32 Meg Cache 600 Drives in 2 Raid 0 arrays
CS5.5.2
3 Layer - 32:15
4 Layer - 35:19

X79 3.3 @ 4.5 GHz
32GB 1333
580GTX 3GB
4 1Tb Sata 32 Meg Cache 600 Drives in 2 Raid 0 arrays
CS5.5.2
3 Layer - 27:43
4 Layer - 30:02

I7 2600K 4.7 GHz 4 core
16GB Blackline 1600 CL 9
570GTX
4 WD 1Tb Sata 64 Meg Cache 600 Drives in 2 Raid 0 arrays
3 Layer - 30:46
4 Layer - 33:36

Just for reference here's my 3930K at 4.4Ghz with JRMARK of 5368
Auto OC which most boards/i7 CPU do on full auto, I'll reboot now and do a bench at 5ghz.
JRMark (version 18.0.120): 5368 @ 4.4Ghz
JRMark (version 18.0.120): 6128 @ 5Ghz


=== Running Benchmarks (please do not interrupt) ===

Running 'Math' benchmark...
    Single-threaded integer math... 3.090 seconds
    Single-threaded floating point math... 2.044 seconds
    Multi-threaded integer math... 0.650 seconds
    Multi-threaded mixed math... 0.537 seconds
Score: 3006

Running 'Image' benchmark...
    Image creation / destruction... 0.158 seconds
    Flood filling... 0.422 seconds
    Direct copying... 0.358 seconds
    Small renders... 0.908 seconds
    Bilinear rendering... 0.609 seconds
    Bicubic rendering... 0.548 seconds
Score: 7327

Running 'Database' benchmark...
    Create database... 0.331 seconds
    Populate database... 1.077 seconds
    Save database... 0.217 seconds
    Reload database... 0.030 seconds
    Search database... 0.811 seconds
    Sort database... 0.779 seconds
    Group database... 0.481 seconds
Score: 5771

JRMark (version 18.0.120): 5368

.... and at 5Ghz...

=== Running Benchmarks (please do not interrupt) ===

Running 'Math' benchmark...
    Single-threaded integer math... 2.727 seconds
    Single-threaded floating point math... 1.806 seconds
    Multi-threaded integer math... 0.626 seconds
    Multi-threaded mixed math... 0.402 seconds
Score: 3417

Running 'Image' benchmark...
    Image creation / destruction... 0.137 seconds
    Flood filling... 0.376 seconds
    Direct copying... 0.299 seconds
    Small renders... 0.800 seconds
    Bilinear rendering... 0.520 seconds
    Bicubic rendering... 0.475 seconds
Score: 8441

Running 'Database' benchmark...
    Create database... 0.299 seconds
    Populate database... 0.923 seconds
    Save database... 0.186 seconds
    Reload database... 0.026 seconds
    Search database... 0.709 seconds
    Sort database... 0.687 seconds
    Group database... 0.466 seconds
Score: 6524

JRMark (version 18.0.120): 6128

[InflatableMouse]

I think until ivy-e ships later this year your best bet is a 3970X or even a 3930K. (with a new MB and 1600Mhz Ram of course)

Do you know which mainboard he has? Chances are that an in-place upgrade of the cpu is possible. Maybe he already has 1600Mhz RAM .

Changing over to a Xeon is only worth it IMHO if you plan on running dual CPU and can run highly parallel apps and compiling.
The top end Xeons are more than twice the price for about 75% of the Ghz of 3970X and only 2 more cores and this cant make up the deficit in Ghz.

3.8 (E5-2690) -> 4Ghz (3970X) = 5%. It's 200Mhz times 6 = 1.2Ghz. The 2 cores from the Xeon would basically add 7.6Ghz, 6.33 times more than the speed increase of the 3970X. So if compiling is as heavily threaded as they say it is, it should scale much better with more cores than it does with a bit more clockspeed.

In all the synthetic and workload tests Ive read the i7 3970X and 3930K both match or beat a single E5 2690 with just a mild overclock in most tests. Not even in highly multithreaded apps will the Xeon beat the 3930K or a moderately OC 2600K.

http://www.tomshardware.com/reviews/core-i7-3970x-sandy-bridge-e-benchmark,3348-8.html

Nuf said .

The i7 3930K will easily OC to 4.4Ghz and the Xeon 8 cores dont OC at all! Here's some benchmarks for you that someone ran over at tomshardware.
Even your current Sandy 2600K with a moderate overclock would be about 10mins faster than the Xeon 2670 in this test below. (Thats 20% FASTER)

An E5-2670 is clocked at 2.6/3.3Ghz against 2.9/3.8Ghz for the E5-2690. Besides, all those benchmarks are taken on different hardware, different chipsets and different memory so that's hardly a valid comparison.

You need to OC an 3970X by 25% to beat an E5-2690 on raw processing power. But even then it has less cache on all levels, something I believe can be quite beneficial for large projects.

You cant beat Ghz!

This used to be true when programs were less multithreaded and often even being single threaded, optimized for single core processors. This is has been changing since multicore cpu's have become more common and I don't believe it holds up anymore, except for some rare situations.

[Hilton]

Do you know which mainboard he has? Chances are that an in-place upgrade of the cpu is possible. Maybe he already has 1600Mhz RAM .
3.8 (E5-2690) -> 4Ghz (3970X) = 5%. It's 200Mhz times 6 = 1.2Ghz. The 2 cores from the Xeon would basically add 7.6Ghz, 6.33 times more than the speed increase of the 3970X. So if compiling is as heavily threaded as they say it is, it should scale much better with more cores than it does with a bit more clockspeed.
http://www.tomshardware.com/reviews/core-i7-3970x-sandy-bridge-e-benchmark,3348-8.html

Nuf said .

An E5-2670 is clocked at 2.6/3.3Ghz against 2.9/3.8Ghz for the E5-2690. Besides, all those benchmarks are taken on different hardware, different chipsets and different memory so that's hardly a valid comparison.

You need to OC an 3970X by 25% to beat an E5-2690 on raw processing power. But even then it has less cache on all levels, something I believe can be quite beneficial for large projects.

This used to be true when programs were less multithreaded and often even being single threaded, optimized for single core processors. This is has been changing since multicore cpu's have become more common and I don't believe it holds up anymore, except for some rare situations.

I still stand by my comments and the tests I provided which are based on realistic OC of the i7 and the hardware was as common as possible for the testing.  The link you posted for the CPU testing was all at stock speeds and as you can clearly see the Xeon is hardly faster even without overclocking the i7.  The 3930K is the best bang for buck by a long shot...

Just look at my JRMark's at 4.4Ghz & 5Ghz.  I think those kinds of increase in performance from OC would leave the Xeon for dead.  Going from 12MB cache on 3930K or 15MB on 3970K to 20MB cache on the Xeon may make a small improvement but i guarantee you it will not catch the OverClocked 12 threads of an i7.

JRMark (version 18.0.120): 5368 @ 4.4Ghz
JRMark (version 18.0.120): 6128 @ 5Ghz

The Beast known as the "Borg Cube"... For good reason. It has assimilated so much hardware!

[InflatableMouse]

That's sweet machine you've got there.

The 3930K is the best bang for buck by a long shot...

I never argued against that 

But that wasn't the question.

It was:

Is there anything available that would be considerably faster, even a Xeon / server machine?

And

Water cooling or hard core overclocking would be a little too much.

But to be clear, I'm not saying you're wrong, you do bring up valid points for Matt to consider  .

[mykillk]

Holy cow, that computer is bigger than my coffee table. Are those NINE 140mm fans in the front AND another layer of nine fans behind it?!? You could power a wind tunnel with that!

And I have to concur with hiltonk's assessment. The biggest gain to be had is in overclocking, and that's even more true for compiling. Cache sizes don't really matter, it's the raw speed and number of cores. I'd go by a simple formula of X Cores * Y Ghz = Z Ghz and go for the whatever gets you the biggest Z for your budget. A six core i7 at 4.5 ghz gives a Z of 27, and you can't beat that with an 8-core Xeon until you go all the way up to an E5-2680 (3.5 ghz Turbo Boost) which is an $1800 CPU! So yea, you might be able to beat an i7 with a Xeon but the difference will be marginal and at a cost of $1200 more. And unless your task is consuming 100% of the available parallelism of 2 additional cores at all times, whatever performance increase you might get with the Xeons will quickly evaporate and the OC'd i7 will again take the lead.

Surprisingly, a bit counter-intuitively, the 32nm Sandy Bridge chips are more overclockable chip than the 22nm Ivy Bridge which has problems with heat dissipation because the chips are so dense with the 3d-transistor design.

My vote also goes to the i7-3930K

http://www.newegg.com/Product/Product.aspx?Item=N82E16819116492

Should overclock to well over 4 ghz no problem, without water cooling, and without even a hint of unstability. "Hardcore" overclocking would be trying to push 5 ghz on air. It also has a four-channel memory controller (another nice boost to parallelism) so you'd want 4 sticks of RAM to take full advantage.

[mykillk]

Also gotta have an elite-level motherboard like:

http://www.newegg.com/Product/Product.aspx?Item=N82E16813131808#top

[mykillk]

I know you mentioned that everything was on SSDs Matt, but are they RAIDed? You mentioned going to SSDs was a big boost, you might get a similar boost going to SSDs in a striped RAID array.

[Hilton]

Holy cow, that computer is bigger than my coffee table. Are those NINE 140mm fans in the front AND another layer of nine fans behind it?!? You could power a wind tunnel with that!

Well close.. There are 18 push & pull 120mm fans for the 3 GPU radiators at the front and another 6 at the back for the CPU radiator. And then there's also 2 exhaust fans at the back as well.  Its not as bad as it sounds. The fans idle at only 700rpm which is completely silent and they rarely go above 1200rpm which is still very quiet.   It certainly makes more noise when under full load with all fans at about 1900RPM but its still quieter than the factory air coolers. 

The system draws 1400W PSU1 and 900W PSU2 from the wall from each PSU at 240V under full load so it needs serious cooling with about 400W for each GPU and the CPU too at 5Ghz. The fans and pumps draw the rest of the power.   At idle and normal web surfing and light app loads it still idles at 230W PSU1 / 170W PSU2 so its not too bad unless im gaming or encoding.

[Matt]

Thanks for all the feedback so far everyone.

This week I spent some time writing a build system that uses a custom C++ program instead of using batch files.  It gives us more flexibility and the ability to automate some extra steps in the build.

Using the new tool it takes about 8 minutes to clean, build, virus check, sign, package, install for a smoke test, and upload the build.  This is using the machine I mentioned in the first post.

The /MP mentioned above shaved some time off, although it's a little touchy.  At first I tried turning it on for all projects so there were 64 compilation threads running at once (8 projects x 8 threads each).  My computer blue screened for the first time ever

Anyway, we're going to try living with the new system for a few weeks and then we'll decide if we should throw a 3930K, RAIDed SSDs, etc. at the problem.

I think optimizing the build program and doing more compiler setting optimization could each shave a minute off the process, although it would be easy to spend a couple days of development time for each of those minutes

[MrC]

My computer blue screened for the first time ever

A blue screen is a hardware or driver problem.

:-)

[mykillk]

Thanks for all the feedback so far everyone.

This week I spent some time writing a build system that uses a custom C++ program instead of using batch files.  It gives us more flexibility and the ability to automate some extra steps in the build.

Using the new tool it takes about 8 minutes to clean, build, virus check, sign, package, install for a smoke test, and upload the build.  This is using the machine I mentioned in the first post.

The /MP mentioned above shaved some time off, although it's a little touchy.  At first I tried turning it on for all projects so there were 64 compilation threads running at once (8 projects x 8 threads each).  My computer blue screened for the first time ever

Anyway, we're going to try living with the new system for a few weeks and then we'll decide if we should throw a 3930K, RAIDed SSDs, etc. at the problem.

I think optimizing the build program and doing more compiler setting optimization could each shave a minute off the process, although it would be easy to spend a couple days of development time for each of those minutes

8 minutes vs what previously?

Have you tried just compiling one project at a time with the /MP feature? 64 concurrent threads actually seems counter-productive. I think you'll have the best luck sticking to 8 compilation threads (4 cores x 2 HW threads per core). Either by not using the /MP feature and doing the 8 projects at a time, or by using /MP with 8 threads but only one project at a time.

[Matt]

Looks like 7-13% from Haswell:
http://www.tomshardware.com/reviews/core-i7-4770k-haswell-performance,3461.html

Overclocking is the big question mark.  It requires no effort to get around 4.3 GHz with a 3770k, and I wonder if a 4770k will best that?

[glynor]

Looks like 7-13% from Haswell:
http://www.tomshardware.com/reviews/core-i7-4770k-haswell-performance,3461.html

Overclocking is the big question mark.  It requires no effort to get around 4.3 GHz with a 3770k, and I wonder if a 4770k will best that?

On the recent Tech Report podcast they discussed some of the early Haswell benchmarks (including these).  The overall bump is as-expected, but Scott Wasson also reported on overclocking a bit, and said that early insider reports are that it is closer to Sandy-like overclockability than Ivy-like overclockability.

Which "fits" based on the focus on efficiency, and a new architecture on a mature process node.  If so, that is Very Good News.  Ivy isn't a particularly good overclocker.