12 GB For the RenderFarm

The global scenery gets cut on "the renderfarm", which is our name for the cluster of computers, each loaded with all of the input data for the global scenery. These computers chug for a few days to churn out the whole planet.

With the next global render I am trying an experiment: using one 8-core Mac Pro as the RenderFarm. We've never had more than 8 processors in the farm at a time; in the past to get 5 processors might have taken 3 machines. The appeal of a single machine is ease of setup; no data to sync between machines, no sorting out which machine did which tile and merging it all back.

Today I upgraded the Mac Pro's memory (again) to 12 GB. I thought the logic of why I did this might be of interest to X-Plane users who are trying to figure out "should I have more memory"?

Basically there are three memory limits we care about:

1. The virtual memory limit per process - generally 3 GB per process for a 32-bit application. If an application wants more memory tan this, regardless of what you have, it is dead.
2. The virtual memory limit for the whole machine. Since the machine virtual memory limit is a function of hard disk space, normal users will never care about this - we can have a huge amount of virtual memory.
3. The physical memory actually used by the sum of all programs actually running. Once all programs need more physical memory than you have, they start using hard drive, and they get really, really, really slow.

In the case of the render farm, our processing program runs on one DSF tile at a time. But with 18,000+ tiles we can take advantage of more cores by processing 8 tiles (using 8 copies of the program running at once) on 8 cores.

This is where memory comes in. Before the upgrade my machine had 4 GB of memory, allowing each of the 8 tiles to use a little bit less than 512 MB of RAM before we ran out of physical memory and started paging. (The OS takes a little bit for itself.)

Normally this is all good -- a typical run might only use 300 MB of RAM. But every now and then we hit New York City or Boston and the RAM use spikes out to 1500 MB or more.

This is okay if just one process hits New York, but what if a bunch of processes hit a big city at once? We blow past the physical RAM of the machine and every tile becomes slow at once. And because they are all slow at once, they take a very long time (hours) to clear this state.

(This state of thrash due to many processes is like 8 people trying to go through a doorway at once. If they would just take turns, they'd be through in a second. But by forcing 8 at once they get stuck. The operating system won't pause a few of the high-memory programs to let the others complete.)

Since the RenderFarm has to run overnight unattended, I upgraded RAM to 12 GB, for 1500 MB per rpocess before thrash. My hope is that for this investment, we'll be able to run the processing through without a human to unjam it.

What About X-Plane

So would a RAM upgrade for X-Plane help? Well we can apply what we know from above to figure it out. Generally, you need enough RAM that X-Plane + all other programs won't run out of physical memory. Since X-Plane is 32-bit (and can only use 3 GB of RAM) you are likely to be fine with 4 GB when running X-Plane + the OS. Any more and X-Plane can't take advantage.

The exception might be if you need to run X-Plane and another big program like PhotoShop at the same time. At that point you might want enough RAM for both to run at the same time.