Tech Support Blog

  • Subscribe to our RSS feed.
  • Twitter
  • StumbleUpon
  • Reddit
  • Facebook
  • Digg

Friday, 4 September 2009

Scalability and apt.dat

Posted on 02:30 by Unknown
Beta users, see the bottom of this post for how scalability turns into possible bugs.

In computer science, a program or architecture is scalable if it doesn't totally vomit up a lung as its constituent parts become bigger. For a cleaner definition, see Wikipedia, source of all internet knowledge.

An ant is not scalable - if you made an ant 100 times larger in every dimension, its tiny legs would break under its new weight. (An ant is not scalable because its weight grows faster than its structural strength. Thus elephants are not built like ants.) Geeks: scalability is to computer science as marginal cost is to economics.

Before X-Plane 940, the apt.dat file was distinctly not scalable. The entire file was loaded into memory; as users created more and more taxiway lines and signs and details, we simply used more and more memory. This approach isn't very scalable because authors have the potential to grow the apt.dat file faster than our system requirements can increase.

X-Plane 940 fixes this by not loading the entire apt.dat file into memory. Instead, only essential airport information is loaded into memory, along with a note as to where in the file the airport lives. Whenever an airport actually has to be built into a 3-d mesh while you fly, we go back to the apt. file and load the rest of the data for the one airport we are building, use it, and throw it out. Since 3-d airport meshes are built on a second core, the cost of loading one airport off disk is pretty harmless.

The problem with this fix is that it introduced a new scalability problem. Consider:
  • Meshes in 940 are built on as many CPU cores as you have - some users have 8!
  • Each CPU core could be working on a different airport, depending on how many are nearby.
  • Each airport has to load up the apt.dat file to get the extra airport data.
This means that at times on an 8 core machine we could easily have allocated 50 * 8 = 400 MB of memory just to temporarily hold 8 copies of the apt.dat file.*

This is of course completely silly - there's no reason to load the whole apt.dat file to get one airport, and the fix that is going into beta 8 lets the airport loader surgically grab just one airport. Thus we will be scalable again, because adding more cores won't cause memory usage to go up.

Beta Users: Please keep an eye out for X-Plane running out of memory - if it starts to do so in the next beta it means that some part of this code change munged memory management. We're running stress tests on the sim now, but touching the low level memory nd file handling code late in beta isn't something I like to do.

* While loading 8 copies of the apt.dat file is wasteful of memory, it is not slow; X-Plane uses memory mapped file I/O, so reading a small part of a large file is very fast - just not very virtual-memory efficient.
Email ThisBlogThis!Share to XShare to FacebookShare to Pinterest
Posted in installer, performance, scenery system | No comments
Newer Post Older Post Home

0 comments:

Post a Comment

Subscribe to: Post Comments (Atom)

Popular Posts

  • Pixel Shaders and Moore's Law
    In my post on 64-bit computing and X-Plane, there's a point that's implicit: there is a cost (in development time) to adopting any ...
  • Hierarchy and WED
    I just finished fixing some bugs in WED regarding the hierarchy view. (To get a sense of what a hierarchy view is like, try AC3D...) In WED...
  • Is Bigger Always Better?
    We've been preaching "one big texture, not lots of little textures" for a while now, and generally speaking, packing a lot of ...
  • Systems Modeling and Physics Modeling
    In past blog posts I have tried to contrast two very different different approaches to simulation, which I would loosely call: "Predic...
  • How Framerate Dies - Glitching
    Back in the good old days (that would be X-Plane 6), X-Plane's framerate would suffer in two ways as you cranked up the rendering option...
  • The X-Desktop and iPhone Are Not Zero-Sum Games
    On the menu this morning: first a whiny rant, then a nerdy one. Someone pointed me at this post. Before I go into my geeky diatribe about l...
  • Clean Airport Layouts
    I have blogged about correct airport layouts before, but let me bring the point up again, because it is so important: You must create struc...
  • So How Big of a Mesh Can You Build?
    I get asked about the maximum mesh density in X-Plane a lot. First, I must note that X-Plane's mesh is adaptive - the triangles are not ...
  • What is a panel region?
    X-Plane 9 introduces a new OBJ feature: panel regions. The basic idea is this: In X-Plane 8 you could use the 2-d panel as a texture in you...
  • Why Do Custom Lights Use the Object Texture?
    I am trying to be disciplined and put documentation on the X-Plane wiki , and limit the blog to announcements, rants, and explanations of wh...

Categories

  • absurdly cute
  • Air Traffic Control
  • aircraft
  • Android
  • animation
  • announce
  • cockpits
  • documentation
  • drivers
  • file formats
  • global scenery
  • Goofy Screenshots
  • hacks
  • hardware
  • hobbies
  • inside x-plane
  • installer
  • ipad
  • iphone
  • legal
  • localization
  • modeling
  • off topic
  • palm pre
  • panels
  • performance
  • plugins
  • political
  • scenery system
  • tools
  • X-Plane 10
  • XSquawkBox

Blog Archive

  • ►  2011 (12)
    • ►  February (1)
    • ►  January (11)
  • ►  2010 (111)
    • ►  December (4)
    • ►  November (4)
    • ►  October (10)
    • ►  September (9)
    • ►  August (12)
    • ►  July (8)
    • ►  June (4)
    • ►  May (13)
    • ►  April (13)
    • ►  March (11)
    • ►  February (12)
    • ►  January (11)
  • ▼  2009 (130)
    • ►  December (16)
    • ►  November (11)
    • ►  October (6)
    • ▼  September (16)
      • Off Topic: Freecycle
      • Closing the Washington Branch
      • Normal Maps Can't Be Mirrored
      • I Hate the Term Photo-Realistic
      • Pie in the Sky: Future Rendering Tricks
      • Removing Third Parties is a Diagnostic Test, Not a...
      • Marking the Ground? Use a .pol or .lin.
      • Higher-Order Beziers? Probably Not
      • Parameterized Lights, Generic Lights and Plugin Co...
      • Yank That Stick Back
      • Proximate And Ultimate Crashes
      • Pay No Attention To That Last Post
      • Caught With My Pants Down
      • Scalability and apt.dat
      • Marketing And Fact
      • Test Your Add-Ons Now!
    • ►  August (12)
    • ►  July (11)
    • ►  June (9)
    • ►  May (5)
    • ►  April (10)
    • ►  March (9)
    • ►  February (9)
    • ►  January (16)
  • ►  2008 (147)
    • ►  December (18)
    • ►  November (10)
    • ►  October (7)
    • ►  September (11)
    • ►  August (15)
    • ►  July (9)
    • ►  June (14)
    • ►  May (9)
    • ►  April (14)
    • ►  March (13)
    • ►  February (6)
    • ►  January (21)
  • ►  2007 (100)
    • ►  December (17)
    • ►  November (13)
    • ►  October (13)
    • ►  September (9)
    • ►  August (17)
    • ►  July (7)
    • ►  June (4)
    • ►  May (6)
    • ►  April (9)
    • ►  March (2)
    • ►  February (3)
Powered by Blogger.

About Me

Unknown
View my complete profile