What Is a File Format

I was reading this blog post on the Microsoft Office binary file formats...to quote:

A normal programmer would conclude that Office’s binary file formats:

• are deliberately obfuscated
• are the product of a demented Borg mind
• were created by insanely bad programmers
• and are impossible to read or create correctly.

Hrm - deliberately obfuscated, a product of a demented borg mind, created by an insanely bad programmer, and impossible to read or write correctly...he must be talking about DSF!

But then Joel discusses why the formats are the way they are...tell me some of these don't apply to DSF:

• They were designed to be fast on very old computers. (Sort of true for DSF - load time on low-end hardware was a DSF design goal.)
• They were designed to use libraries. (True - DSF is a lot less weird when you consider the layered structure, and relatively easy to work with via DSFLib.)
• They were not designed with interoperability in mind. (I'll get to this later.)
• They have to reflect all the complexity of the applications.

In fact, the only reason Joel sites for byzantine Office formats that doesn't apply to DSF is:

• They have to reflect the history of the applications.

In our case DSF is a new invention of X-Plane 8, so you don't see a lot of past history artifacts in it.

The File Format as API vs. Data Container

File formats can serve two fundamentally different purposes:

1. A file format can specify a data container, used to save information for use later. DSF fits this category: the purpose of the DSF is to hold scenery data in a compact and fast-to-read format for X-Plane. (The other goal of DSF is to be structured so that we don't have to change it all the time.)
2. A file format can be an interchange standard between two programs, similar to a programming API. DSF2Text text files are like that.

You pay for every design goal with more engineering time - if DSF had to be an interchange standard and a high performance data container, it would be a lot harder to design and probably not do either job terribly well.

X-Plane Owns Its Own File Formats

One of the decisions I made early on in my work with the X-Plane scenery system was to not try to use open standards for most parts of the scenery system. (OBJ8, DSF, these are X-Plane specific inventions.) Does this make me a closed-source fascist? Perhaps, but the intent was not to lock people out of X-Plane.

Instead the goal was to separate the problems of interchange and data storage in the places where the two would come into conflict. What became immediately clear was that we couldn't meet all of our goals for performance, compatibility, and interoperability for a scenery file format with one giant scheme.

So instead we have DSF, whose goal is to be a cryptic but fast final step in the scenery pipeline, and we have DSF2Text and MeshTool, which take simpler text file formats, meant for interchange, and create DSFs.

The intended user of a MeshTool or DSF2Text text file is another program that is outputting its data format. Thus by forming a number of small links in a chain, no one file format is meant to be the "super-format" that has to do everything for everyone.