Bringing serialization into the 21st century... bit by bit.
Project powered by:
The history and goals of s11n...
Most of this is pasted in from a mail sent to Ton Oguara in Februrary, 2004...Since 1997 i've been working on a "universal" boardgame interface, first in Java and now (since 2000) in C++ (QUB: http://qub.sourceforge.net). i learned, very painfully, through that effort that saving data is very easy, but loading anything but the most trivial data is amazingly hard. At one point a co-developer on that project, Rusty Ballinger, wrote a complete classloading layer for QUB and an XML serialization layer based upon the classloader, both based on the Qt libraries (http://www.trolltech.com/qt). (He coined the term "s11n" some time in 2003, AFAIK.) After a couple weeks of working with it, it revolutionized the way we developed the application. For example, we could then trigger the loading of objects simply by loading XML files. Having that framework in place was a godsend, saving us hundreds of development hours and opening up all kinds of new feature possibilities. Long-term, Rusty's model revolutionized the way i thought about application design. Though s11n and his "libFunUtil" share no code, libFunUtil is 100% the conceptual forefather of libs11n.
i learned C++ by writing QUB, and thus my C++ experience was limited to using the Qt library. At one point i decided i wanted to learn to be a "real C++ coder" and do UI-independent code (for example, by learning how to use the STL). A year or so after that i decided to rewrite the core of QUB to not be tied to the Qt libraries. Understanding fully that the heart of QUB's code-level successes lie in libFunUtil, i set out rewrite Rusty's serialization and classloading framework in a platform-neutral manner. i knew very early on that i had a LOT to learn before i could implement it the way i envisioned it, so i took my time thinking it over. My main goals were to take Rusty's model and:
- strip it of dependencies on 3rd-party libraries. i.e., re-implement it using only the STL and standard system libs. (Some common lib support, like zlib, is optional.)
- release it under a non-restrictive license (QUB is GPL because of its Qt heritage).
- refine the architecture, adding some levels of abstraction and making it more generic/maintainable/extendable.
- It should be able to save and load almost anything, hiding these details behind a unified interface.
- The lib should be as simple as possible to use, because i personally hate to use difficult-to-use libraries.
- The lib should be as non-intrusive on client code as possible. More generally, the overall client-side coding and maintenance costs of employing s11n in a project should be as low as practically possible.
- It must, of course, still provide clients with all the features C++ applications often need for serializing data. (My experience with libFunUtil would initially serve as the basis for determining "what clients normally need.")
- As few 3rd-party lib dependencies as possible. (Currently only standard system libraries are required.)
- The source base must be relatively platform-neutral (Unix platforms, anyway ;). That said, s11n shamelessly uses newer features of C++ which older compilers often won't support. (Should work with gcc 3.2+, and is normally developed with whatever gcc version ships on the current Suse Linux distribution.)
It took me the next 2 years or so to understand C++ and templates well enough to finally implement the classloader the way i wanted to. That is a *critical* component of the s11n architecture, and "classical" C++ does not offer a solution i deemed "acceptable" [i could go on for hours about the shortcomings of traditional classloaders!]. i read Modern C++ Design three or four times before i finally "got it" and realized an acceptable classloader implementation (based on factories). Once the classloader was implemented, developing the rest of the library went fairly quickly: in just a few days i had a working version (much different than the current code). In September and October of 2003 i could think of NOTHING but this library - i was completely enchanted by the idea of FINALLY being able to solve this problem i had spent YEARS pondering. i programmed approximately 200 hours in those 2 months (not including the standard 40-hour work weeks). i spent every ounce of energy tweaking it and writing the documentation. i've learned more about C++ from writing s11n than i ever knew the language offered. Up until late 2000 i thought C++ was beginning to go out of style, but now i'm a True Believer ;). (Now go read MC++D!)
At some point finally decided to register a domain and present it to the world. i've been coding since i was a child, but this project represents what i consider to be my first generically useful, non-niche-market code. Truth be told, i use s11n in all of my C++ code nowadays, and it has adapted well to the needs of the client apps in all use-cases to date. Well... whole truth be told: in some cases s11n was adapted in order to be able to adapt to the client. ;)
... so here it is ...
It's significantly different than i originally envisioned. In most ways it's much better than i had originally planned on, and in some ways... well, it needs some tweaking, and only experimentation and use will reveal what those tweaks need to be. There's still lots of experimentation left to do with the whole thing, but in general i think the model has proven to be remarkably effective at solving the primary problems it sets out to solve.
Nowadays (late Feb/early March 2004... is today Monday? Thursday?) the library is about to take another evolutionary step. A new, subtley-yet-significantly different, library will be rolled out in 0.7.0. It's still in its early stages, but appears to work well, and eliminates or reduces some limitations (and eases some formerly difficult-to-implement cases). The new model is more container/functor/algorithm-centric, allowing some parts (like the classloader) to be plugged in and out with relative ease. There's a whole lot to say about it, but i'd rather be coding ;)... let's just say that the ChangeLog for 0.7.0 is as long as my hair [updated Nov 2004].
...
Where is s11n headed? It has no "master plan", no specification which guides its development other than the requirement, "it must be useful to those who use it." So, as Zen says, "eat when hungry, sleep when tired." In other words...
We'll see what happens...