I've been working on this for a while now, its meant to use primarily for future personal projects once i have the time to write a parser for it, but in the meantime i want people to judge just how crazy I am
Yes, I know this may suffer from a severe case of scope creep, or some might use an existing solution, but what's the fun in that?
not at all, first, theres BSON for that already, and my format itself while perfectly capable of mimicking json has many more tools than just "JSON but binary"
Which application will use all of those tools? Or, in other words, why would anyone bother implementing that huge specification instead of coming up with a simple binary format tailored specifically around the data that needs to be serialized?
i believe you didn't read enough of the document (the spec itself is like 10 pages), but the answer is, you dont, you adopt the level of complexity you need
But the only way to get rid of the unnecessary complexity is to use a different format. If you know ahead of time what data you'll be sending between your applications, you don't need to include any information about how the data is structured, so including it is unnecessary complexity. If you don't know ahead of time what data you'll be sending between your applications, you need to include more information about how the data should be interpreted beyond just the data type associated with each field, so now you have the unnecessary complexity of transmitting the interpretation of each field separately from its type.
well yes, but you're looking at one use case, plus the overhead is minimal if you use more advanced features, a huge collection of complex entries can have a minimal overhead of just 1 byte, plus 8 bytes for a 64 bit entry count, plus 2 bytes to identify the schema, the entries themselves can be petabytes of data., for single entry stuff we have a specia case that has 1 byte, 2 bytes for a type, and then watever thing youre storing.
and thats just a single use case, plus it is stated pretty clearly that compact self description is one of the main points of the format, knowing the data shape you're transmitting requires very minimal overhead using this format if you use the right things, but there are other ways of using data as well.
of course self description has a limit, otherwise i need to create a universe every time i want to make a raspberry pie, but i dont think i fully underesood the question either
For example, you have a field in your serialized data that is a UTF-8 string. How does your application know what to do with that string? The shape of the data (a UTF-8 string) doesn't tell you how to interpret the data (perhaps a username, to use one of your examples).
yes, thats fine, thats by design, we dont carry ALL semantic meaning, you could pair ypur utf-8 with another type, cram it into an enum, etc, the application layer is the one responsible for interpreting the utf-8, not xDD, XDD responsability ends by saying "this is a utf-8 string, of this size, in this field. xDD doesn't even validate if its correct utf-8.
If it's up to the application layer to figure out how to interpret the data, why do you have so many redundant types? There's no difference between a Uint32 and a Count32 aside from how the application chooses to interpret the data.
you're correct, but let me reframe it, xdd is a common language to build stuff on top on, differing implementations may have differing alignments and paddings, but they all speak the same lamguahe at the end, thats why there are so many structurally equivalent types, so the data can be most precisely labeled with the types, its not the same to have uint8 uint8 entries than key8 counter8 entries, it helps to understand the data by having the extra type expressiveness even within the same applocation, even more for heavily heterogeneous data ive done some comparisons to other data formats (dont quote me on this one) and XDD can be on par or more efficient (size wise) representing the same data, and differing implementations can understand each other, not inmediatly, but the friction is much lower if two xdd systems want to exchange data even if their implementations differ wildly
My feedback is that it feels like it is written by an LLM.
Octocontrabass wrote: ↑Fri Aug 07, 2026 3:52 pm
If it's up to the application layer to figure out how to interpret the data, why do you have so many redundant types? There's no difference between a Uint32 and a Count32 aside from how the application chooses to interpret the data.
nope it isnt, in the gist there is already a revision history, and i also have some older versions of the document in my pc. i suspected this would happen anyways, if i were telling AI to do this i would ask the AI for the feedback as well, why would i bother to get real feedback from humans for an hallucinated format? it may be a shitty format and i could agree with that, but i spent a fair chunk of my time writing it, reorganizing it, changing single words so things become clearer, if you read it its pretty clear its not AI written anyways, i have the idea that you havent read the doc for more than 2 minutes
