The marshalling cost for JSON is negligible. Yes, it might be a bit slower than GVariant for example, but only by some fractional linear factor. And on small messages (which D-Bus currently always is, due to message size constraints enforced by broker) the difference is impossible to measure. To a point it really doesn't matter, in particular as JSON parsers have been ridiculously well optimized in this world.
What does matter though are roundtrips. In Varlink there are much fewer required for typical ops than there are in D-Bus. That's because D-Bus implies a broker (which doubles the number of roundtrips), but also because D-Bus forces you into a model of sending smaller "summary" messages when enumerating plus querying "details" for each listed objects, because it enforces transfer rate limits on everything (if you hit them, you are kicked off the bus), which means you have to refrain from streaming too large data.
Or in other words: marshalling is quite an irrelevant minor detail when it comes to performance, you must look at roundtrips instead and the context switches it effects, instead.
Using JSON for this has two major benefits: the whole world speaks JSON, and modern programming languages typically pretty natively. And it's directly readable in tools such as strace. With a simple "strace" I can now reasonably trace my programs, which a binary serialization will never allow you. And if you tell me that that doesn't matter, then you apparently live in an entirely different world than I do, because in mine debuggability does matter. A lot. Probably more than most other things.
To save others the click: Their issues were simply that Swift has no fast JSON impl, and in Rust, when using serde (most popular library handling JSON marshalling), it leads to binaries getting a bunch bigger. That's it. So yeah, same perspective -- unless either of the above matter in your case (in 90%+ of cases they don't), JSON is just fine from a perf perspective.
Serde is a rather chunky dependency, it's not just a matter of binaries getting bigger, but also compile times being dramatically slower.
IMO CBOR would be a better choice, you aren't limited to IEEE 754 floats for your numeric types. Yeah, some (de/en)coders can handle integer types, but many won't, it's strictly out of spec. I don't think building something as fundamental to an OS as relying on out-of-spec behavior is a great idea. It will result in confusion and many wasted hours sooner or later.
> CBOR would be a better choice, you aren't limited to IEEE 754 floats for your numeric types.
The other side of this coin, of course, is that now you have to support those other numeric types :) My usual languages of choice somehow don't support "negative integers in the range -2^64..-1 inclusive".
I mean, you don't have to support those? You still would need something on the other end to produce that type of datatype, which can be documented that it will never happen: you're making an interface anyways. The problem is if you literally don't have the option to represent common datatypes it will be a problem, not a hypothetical one just because the encoding layer can support it. Those are different problems.
And JSON, technically, allows use of unlimited-precision fractions, but also allows implementations to set arbitrary limits (it actually does, you're not required to parse JSON numbers as doubles). So the situation is not really different from CBOR, isn't it? Just™ make both sides to agree to stick to some common subset (e.g. integers-in-int64_t-range-only for some fields) and you're done; no need to support double-precision floating point numbers.
Huh, I went and referenced the ECMA JSON spec and you're right that it treats numbers only as sequences of digits which would make these effectively the same problem
The marshalling cost might be negligible for come use cases, but the bandwidth usage definitely is not. I think the best interface description protocol is one where the serialization format is unspecified. Instead, the protocol describes how to specify the structure, exchange sequences, and pre/post-conditions. A separate document describes how to implement that specification with a certain over the wire format. That way the JSON folks can use JSON when they want (unless they are using large longs), and other folks can use what they want (I like CBOR).
I read the slides, and I found it refreshing that you said at the end: don't create per-language bindings for the libraries shipped with systemd, but simply use a JSON parser for your language. That underlined that you've specified a simple protocol.
Also, there have clearly also been several attempts over the years to make a faster D-Bus implementation (kdbus, BUS1), which were never accepted into the kernel. It makes a lot of sense to instead design a simpler protocol.
There is clearly also a cautionary take about how microbenchmarks (here, for serialisation) can mask systemic flaws (lots of context switches with D-Bus, especially once polkit had to be involved).
The danger I see is that JSON has lots of edge behavior around deserialization, and some languages will deserialize a 100 digit number differently. If the main benefit is removing the broker and the need for rate limiting - it could have been accomplished without using JSON.
You are writing this as if JSON was a newly invented thing, and not a language that has become the lingua franca of the Internet when it comes to encoding structured data. Well understood, and universally handled, since 1997.
A 100 digit number cannot be encoded losslessly in D-Bus btw, nor in the far majority of IPC marshallings on this word.
Having done systems-level OS development since 25y or so I never felt the burning urge to send a 100 digit number over local IPC.
Not that 100 digit numbers aren't useful, even in IPC, but typically, that's a cryptography thing, and they generally use their own serializations anyway.
You are writing this as if security was a newly invented thing. Having done systems level security development for 12 years, anything that can be produced maliciously will be. By using JSON, you've invented a new vulnerability class for malicious deserialization attacks.
Actually, not new. Earliest CVE I found was from 2017, which feels a decade later than it should be. I guess no one thought of pushing JSON over trusted interfaces, and probably for good reason.
> A 100 digit number cannot be encoded losslessly in D-Bus btw
I think the concern is that large numbers can in fact be encoded in JSON, but there is no guarantee that they will be decoded correctly by a receiver as the format is underspecified. So you have to cater for the ill defined common denominator.
Honestly, the only thing that surprises me is you're being pedantic, and encoding int64s as strings.
I know you know JSON is nominally only 53-bit safe, because JS numbers are doubles. But in practice I'd wager most JSON libraries can handle 64-bit integers.
What if varlink supported both JSON and a subset of cbor with the "cbor data follows" tag at the beginning (so the server can determine if it is json or cbor based on the beginning of the message)?
It would add a little complexity to the server, but then clients can choose if they want to use a human readable format that has more available libraries or a binary format.
As for strace, tooling could probably be added to automatically decode cbor to json, either as part of strace, or in a wrapper.
There could also be a varlink proxy (similar to varlink bridge) that could log or otherwise capture requests in a human readable format.
Not really. We use two text based formats for logging: BSD syslog, and systemd's structured logging (which is basically an env block, i.e. a key-value set, with some tweaks). Programs generate text logs, journald reads text logs hence. Programs that read from the journal get the text-based key/value stuff back, usually.
(Yes, we then store the structure log data on disk in a binary format. Lookup indexes are just nasty in text based formats).
Hence, not sure what the Journal has to do with Varlink, but any IPC that the journal does is text-based, and very nicely strace'able in fact, I do that all the time.
[Maybe, when trying to be a smartass, try to be "smart", and not just an "ass"?]
Sure the interface with the log might be text based, but my understanding is that the at rest format is binary and you need specialized tools to read it, standard unix grep is not going to cut it.
Although I use strace all the time, I hardly ever look at the payload of read and write calls, although I could see why it would be useful. But given a binary protocol it wouldn't be terribly hard to build a tool that parses the output of strace.
> [Maybe, when trying to be a smartass, try to be "smart", and not just an "ass"?]
thanks for the kind words and elevating the tone of the discussion.
is this true of future desktop uses cases where every basic function will cause a torrent of traffic on that? or you're talking from a server start/stopping services only point of view?
i honestly don't really get the angle of debugging via strace - i'd much rather prefer something more wireshark-like, where I can see all messages processes are sending to each other, since that would make it easier to decipher cases where sending a message to a service causes it to send other messages to its backends
I’ve worked with profiling code where the marshaling cost for JSON was the biggest cost. Namely it involved a heap allocation and copying a ton more data than was actually needed, and I ended up fixing it by turning the JSON into a static string and dropping the values in manually.
The systemd maintainers have probably done their due diligence and concluded that it isn’t an issue for their forseeable use cases, but it does lock everything in to doing string processing when interfacing with systemd, which is probably unnecessary. And you can’t trivially swap systemd out for something else.
systemd is so pervasive that it would be fine to add a binary-format-to-JSON translation ability into strace. That shifts the cost of debugging to the debug tools, rather than slowing down production code.
Doing any string processing tends to require a lot of branching, and branch mispredictions are most likely to slow down code. It also turns every 1-cyle load/store instruction into N-cycles.
String processing in C, which is what systemd and a lot of system tools are written, is pretty abysmal.
systemd is also non-optional, so if it turns out that it’s causing cache thrashing by dint of something generating a lot of small events, it’s not something you can do something about without digging into the details of your lowlevel system software or getting rid of systemd.
And it’s potentially just that much more waste on old or low-power hardware. Sure, it’s probably “negligible”, but the effort required to do anything more efficient is probably trivial compared to the aggregate cost.
And yeah, it may be better than D-Bus, but “it’s not as bad as the thing that it replaced” is pretty much the bare minimum expectation for such a change. I mean, if you’re swapping out things for something that’s even worse, what are you even doing?
I see there’s a TCP sidechannel, but why increase the complexity of the overall system by having two different channels when you could use one?
Dunno. This isn’t really an area that I work in, so I can’t say for sure it was the wrong decision, but the arguments I hear being made for it don’t seem great. For something fundamental like systemd, I’d expect it to use a serialization format that prioritizes being efficient and strongly-typed with minimal dependencies, rather than interoperability within the application layer with weakly-typed interpreted languages. This feels like a case of people choosing something they’re more personally familiar with than what’s actually optimal (and again, the reason I’d consider being optimal in this case being worth it is because this is a mandatory part of so many devices).
EDIT: Also, the reason that binary serialization is more efficient is because it’s simpler - for machines. JSON looks simpler to humans, but it’s actually a lot more complex under the hood, and for something fundamental having something simple tends to be better. Just because there’s an RFC out there that answers every question you could possibly have about JSON still doesn’t mean it’s as good as something for which the spec is much, much smaller.
JSON’s deceptive simplicity also results in people trying to handroll their own parsing or serialization, which then breaks in edge cases or doesn’t quite 100% follow the spec.
And Just because you’re using JSON doesn’t force C/++ developers to validate it, someone can still use an atoi() on an incoming string because “we only need one thing and it avoids pulling in an extra dependency for a proper json parser”, then breaks when a subsequent version of systemd changes the message. Etc. If the goal is to avoid memory safety issues in C/++, using more strings is not the answer.
There are a lot of problems with D-Bus, but that it wasn't using JSON wasn't one of them.
The tooling for D-Bus is terrible. It has composite types but they are very primitive, e.g. you can't name fields in a structure without extensions (there's like three different extensions for this.) A lot of the code generators can't actually handle real-world schemas.
Now I understand that D-Bus itself is also a problem (as in, the protocol design beyond serialization) but varlink looks like a very solid step backwards. In any single individual use case JSON serialization is unlikely to be a huge bottleneck, but in practice if this will be used for all kinds of things it will add up.
I really wish the Linux desktop would coalesce around something like Cap'n'proto for serialization.
P.S.: It'd probably be possible to shoehorn in multiple formats but I think that's a mistake too. Something like capnp will have a strong impedance mismatch with JSON; mixing formats with different degrees of self-describability is unwise. But then formats like MsgPack and BSON which are direct JSON analogues miss out on the best benefits of using binary serialization formats, which makes them almost the worst of both worlds, and you'd probably be better off with using a highly-optimized JSON library like simdjson.
I've suggested it before and the systemd folks didn't seem completely opposed to it. Also because cbor parser is already in the dependencies due to FIDO2 disk unlocking
I genuinely dislike CBOR. Formats which require me to calculate the length of the message before sending it lead to bad library design which often leads to easily compromised code. Add in an "indefinite length" option and you've got potentially unbounded client memory usage to watch out for. As if that wasn't enough you get extensible tags so the meaning of any message is entirely dependent on the context it was sent in.
It gives you a lot of decent protocol wire design and then flatly ignores everything we've learned about these types of designs in the last 3 decades. Be on the lookout for client libraries to slowly add in all of these checks as the vulnerabilities are discovered in them.
JSON's everything is "indefinite length". Also its implementations are wildly inconsistent [1] (is anyone going to read the code to figure out which is the blessed systemd parser?). Also it doesn't have integers. A lot of things with JSON will deteriorate into stringifying anyway, for example dates.
I'm a fan of DAG-CBOR, which is a deterministic subset of CBOR with most of the weird bits excluded (like indefinite lengths, multiple floating point formats, etc.), but it can still represent any sensible JSON object.
ASN suffers from some of the other problems regardless of the encoding.
I was thinking about this more and I think FTP actually had a lot of the right ideas here. You want two channels. One for fast interactive command messaging and a second one coordinate especially just for bulk transfers arranged over the command channel. The design flaw in FTP of putting these on two separate ports put it immediately in conflict with firewall best practices so I think it went mostly unnoticed that it fundamentally is a good arrangement.
What you want is one channel, sequenced packets, with a defined maximum message length that is negotiated at startup and never changes during the life of the channel. This should probably never be more than 65k. There should be a known length packet type, and an unknown length packet type, with any attempt to send more than the negotiated maximum triggering an error and disconnect.
If you do need to send more than the negotiated amount you should open a new connection, in a bulk transfer mode, that after the initial handshake, has no protocol and is merely an associated stream of bytes that are wholly uninterpreted by the middleware layer other than to help you associate it with the sequenced packet that requested its initiation.
You'd actually be able to use TCP (with DSCP even), mostly avoid head of line blocking and multiplexer latencies, and have a reasonable security guarantee in the sequenced packet mode, and never have a protocol which pretends that 4GB strings in the middle of a packet are necessary or even a good idea to "support."
The downfall of this is that it would be much harder to implement it on a serverless architecture and would be nearly as complicated as a protocol as WebSocket ends up being. It might be worth playing with as a concept anyways.
You would need a pretty intense QoS system built for this if you are using something like TCP for this using two different connections. Underneath it would still use IPv4/6 packets and you only have so much bandwidth. The bulk transfer channel might stall the transfer of control packets temporarily.
You would need to tightly control and multiplex the messages yourself, needing to do this in one connection or using something like TCP priority flag.
Personally I just think that TCP is a shitty protocol for building applications. In almost every use case either UDP or SCTP are a better choice. With raw UDP datagrams you aren’t guaranteed delivery but it’s great for the kind of telemetry where the last message in is what matters. You can also build quite flexible stuff on top of it.
SCTP gives you in sequence reliable datagrams and congestion control. This means you don’t have to devise message length communication into your application layer protocol and can rely on your transport. It also has multiplexing built right in. If it had a built in checksum it would truly be ideal. Not sure if something like secure communication really belongs at this level but if it had that I double we would ever use anything else. From SCTP’s Wikipedia page:
> SCTP applications submit data for transmission in messages (groups of bytes) to the SCTP transport layer. SCTP places messages and control information into separate chunks (data chunks and control chunks), each identified by a chunk header. The protocol can fragment a message into multiple data chunks, but each data chunk contains data from only one user message. SCTP bundles the chunks into SCTP packets. The SCTP packet, which is submitted to the Internet Protocol, consists of a packet header, SCTP control chunks (when necessary), followed by SCTP data chunks (when available).
I think the protocol on which PDUs are exchanged on is a bit orthogonal to the PDU protocol itself here, but the split-mode networking protocols (FTP, IRC DCC, mosh, SIP, etc) always do end up getting less used because establishing two different connections is a bigger burden and there really needs to be a great advantage to doing so, and often require other considerations (better connection tracking, TURN/ICE servers, etc).
For a localhost/UNIX domain protocol, it might work since most of these considerations are significantly reduced.
As to DSCP, it might be useful across controlled networks but in general, in my experience it's not really ever honored broadly.
Back on the original topic, when I was building this for my Linux distribution I ended up just using basic JSON with TLS (for client certificate authentication) though since having authentication for remote management was a goal, and once I was already having to perform a TLS negotiation then the PDU consideration for performance wasn't something to really spend too much time on.
More complex buffer management in non-garbage collected languages.
man 3 cmsg
For a glimpse into the darker corners of hell. At least you get to grip it there. In GCed languages you have to hope the library designer gave you reasonable controls for timeouts, aborting large data transfers, and releasing buffers efficiently in a bursty network environment.
See all the controls any HTTP server gives you for this. "Maximum header size", "Maximum header(s) size", "Maximum Body Size", "Request Header Timeout", "Total Request Timeout."
None of those were added by default or by tasteful library authors. They were all added to answer to specific emergent security vulnerabilities that were discovered and then were demanded by the users.
I mean conversely every second week on HN someone complains about "inefficient modern development practices" and this is the sort of minor thing which would be trotted out as an example.
So I'd argue it's not an unreasonable question (although I lean closer in: better types then JSON is the problem, since a decent text serialization format is always going to be needed for debugging and development).
Data format is absolutely important for operating system IPC. JSON is raw text. Messages occupy more RAM than a binary format, which increases your working set, and it requires at least some nontrivial parsing. This ultimately limits the number of messages you can send per second.
Personal opinion: don't put floating point numbers into an API. If you really want real numbers passing them as decimal string is actually sensible. This can be done without using JSON, of course. Can also do fixed precision as well, which will store reasonably well into a VLQ.
OTOH this is pretty cherry picked. It is questionable if you really need floating point numbers if your numbers are all this small and low precision, but in actuality unless this comprises your entire message, you'll probably still lose to JSON overhead eventually anyways, whereas non-self-describing serialization like capnp has effectively no overhead by default. The advantage of encoding as doubles is that it is predictable and won't explode into taking massively more, its always the same size per number. If you want it to be smaller for some reason, compression is always an option, though I suspect for IPC it's the wrong thing to be concerned about.
Or an ASN.1 DER encoded sequence -- ASN.1 is more common than CBOR (being in a lot of PKI stuff) and there are already generators for DER (or JSON or XML...) for a given schema.
Which makes me wonder how performance sensitive is the task D-Bus handles?
Most task done over it seems to either fall into the "from time to time but not really performance sensitive at all" category or the "a mass of system events which can become huge" category.
> miss out on the best benefits the best benefits of using binary serialization format
I disagree (wrt. MsgPack), the biggest benefit is of binary formats is that you can ship binary blobs without any encoding (mainly space) overhead while for JSON this can easily be ~33%+ space overhead. And even for just e.g. numbers the space overhead can be bad. I'm not sure what you see as the biggest benefit but not having a minimalist structure isn't a property specific to binary formats (through more common there, and MsgPack doesn't require field names to be included). And bit fiddling micro space optimizations aren't that grate/as much a benefit as drawback. And as a system bus handles (emits) potentially a _ton_ of events saving 33% can matter I think.
A lot, D-Bus in a way has the same role as XPC, COM and Binder on other platforms, so many desktop applications use D-Bus for their plugins, now imagine the traffic of something like Thunar doing D-Bus calls for generating image previews for the new vacations album someone just created.
I've done similar things in the past (json over udp-to-localhost, with a schema description used to generate the code for both parsing and generating on each end). It's a totally reasonable point in the design space, and I literally never saw a reason to revisit the decision to use json for that system.
You'd do that because everything under the sun knows how to generate and parse json. Performance looks like a total non-issue for this use case. The messages are tiny, local and infrequent. You wouldn't want to do this on some kind of a high performance application on the data plane, but on the control plane it's totally fine.
Even if you expect that all production use cases will use some kind of higher level library that does schema validation, it can still be quite nice during debugging to a) be able to inspect the messages on the wire when debugging, b) to be able to hand-write and inject messages.
Fair. What I meant is that the performance delta between JSON and any other interchange data format should be a non-issue in this context. I would be quite surprised to hear that any performance problems systemd has with D-Bus was with serialization/de-serialization.
> performance delta between JSON and any other interchange data format should be a non-issue
it's hard to say, for most D-Bus taks perf. is an absolute non issues, but still you have bus-1 and all kind of tries to make D-Bus perform faster, reason is there are edge cases where it matters
and even if the marshaling speed might not matter the size can, like if something goes really wrong and tens of thousand of system events get spammed the difference between the memory usage of in-flight messages of a compact format vs. json can make the difference between memory pressure effectively killing your server and you recovering reliable
though what is an open question is how relevant that is in pracive
and that is quite hard to say without inside information for companies running systems where such a thing could happen on a scale where such a thing does happen (e.g. google)
The problem isn't JSON per se. Parsing JSON is fast. The problem is the simplistic interaction and data model that's just going to move complexity from universal IPC layers to ad hoc application logic that every person has to learn individually. Terrible.
Lennart pointed out the fact you can see readable messages via strace to be a benefit of json. If their average message size is small and they don't expect to ever need high volume or large messages, then it's not really likely to be a problem in practice. He also pointed out that they waste far more cycles today on context switches (he suggested something on the order of 6 per IPC message).
Even if they eventually come to the conclusion they need something more performant in the future, it's probably still a net win to make the switch today. It's also possible it turns out fine for their use case.
I personally would think they would want to benefit from an IPC system that includes provisions for sending file descriptors.
Modifying the existing journal really sounds like the wrong solution. Just "journalctl --rotate" the file and throw out the one with accidental PII. Journal files are not great for long-term storage or search. You can export the old file and filter out manually if you really want to preserve that one https://www.freedesktop.org/wiki/Software/systemd/export/
In what situations is it a harder problem than this?
> I once typed an SSH password in the username field, and the only way to erase that was to erase all the logs. So this has some significant downsides.
I hope this was a personal system. Changing logs in this manner would have almost certainly led to your dismissal anywhere I ever worked. This anecdote just re-enforces the need for Forward Secure Sealing.
Varlink is designed for IPC, so the expected data rate is maybe hundreds of messages per second in the worst case.
JSON can be parsed at 3 Gigabytes/second [0]. Even unoptimized stdlib parsers in scripting languages get 50 MB/sec.
That's more than enough, standard library support is much more important for a project like this.
And if there is ever a reason to send tar archive or video data, there is always "upgrade" option to switch to raw TCP sockets, with no JSON overhead at all.
It's the lingua-franca serialization format that has bindings to every programming language already, doesn't require agreeing on the payload layout beforehand, is beyond overkill for the task being asked of it, and is human debuggable over the wire.
I'm begging you to read the format it's replacing. https://dbus.freedesktop.org/doc/dbus-specification.html Ctrl+f Type System. JSON is beautiful, elegant, and minimal by comparison. yyyyuua(yv) It's like the printf message protocol but worse.
it isn't, fundamentally can't be as it has no support for binary blobs
and base64 encoded strings are a pretty terrible solution for this
and the space overhead of many small number can also be pretty bad
and while a lot of IPC use cases are so little performance sensitive that JSON is more then good enough and large blobs are normally not send over messages (instead you e.g. send a fd) there are still some use-cases where you use IPC to listen to system events and you have a situation where a pretty insane amount of them is emitted, in which case JSON might come back and bite you (less due to marshaling performance and more due to the space overhead which depending on the kind of events can easily be 33%+).
But what I do not know is how relevant such edge cases are.
Probably the biggest issue might be IPC system mainly used on 64bit systems not being able to properly represent all 64 bit integers .... but probably that is fine too.
The protocol supports “upgrading” requests. If your service relies on sending large binary blobs (over this? Why?), then it doesn’t have to be done with JSON.
For example, the metadata of the blob could be returned via JSON, then the request is “upgraded” to a pure binary pipe and the results read as-is.
binary blobs is just the biggest example and was only mentioned in relation to the "lingua franca" argument, many other common things are also larger in JSON. Only if you have many larger not escaped utf-8 strings does this overheads amortize. E.g. uuids are something not uncommonly send around and it's 17 bytes in msgpack as a bin value and 38 bytes in json (not inlcuding `:` and `,` ). That 211% the storage cost. Multiply it with something going on and producing endless amounts of events (e.g. some unmount/mount loop) and that difference can matter.
Through yes for most use cases this will never matter.
I get your point, but you have to understand that for every second you’ve spent writing that comment, globally hundreds of millions of HTTP responses have been processed that contain UUIDs of some kind.
Yes, there’s a more optimal format than hex-encoding UUID values. However it simply does not matter for any use case this targets.
16 bytes vs 38 bytes is completely meaningless in the context of a local process sending a request to a local daemon. It’s meaningless when making a HTTP request as well, unfortunately.
I’d have loved Arrow to be the format chosen, but that’s not lowering the barrier to entry much.
> JSON is beautiful, elegant, and minimal by comparison.
True, but binary formats like CBOR or MessagePack are also that and efficient and have support for binary strings. This JSON is human readable thing is the nerverending argument of high level programmers. Anyone can read hex given enough time.
JSON isn't simple. It's simplistic. It's too simple. By refusing to acknowledge the complexity of the real world at the protocol layer, varlink shifts the burden of dealing with the complexity from one transport layer to N ad hoc application layers, making us all worse off.
Do you mean https://varlink.org/Interface-Definition? JSON is just the marshaling format. All the language bindings don't need a fancy parser outside of json.loads. I wish the interface definitions themselves had been specified in JSON as well.
Peep the Python interface parser: https://github.com/varlink/python/blob/master/varlink/scanne.... This is the quality you can expect for an ad-hoc format, which is to say, pretty bad. Making the hardest part farming out to a parsing library you already have and is a built-in in your language is a no-brainer choice.
It just seems strange that the message rate is assumed to be in some very specific range where the performance of D-Bus is a problem (mentioned in the slides) yet the performance overhead of JSON is not a problem.
There was a Varlink talk[0] a few days ago at All Systems Go, in that talk, Lennart mentioned that JSON is unfortunate (primarily due to no 64 bit ints) but it has the surprising benefit of being able to understand the bus messages when using `strace` for debugging.
With this logic (oft repeated) we should be sending TCP as JSON. { sourcePort: 443, destinationPort: 12345,...}
Debugability is important but the answer is to build debugging tools, not to lobotomise the protocol for the vanishingly tiny fraction of packets that are ultimately subject to debugging.
I love your TCP over JSON idea. Maybe we can implement layer 2 that way as well, with CSMA/CD: {startTime:2024-09-29T13:56:21.06551, duration:00:00:00.0024, packet: …}
Oh my God. I'm genuinely struggling to avoid using an endless stream of profanity here. If our problem is that our observability tools suck, the solution is to improve these tools, not mutilate our IPC mechanism to accommodate the limitations of these tools.
The observability tools in question (strace) follow the UNIX tradition of displaying and processing data as text by default. I don’t think that means that they suck.
I’ll go even stronger than that: IPC and RPC should prefer plaintext-representable forms by default and only abandon them for binary once the real world costs of the textually-representable protocol are found to be unacceptable and unmitigateable.
The benefit of being able to use pre-existing introspection tools that were not designed with your protocol in mind—-and use those tools without extensive configuration—-is huge.
I think the existence of bad textual formats (e.g. JSON) and the presence of largely-orthogonal-to-binaryness useful affordances in popular binary formats (e.g. schema validation and strong typing in Protobuf) muddies the underlying truth: textual-representability-by-default is rarely costly and often a huge boon to protocol implementors and protocol debuggers.
I do not understand where the 64bit integers not working with JSON comes from.
JSON the format had no limit on integer size. And all Java JSON libraries I know can handle arbitrary prevsion integers (BigInt) and 32/64bit int/long types when serializing and deserializing.
Quick googling shows that also JavaScript has proper support for 64bit integers with their BigInt type, and it can be used to deserialize incoming data correctly with the default parser, albeit requiring a bit more work to annotate the fields.
Personally I often explicitly make sure that the integers I return in trust environments as identifiers in REST APIs are by default longer than 52bits so that buggy parser libraries are caught early.
The number type in JSON is 64 bit float, limiting integers without loss of precision to 2⁵³-1.
BigInt is a new concept and not technically supported. So whether it works in your random library of choice is probably a potshoot.
"Use within JSON: Using JSON.stringify() with any BigInt value will raise a TypeError, as BigInt values aren't serialized in JSON by default. "
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
If you look at a parser matrix, flouting this rule is asking for trouble. Which is why the recommendations on MDN for bigint recommend an explicit parsing from a string as needed.
(I'm genuinely curious why this is being downvoted. Does someone out there feel so strongly that because an arbitrary string of characters technically can be parsed as any infinitely large number, that it's wise to ignore RFCs and dozens of existing implementations when making use of JSON as a common interchange format?)
You are mixing up two different standards here, JSON and I-JSON. I-JSON is a subset of JSON with stricter requirements.
JSON recommends (for interoperability) that numbers be limited to those representable as IEEE 64-bit binary floats-but does not require that. A document which ignores that recommendation is still officially valid JSON. By contrast, I-JSON, as a subset of JSON, upgrades that recommendation to a requirement, so documents which ignore it are not valid I-JSON
Yes, they are different standards, but the purpose of I-JSON was specifically to solve interoperability problems. Those interoperability problems are due to the looseness of the original spec.
The others do not forbid larger numbers but note that doing that will make for bad times, and if you look at JSON parsing matrices (say, the "Parsing JSON is a Minefield" one), you can see that the majority of implementations do not do allow larger.
So. float64 is a defacto standard and I-JSON is simply clarifying this. Given the main purpose of JSON is an interoperable data exchange format it would be a very bad idea to do otherwise.
there is more then one JSON standard and while the JS in JSON stands for JavaScript it doesn't directly means it is using JS types as confusing as it might seem.
the initial definition of JSON by itself has no limit on integer size at all, JSON with number which can only be represented with BigInt _totally is a thing_ just not so much in the Js ecosystem. Furthermore systems accidental expecting full i64 or u64 number ranges are as much a thing, and this leading to subtle bugs the moment a different (de-)serializer is used is another issues.
jq couldn't handle them until relatively recently. This isn't a few bad parsers. You can't assume a json parser will handle bigints correctly and when you're angling to be low-level plumbing and work with every language and software that hasn't been recompiled in years you have to be pretty conservative in what you send.
Yeah, and it isn't like jq is being incorrect in this, the JSON RFCs recommend not using values that can't be represented as a 64 bit float (pasted links to a few RFCs in another response). So if you want to represent a large number safely, better to put it in a string, where special handling can be done after the initial JSON parse without loss of information.
If you fully work within a trusted environment, why bother with JSON? Use your favorite binary serialization with codegen.
The whole point of JSON is almost every programming language can read and write it - and if you want this to be the case, stringify anything unusual, like large integers.
Not a fan of JSON per se for this type of messaging, but I don't think binary parser+validator combinations are that much faster. Plus, this forces C(++) developers to actually validate the data they're exchanging rather than casting raw structs from data buffers, which solves a huge vulnerability. Then again, the lack of 64 bit integers makes the protocol design comically unsuited.
I don't think the serialization/deserialisation overhead matters much when also taking into account the bus connection overhead. I would've preferred a better data exchange format but out of all problems I have with Linux IPC, the performance of JSON is the least of my worries.
The 64bit issue is certainly an issue, but very much overblown.
First of all, in systemd, which is a heavy D-Bus user, we effectively IRL only send integers > 2^53 when we use UINT64_MAX as a special "niche" marker meaning "unlimited"/"unset"/"undefined". But the thing is that JSON has a proper concept for this: the "null" value (or simply not specifying a specific JSON field). Hence, in reasonably clean APIs this is mostly a non-issue.
That said, sd-varlink/sd-json (i.e. systemd's implementation of it), of course is 64bit signed and unsigned integer clean when it processes JSON. More-over it automatically handles if you do what the various specs on the internet suggest you do if you have an integer > 2^53: you encode it as decimal value as a string.
Would it be better if JSON would have been more precise on this, yes. Is it a big issue? No, not at all.
To be honest I don't see why they don't use the existing IPC mechanism created for Android - binder and ashmem. It has been tested and developed in real world scenarios for over a decade by now.
Same reason Android (or PalmOS) didn't decide to use any of the existing IPC mechanisms, and why then Luna/webOS also didn't decide to use Binder even though they had experience with it and reinvent something else.
That urge that forces every developer to reinvent RPC every other week.
Binder solves a real-world problem — priority inversion. It is also the only IPC on Linux that allows to perform a blocking call from process A to process B (and have process B synchronously call back to process A if needed).
D-Bus, Varlink and other "IPC solutions" on top of sockets can not do those things. Android uses synchronous Binder calls for 90% of it's application APIs, so there is clearly a real use case for it.
Incidentally, Binder itself does not specify message format. A message is a simple memory buffer. Android runtime uses a custom binary serialization format, but you can use JSON or whatever. Binder is an alternative to sockets/pipes, not D-Bus or other wrappers on top of them.
Binder was not even designed for Linux. The primary reason it is used is that (at the time) they wanted to abstract from Linux. "They" here is not Android/Google.
Binder also specifies a message format -- even if not fully, since the kernel is going to peek into your message for things like FDs, binder objects, etc. Or your userspace is going to need to "special treat" these fields somehow.
It competes in the same space as D-Bus no doubt. If the story of Android had been different, with the companies spinning off some years later, it may have very well ended up using Luna RPC (https://www.webosbrew.org/pages/luna-service-bus.html ) (which is practically indistinguishable from D-Bus, bar the JSON) or anything else, really.
> D-Bus, Varlink and other "IPC solutions" on top of sockets can not do those things.
WTF? Sure you can do on top of even pipes. Even XDR could...
"Synchronous" is a very loose word here if you mean to be interrupted in the middle of a RPC call, anyway.
Binder was invented at Be, Inc. for BeOS, and many Be refugees joined Android in the early days. (I just learned this recently on HN, so congratulations!)
> WTF? Sure you can do on top of even pipes. Even XDR could...
Of course, you "can". Implement message-based communication on top of streaming. Emulate blocking calls on top of non-blocking socket API. Implement authentication via ancillary data (try not to throw up in process). Use some clever tricks to solve priority inversion. Somehow distinguish your fds from all others file descriptors to ensure that no confused deputy problems arise.
Congratulations! You have reached feature parity with Binder.
What is your point here? You claimed that "other IPC solutions" _could not_ do this on top of sockets, but as I've shown and now you admit, you definitely _can_. Obviously I'm not suggesting you roll this by hand, nor claiming that a plain UNIX pipe is a fully featured RPC system; just that there's a million RPC systems out there that do it and don't need anything else than sockets.
I don't know what binder does exactly, but rendez-vous synchronization is surprisingly hard to do efficiently on POSIX systems without excessive context switches.
Varlink was specifically designed to be available as early as possible when the kernel is booting and requires virtually nothing apart from good old sockets. But yes, definitely getting yet another one vibe from this.
Then again, Dbus is not really what I would call an incredible piece of software - we are very much in the adequate territory if even - so anything shaking up the status quo can’t be too bad.
It's probably the first time in 14 years of systemd I appreciate a technical choice coming from the systemd team.
It's a small format, it has limited pitfalls, it's fast to marshal and unmarshal (and with simd gets even more fast), it's deflate-friendly, would performance be an issue they can pack that through CBOR or msgpack or what-have-you.
Lets take it a level up. I'm still not sure why we even need a message bus in there in the first place. The whole Linux boot, init, systemd pile is a Rube Goldberg machine that is very difficult to understand at the moment. The only reason I suspect most people aren't complaining is that our abstraction level is currently a lot higher (docker / kubernetes etc) and machines are mostly ephemeral cattle and few people go near it.
As for JSON, please don't even start me on that. It is one of the worst serialisation decisions we ever made as a society. Poorly defined and unreliable primitive types, terrible schema support and expensive to parse. In a kernel, it does not belong! Give it a few months and varlink will be YAML over the top of that.
I think it doesn't have to be a message bus per-se, that design decision is mostly just because it's convenient. Even D-Bus can actually be used without the bus, if you really want to.
D-Bus is just a bus so it can solve a bunch of different IPC problems at the same time. e.g. D-Bus can handle ACLs/permissions on RPC methods, D-Bus can handle multiple producers and multiple consumers, and so on. I think ultimately all that's really needed is IPC, and having a unified bus is just to allow for some additional use cases that are harder if you're only using UNIX domain sockets directly.
If there are going to be systemd components that have IPC, then I'd argue they should probably use something standard rather than something bespoke. It's good to not re-invent the wheel.
Not that I think Varlink is any better. It seems like at best it's probably a lateral move. I hope this does not go forward.
> If there are going to be systemd components that have IPC, then I'd argue they should probably use something standard rather than something bespoke. It's good to not re-invent the wheel.
This is my point.
My favourite DBus situation a number of years ago was a CentOS 7 box that reboot stopped working on with a cryptic DBus error that no one has ever seen before. I had to sync it, power cycle the node from the ILO card and cross my fingers.
I really don't give a shit about this. I just wanted to run my jobs on the node, not turn into a sysadmin due to someone else's dubious architectural decisions.
Yes but the systemd developers don't want to implement their own protocols with e.g. ACL checking, and given some of their track record I kind of think you don't want them to, either. I'm pretty sure the error conditions would be even more bespoke if they "just" used UNIX domain sockets directly. Don't get me wrong, there's nothing particularly wrong with UNIX domain sockets, but there's no "standard" protocols for communicating over UDS.
This is systemd we’re talking about. A service manager that already mucks with mount namespaces.
It would be quite straightforward to map a capability-like UNIX socket into each service’s filesystem and give it a private view of the world. But instead…
> Public varlink interfaces are registered system-wide by their well-known address, by default /run/org.varlink.resolver. The resolver translates a given varlink interface to the service address which provides this interface.
…we have well known names, and sandboxing, or replacing a service for just one client, remains a mess. Sigh.
> It would be quite straightforward to map a capability-like UNIX socket into each service’s filesystem and give it a private view of the world. But instead…
Can you link to your PR where you solved the problem?
Well there sort of is but people don't tend to know or use it. If it's within the same machine and architecture, which should be the case for an init system, then a fixed size struct can be written and read trivially.
C structs are a terrible serialization format, since they are not a serialization format at all. Nothing guarantees that you will get consistent struct behavior on the same machine, but also, it only really solves the problem for C. For everything else, you have to duplicate the C structure exactly, including however it may vary per architecture (e.g. due to alignment.)
And OK fine. It's not that bad, most C ABIs are able to work around this reasonably OK (not all of them but sure, let's just call it a skill issue.) But then what do you do when you want anything more complicated than a completely fixed-size type? Like for example... a string. Or an array. Now we can't just use a struct, a single request will need to be split into multiple different structures at the bare minimum.
And plus, there's no real reason to limit this all to the same machine. Tunneling UNIX domain sockets over the network is perfectly reasonable behavior and most* SSH implementations these days support this. So I think scoping the interoperability to "same machine" is unnecessarily limiting, especially when it's not actually hard to write consistent de/serialization in any language.
* At least the ones I can think of, like OpenSSH[1], Go's x/crypto/ssh[2], and libssh2[3].
BTW, Lustre's RPC system's serialization is very much based on C structs. It's receiver-makes-right to deal with endianness, for example. It's a pain, but it's also fast.
Making an RPC serialization system that is zero-overhead i.e. can use the same format on the wire as it does on disk is not a terrible idea. Capnp is the serialization format that I've been suggesting as a potential candidate and it is basically just taking the idea of C structs, dumping it into it's own schema language, and adding the bare minimum to get some protobuf-like semantics.
Well, Lustre RPC doesn't use on-disk data structures on the wire, though that is indeed an interesting idea.
In Lustre RPC _control_ messages go over one channel and they're all a C structure(s) with sender encoding hints so the receiver can make it right, and any variable-length payloads go in separate chunks trailing the C structures.
Whereas bulk _data_ is done with RDMA, and there's no C structures in sight for that.
Capnp sounds about right for encoding rules. The way I'd do it:
- target 64-bit architectures
(32-bit senders have to do work
to encode, but 64-bit senders
don't)
- assume C-style struct packing
rules in the host language
(but not #pragma packed)
- use an arena allocator
- transmit {archflags, base pointer, data}
- receiver makes right:
- swab if necessary
- fix interior pointers
- fail if there are pointers
to anything outside the
received data
- convert to 32-bit if the
receiver is 32-bit
(That's roughly what Lustre RPC does.)
As for syntax, I'd build an "ASN.2" that has a syntax that's parseable with LALR(1), dammit, and which is more like what today's devs are used to, but which is otherwise 100% equivalent to ASN.1.
Out of curiosity, why not use offsets instead of pointers? That's what capnp does. I assume offset calculation is going to be efficient on most platforms. This removes the need for fixing up pointers; instead you just need to check bounds.
It's more work for the sender, but the receiver still has to do the same amount of work as before to get back to actual pointers. So it seems like pointless work.
Having actual interior pointers means not having to deal with pointers as offsets when using these objects. Now the programming language could hide those details, but that means knowing or keeping track of the root object whenever traversing those interior pointers, which could be annoying, or else encoding an offset to the root and an offset to the pointed-to-item, which would be ok, and then the programming language can totally hide the fact that interior pointers are offset pairs.
I've a feeling that fixing up pointers is the more interoperable approach, but it's true that it does more memory writes. In any case all interior pointers have to be validated on receiving -- I don't see how to avoid that (bummer).
Note within the domain of this problem was the point. Which means on the same machine, with the same architecture and both ends being C which is what the init system is written in.
You are adding more problems that don't exist to the specification.
As for strings, just shove a char[4096] in there. Use a bit of memory to save a lot of parsing.
> You are adding more problems that don't exist to the specification.
D-Bus does in fact already have support for remoting, and like I said, you can tunnel it today. I'm only suggesting it because I have in fact tunneled D-Bus over the network to call into systemd specifically, already!
> As for strings, just shove a char[4096] in there. Use a bit of memory to save a lot of parsing.
OK. So... waste an entire page of memory for each string. And then we avoid all of that parsing, but the resulting code is horribly error-prone. And then it still doesn't work if you actually want really large strings, and it also doesn't do much to answer arrays of other things like structures.
Can you maybe see why this is compelling to virtually nobody?
Even being run on the same machine doesn't guarantee two independent processes agree on C struct layout compiled from the same source. For one, you could have something as simple as one compiled for 32bit, one 64, but even then compiler flags can impact struct layout.
> As for strings, just shove a char[4096] in there.
For the love of God, use a proper slice/fat pointer, please.
Switching over to slices eliminates 90%+ of the issues with using C. Carrying around the base and the length eliminates a huge number of the overrun issues (especially if you don't store them consecutively).
Splitting the base and the offset gives a huge amount of semantic information and makes serialization vastly easier.
Broadly, I agree with you. C strings were a mistake. The standard library is full of broken shit that nobody should use, and worse, due to the myriad of slightly different "safe" string library functions, (a completely different subset of which is supported on any given platform,) which all have different edge cases, many people are over-confident that their C string code is actually correct. But is it? Is it checking errors? Does your function ensure that the destination buffer is null-terminated when it fails? Are you sure you don't have any off-by-one issues anywhere?
Correct as you may be though the argument here is that you should just write raw structs into Unix sockets. In this case you can't really use pointers. So, realistically, no slices either. In this context a fixed-size buffer is quite literally the only sensible thing you can do, but also, I think it's a great demonstration of why you absolutely shouldn't do this.
That said, if we're willing to get rid of the constraint of using only one plain C struct, you could use offsets instead of pointers. Allocate some contiguous chunk of memory for your entire request, place struct/strings/etc. in it, and use relative offsets. Then on the receiver side you just need some fairly basic validation checks to ensure none of the offsets go out of bounds. But at that point, you've basically invented part of Cap'n'proto, which begs the question... Why not just use something like that instead. It's pretty much the entire reason they were invented.
Oh well. Unfortunately the unforced errors of D-Bus seem like they will lead to an overcorrection in the other direction, turning the core of our operating system into something that I suspect nobody will love in the long term.
> But at that point, you've basically invented part of Cap'n'proto
The only problem I have with Cap'n Proto is that the description is external to the serialization. Ideally I'd like the binary format to have a small description of what it is at the message head so that people can process messages from future versions.
ie. Something like: "Hmm, I recognize your MSG1 hash/UUID/descriptor so I can do a fast path and just map you directly into memory and grab field FD. Erm, I don't recognize MSG2, so I need to read the description and figure out if it even has field FD and then where FD is in the message."
I thought about this for a bit. I think largely to do things with messages you don't know about is probably a bad idea in general; writing code that works this way is bound to create a lot of trouble in the future, and it's hard to always reason about from every PoV. However, there are some use cases where dealing with types not known at compile-time is useful, obviously debugging tools. In that case I think the right thing to do is just have a way to look up schemas based on some sort of identity. Cap'n'proto is not necessarily the greatest here: It relies on a randomly-generated 64-bit file identifier. I would prefer a URL or perhaps a UUID instead. Either way, carrying a tiny bit of identity information means that the relatively-niche users who need to introspect an unknown message don't cause everyone else to need to pay up-front for describability, and those users that do need introspection can get the entire schema rather than just whatever is described in the serialized form.
It's better to design APIs to be extensible in ways that doesn't require dynamic introspection. It's always possible to have a "generic" header message that contains a more specific message inside of it, so that some consumers of an API can operate on messages even when they contain some data that they don't understand, but I think this still warrants some care to make sure it's definitely the right API design. Maybe in the future you'll come to the conclusion it would actually be better if consumers don't even try to process things they're not aware of as the semantics they implement may some day be wrong for a new type of message.
> I think largely to do things with messages you don't know about is probably a bad idea in general
Versioning, at the least, is extremely difficult without this.
Look at the Vulkan API for an example of what they have to do in C to manage this. They have both an sType tag and a pNext extension pointer in order for past APIs to be able to consume future versions.
But how do you know that the field called "FD" is meaningful if the message is a totally different schema than the one you were expecting?
In general there's very little you can really do with a dynamic schema. Perhaps you can convert the message to JSON or something. But if your code doesn't understand the schema it received, then it can't possibly understand what the fields mean...
Yeah, how will number/float serialization go? Are we going to serialize them as strings and parse them? That abstraction isn't handled the same way across multiple languages.
There are a world of serialization formats that can offer a similar interoperability story to JSON or JSON-but-binary formats. And sure, implementing them in every language that someone might be interested in using them in might require complication, but:
- Whatever: people in more niche languages are pretty used to needing to do FFI for things like this anyhow.
- Many of them already have a better ecosystem than D-Bus. e.g. interoperability between Protobuf and Cap'n'proto implementations is good. Protobuf in most (all?) runtimes supports dynamically reading a schema and parsing binary wire format with it, as well as code generation. You can also maintain backwards compatibility in these formats by following relatively simple rules that can be statically-enforced.
- JSON and JSON-but-binary have some annoying downsides. I really don't think field names of composite types belong as part of the ABI. JSON-like formats also often have to try to deal with the fact that JSON doesn't strictly define all semantics. Some of them differ from JSON is subtle ways, so supporting both JSON and sorta-JSON can lead to nasty side-effects.
Maybe most importantly, since we're not writing software that's speaking to web browsers, JSON isn't even particularly convenient to begin with. A lot of the software will be in C and Rust most likely. It helps a bit for scripting languages like Python, but I'm not convinced it's worth the downsides.
I don't know how to tell you this, but, you don't need to implement an RPC protocol in bash, nor do you need FFI. You can use CLI tools like `dbus-send`.
I pray to God nothing meaningful is actually doing what you are insinuating in any serious environment.
This is a quite frankly ridiculous point. Most of that garb came from the HPC people who built loads of stuff on top of it in the first place. It's absolutely fine for this sort of stuff. It's sending the odd little thing here and there, not on a complex HPC cluster.
As for JSON, are you really that short sighted that it's the only method of encoding something? Is "oh well it doesn't fit the primitive types, so just shove it in a string and add another layer of parsing" acceptable? Hell no.
Structs are a part of C semantic. They are not an ipc format. You can somewhat use them like one if you take a lot of precaution about how they are laid out in memory including padding and packing but it’s very brittle.
Asn.1 is both quite complicated and not very efficient.
They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice?
I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part.
ASN.1 BER/DER is more or less the same thing as CBOR. The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”).
That "convoluted telco/ITU-T style" yields amazingly high quality specifications. I'll take X.68x / X.69x any day over most Internet RFCs (and I've written a number of Internet RFCs). The ITU-T puts a great deal of effort into its specs, or at least the ASN.1 working group did.
ASN.1 is not that complicated. Pity that fools who thought ASN.1 was complicated re-invented the wheel quite poorly (Protocol Buffers I'm looking at you).
For our sins, our industry is doomed to suffer under unbearable weight of endless reinvented wheels. Of course it would have been better to stick with ASN.1. Of course we didn't, because inexperience and hubris. We'll never learn.
It sure seems that way. Sad. It's not just hubris nor inexperience -- it's cognitive load. It's often easier to wing something that later grows a lot than it is to go find a suitable technology that already exists.
One thing I liked about a Vernor Vinge sci-fi novel I read once was the concept of "computer archeologist". Spool the evolution of software forwards a few centuries, and we'll have layers upon layers of software where instead of solving problems with existing tooling, we just plaster on yet another NIH layer. Rinse and repeat, and soon enough we'll need a separate profession of people who are capable of digging down into those old layers and figure out how they work.
> The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”).
I can’t resist pointing that it’s basically a longer way of saying quite complicated and not very efficient.
> I can’t resist pointing that it’s basically a longer way of saying quite complicated and not very efficient.
That's very wrong. ASN.1 is complicated because it's quite complete by comparison to other syntaxes, but it's absolutely not inefficient unless you mean BER/DER/CER, but those are just _some_ of the encoding rules available for use with ASN.1.
To give just one example of "complicated", ASN.1 lets you specify default values for optional members (fields) of SEQUENCEs and SETs (structures), whereas Protocol Buffers and XDR (to give some examples) only let you specify optional fields but not default values.
Another example of "complicated" is that ASN.1 has extensibility rules because the whole "oh TLV encodings are inherently extensible" thing turned out to be a Bad Idea (tm) when people decided that TLV encodings were unnecessarily inefficient (true!) so they designed efficient, non-TLV encodings. Well guess what: Protocol Buffers suffers from extensibility issues that ASN.1 does not, and that is a serious problem.
Basically, with a subset of ASN.1 you can do everything that you can do with MSFT RPC's IDL, with XDR, with Protocol Buffers, etc. But if you stick to a simple subset of ASN.1, or to any of those other IDLs, then you end up having to write _normative_ natural language text (typically English) in specifications to cover all the things not stated in the IDL part of the spec. The problem with that is that it's easy to miss things or get them wrong, or to be ambiguous. ASN.1 in its full flower of "complexity" (all of X.680 plus all of X.681, X.682, and X.683) lets you express much more of your protocols in a _formal_ language.
I maintain an ASN.1 compiler. I've implemented parts of X.681, X.682, and X.683 so that I could have the compiler generate code for the sorts of typed holes you see in PKI -all the extensions, all the SANs, and so on- so that the programmer can do much less of the work of having to invoke a codec for each of those extensions.
A lot of the complexity in ASN.1 is optional, but it's very much worth at least knowing about it. Certainly it's worth not repeating mistakes of the past. Protocol Buffers is infuriating. Not only is PB a TLV encoding (why? probably because "extensibility is easy with TLV!!1!, but that's not quite true), but the IDL requires manual assignment of tag values, which makes uses of the PB IDL very ugly. ASN.1 originally also had the manual assignment of tags problem, but eventually ASN.1 was extended to not require that anymore.
Cavalier attitudes like "ASN.1 is too complicated" lead to bad results.
> That's very wrong. ASN.1 is complicated because it's quite complete by comparison to other syntaxes
So, it's quite complicated. Yes, what I have been saying from the start. If you start the conversation by "you can define a small subset of this terrible piece of technology which is bearable", it's going to be hard convincing people it's a good idea.
> Cavalier attitudes like "ASN.1 is too complicated" lead to bad results.
I merely say quite complicated not too complicated.
Still, ASN.1 is a telco protocol through and through. It shows everywhere: syntax, tooling. Honestly, I don't see any point in using it unless it's required by law or by contract (I had to, I will never again).
> but it's absolutely not inefficient unless you mean BER/DER/CER, but those are just _some_ of the encoding rules available for use with ASN.1.
Sorry, I'm glade to learn you can make ASN.1 efficient if you are a specialist and now what you are doing with the myriad available encodings. It's only inefficient in the way everyone use it.
Subsets of ASN.1 that match the functionality of Protocol Buffers are not "quite complicated" -- they are no more complicated than PB.
> Still, ASN.1 is a telco protocol through and through.
Not really. The ITU-T developed it, so it gets used a lot in telco protocols, but the IETF also makes a lot of use of it. It's just a syntax and set of encoding rules.
And so what if it were "a telco protocol through and through" anyways? Where's the problem?
> It shows everywhere: syntax, tooling.
The syntax is very much a 1980s syntax. It is ugly syntax, and it is hard to write a parser for using LALR(1) because there are cases where the same definition means different things depending on what kinds of things are used in the definition. But this can be fixed by using an alternate syntax, or by not using LALR(1), or by hacking it.
The tooling? There's open source tooling that generates code like any XDR tooling and like PB tooling and like MSFT RPC tooling.
> Sorry, I'm glade to learn you can make ASN.1 efficient if you are a specialist and now what you are doing with the myriad available encodings. It's only inefficient in the way everyone use it.
No, you don't have to be a specialist. The complaint about inefficiency is about the choice of encoding rules made by whatever protocol spec you're targeting. E.g., PKI uses DER, so a TLV encoding, thus it's inefficient. Ditto Kerberos. These choices are hard to change ex-post, so they don't change.
"[T]he way everyone use it" is the way the application protocol specs say you have to. But that's not ASN.1 -- that's the application protocol.
Uh, no, structs, records, whatever you want to call them, are in many, if not most programming languages. "Structs" is not just "C structs" -- it's just shorthand for "structured data types" (same as in C!).
> Asn.1 is both quite complicated and not very efficient.
Every rich encoding system is complicated. As for efficiency, ASN.1 has many encoding rules, some of which are quite bad (BER/DER/CER, which are the first family of ERs, and so many thing ASN.1 == BER/DER/CER, but that's not the case), and some of which are very efficient (PER, OER). Heck, you can use XML and JSON as ASN.1 encoding rules (XER, JER).
> They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice?
Protocol buffers is a tag-length-value (TLV) encoding, same as BER/DER/CER. Having to have a tag and length for every value encoded is very inefficient, both in terms of encoding size as well as in terms of computation.
The better ASN.1 ERs -PER and OER- are much more akin to XDR and Flat buffers than to protobufs.
> I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part.
> Uh, no, structs, records, whatever you want to call them
It's plenty clear from discussion context that OP is talking about C struct but yes, replace C with any languages which suit you. It will still be part of the language semantic and not an IPC specification.
The point is you can't generally use memory layout as an IPC protocol because you generally have no guarantee that it will be the same for all architectures.
If it's IPC, it's the same architecture (mostly; typically there's at most 3 local architectures). The receiver can always make right. If there's hidden remoting going on, the proxies can make things right.
It's "structs" when the sender and receiver are using the same architecture, and if they're using the same int/long/pointer sizes then the only work to do is swabbing and pointer validation / fixups. That's a lot less work than is needed to do just about any encoding like protocol buffers, but it's not far from flat buffers and capnp.
I think it is primarily for compatibility. Practically every language has a readily available library, often part of the standard library, for json. That is less true for something like Msgpack or cbor, and even less so if make a new binary format.
Also, there are benefits to the format being human readable.
As a little bit of perspective, I thought along the same lines when I saw the first web browser. Text was good, but why did they have all these inline images. It was so wasteful of bandwidth.
(also, unix uses files for everything, including /proc talking to drivers!)
> So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that?
Someone took a look at D-Bus and decided "You know what? It just isn't shitty enough. It's 2024 and D-Bus runs halfway decently on the average laptop. Let's fix that."
I wouldn't be surprised if the Scooby Gang pulled the mask off the head dev, revealing ThePrimeagen, in a scheme of long-form comedy that's a cross between one of his "let's write a network protocol" streams and his new enterprise development satire series.
Why don't they just adopt the same fundamental IPC design as Wayland? Swap out the root wl_display singleton object for something dbus appropriate and away you go?
More or less all the display specific stuff is an extension on top of the core design.
That would be going full circle, given that D-Bus was predated by
Bonono on GNOME side, and DCOP on KDE, which was based on X11 Inter-client communication.
D-Bus evolved from merging their needs into a single approach.
Why do people keep replicating this terrible design decision? It puts the lowest-entropy part of the name at the beginning, so that you have to parse (or type) the maximum number of characters before you get a unique specification. Try tab-completing a "flatpak run" command sometime.
Setting the serialization format issue aside, why is the D-Bus broker an issue?
Why was it designed that way in the first place if all we need (supposedly) is a socket and a simple request/response protocol? If it wasn't needed, why does D-Bus have it? And if it is needed, what will replace it when systemd switches to Varlink?
systemd has been using varlink for a while now along side dbus, its not something just now being introduced. They don't have a problem living side-by-side.
Ok? I don't see how that is relavant to the question at hand (to be fair, I did assume the parent meant "will this cause any problems since firefox relies on dbus?").
Anything that add new features (early boot IPC in this case) is going to require "more code, more bloat, more complexity, and more attack surface area" unless you rip other features out at the same time.
That looks interesting, I especially appreciate the intentional design decision for documentation and schemas. While it didn't work quite well with dbus Ime it is still a good idea to make APIs more accessible.
There is something impressive about Lennart’s work, in that treads a thin line between “completely triggering” to some people and “genuinely quite a good idea” to others.
Lots of discussion here has been around the choice of JSON as a format, with some valid points being brought up about specific limitations and possible footguns.
But these miss the point: this massively moves the needle with regard to lowering the barrier to entry when writing services. I’m confident I can write a simple Varlink service in about 20 lines of Python, with no dependencies after reading the linked docs. You cannot in good faith say the same thing about dbus.
That’s nuts. Yeah - if my service is for some reason sending/receiving high-precision 128 bit numbers or huge binary files then it’s not perfect.
But nothing really does that. It’s not worth making the entire stack more complex and less performant for a possible use case that doesn’t really fit.
He's living churn. It's just constant change often for gains that don't matter to a lot of people and losees that do. Linux is radically different because of him. People were deeply invested in how things were. Think about how hard it is to learn a laguage as an adult compared to when you're a kid. Even when some of his stuff is amazing, it's still going to cause stress for a lot of people due to its radical nature.
But what is the excuse to start from scratch? Rather than a compat layer that convinces the world that raw your interfaces, libraries and clients are so much nicer to work with that the other thing should be demoted to compat layer eventually?
Red Hat FOSS system software is imposing foss-ified client work to the general public. The cloud industry benefited greatly from their work, the other users are involved in a multi-decade flamewar.
Someone explain to me please, how type-safe and JSON can even exist in the same sentence? JSON has so many edge cases for handling numbers, floats, bools etc. Why would you not use gRPC or even something like CapnProto if you want to maintain that type-safety-ness?
I think that there are problems with both Varlink and D-Bus. JSON is one of them but not only one. One of the problems with JSON is the data types it uses. There is no support for character codes other than Unicode (I think that "Unicode string types considered harmful"; they are not needed for handling text (even Unicode text)), and binary data must be encoded as hex or base64, which just makes it inefficient. There is other problems too, including problems with implementations (including JavaScript, which has a big integer type now but it is not the same type as numbers in JSON). I also think there are problems with the way that the message buses are working. D-Bus has some problems and Varlink has some other problems. And, it isn't true that "the whole world speaks JSON".
Yeah, using serialization format that doesn't even have obvious way of representing every possible Linux filename (nothing mandates filenames to be valid UTF-8, only thing they can't contain is NUL and '/') seems bad fit for low-level system services.
I can't wait to have apps and services crash and burn left and right due to accidentally (or maliciously) created non-UTF8 filenames in world readable directories.
> Yeah, using serialization format that doesn't even have obvious way of representing every possible Linux filename
That is hardly the only problem with such a serialization format, although it is one of them.
> I can't wait to have apps and services crash and burn left and right due to accidentally (or maliciously) created non-UTF8 filenames in world readable directories.
It is not only accidentally or maliciously. Sometimes it may also be done deliberately because it is a character set other than Unicode, without the intention to be malicious.
(A user might also deliberately want to display file names using a different character set, perhaps in order to ensure that all characters can be displayed unambiguously. Of course, this is not necessarily what everyone intends, but some people will.)
Looks like the transformation of RHEL into a Windows clone is nearly complete. I'm all for it, as I expect a modern fork without systemd will coalesce for those of us that prefer Linux over Windows.
This is JSON-RPC basically. Microkernel architecture personified where everything is a service. Unix was all about files. The emerging next standard is probably an OS built entirely of services.
Next time someone asks you why computers are getting slower despite hardware being faster, everyone who thinks this is a good idea, please remember to blame yourselves.
All the the hate for json here, but at the bottom there is a bridge screenshot. Varlink is designed such that it can trivially be piped across different systems and be maximally interoperable due simple to parse and encode protocol. We might end up with some IPC interface that a lot me tooling can understand. Eg imagine that instead of some daemons supporting sighup to reload, they could also confirm that they reloaded, provided Prometheus style metrics etc
I just happened to read Lennart Pottering's original post on sd-dbus yesterday actually. It gives a good introduction to D-Bus for those not up to date.
Christ almighty, I have seldom seen a technical proposal as bad as this one. I want off Mr. Poettering's wild ride. No 64 bit integers? No file descriptor transport? No message sequence numbers? Just cut it out already. Dbus doesn't do something? Extend dbus. Don't make some other random and worse things.
How have we survived without such a feature up to now?
It is a common occurrence that whatever Lennart comes up with fulfills several, erm, novel requirements while not fulfilling some old ones that aren't even mentioned. The parallels to binary logging are there. "Fancy" features that no one asked for - check. Simplicity and thoughtful use of existing functionality (the "full exploitation" principle of the Unix design philosophy) - nope.
Hold on, (even though kicking JSON while it's down is funny) how do you make sure that the reply you've got is to the call you issued with this thing? Surely not pinky promise?
Is modularity an unalloyed good? Modularity comes with tradeoffs that mean you end up with, well, a bunch of discrete modules rather than something that works cohesively. That's why systemd was adopted pretty much everywhere, ignoring the arbitrary modular boundaries results in a more useful tool.
Some analogs: modern filesystems like ZFS and BTRFS that combine volume management with the filesystem, every service that's consciously chosen to deploy a monolith instead of microservices, and so on, every deployment that chooses to statically link, etc.
So does this mean I can shove SQLite databases into the file structure, abstract away an entire SQLite engine into a single header that you are forced to use, and declare that IPC? And you know, since we _are_ already hiding databases within the file structure, why not use it to also unify configuration and installation files too? What could go wrong?
Yes, you can certainly send a whole SQLite database using any binary communication protocol as a way to exchange data today. You can even compress it before if you feel like it.
It will not make for a good ipc format because, well, it’s not designed to be one but it would most definitely work.
There is this one somewhat widely deployed niche system that uses sqlite databases as RPC serialization format. The original idea behind that was limited virtual address space of the CE-based embedded devices, but the design got stuck.
So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that?
Apparently, we don't have enough wasted cycles in modern software, just add a bunch more in a fundamental IPC infrastructure.
The marshalling cost for JSON is negligible. Yes, it might be a bit slower than GVariant for example, but only by some fractional linear factor. And on small messages (which D-Bus currently always is, due to message size constraints enforced by broker) the difference is impossible to measure. To a point it really doesn't matter, in particular as JSON parsers have been ridiculously well optimized in this world.
What does matter though are roundtrips. In Varlink there are much fewer required for typical ops than there are in D-Bus. That's because D-Bus implies a broker (which doubles the number of roundtrips), but also because D-Bus forces you into a model of sending smaller "summary" messages when enumerating plus querying "details" for each listed objects, because it enforces transfer rate limits on everything (if you hit them, you are kicked off the bus), which means you have to refrain from streaming too large data.
Or in other words: marshalling is quite an irrelevant minor detail when it comes to performance, you must look at roundtrips instead and the context switches it effects, instead.
Using JSON for this has two major benefits: the whole world speaks JSON, and modern programming languages typically pretty natively. And it's directly readable in tools such as strace. With a simple "strace" I can now reasonably trace my programs, which a binary serialization will never allow you. And if you tell me that that doesn't matter, then you apparently live in an entirely different world than I do, because in mine debuggability does matter. A lot. Probably more than most other things.
Lennart
The retrospective on the Xi editor project has a few notes on issues with JSON (at least in the context of Rust and Swift).
This is from someone who initially seemed to have a very similar perspective to you.
See the JSON section of the post: https://raphlinus.github.io/xi/2020/06/27/xi-retrospective.h...
To save others the click: Their issues were simply that Swift has no fast JSON impl, and in Rust, when using serde (most popular library handling JSON marshalling), it leads to binaries getting a bunch bigger. That's it. So yeah, same perspective -- unless either of the above matter in your case (in 90%+ of cases they don't), JSON is just fine from a perf perspective.
Serde is a rather chunky dependency, it's not just a matter of binaries getting bigger, but also compile times being dramatically slower.
IMO CBOR would be a better choice, you aren't limited to IEEE 754 floats for your numeric types. Yeah, some (de/en)coders can handle integer types, but many won't, it's strictly out of spec. I don't think building something as fundamental to an OS as relying on out-of-spec behavior is a great idea. It will result in confusion and many wasted hours sooner or later.
> CBOR would be a better choice, you aren't limited to IEEE 754 floats for your numeric types.
The other side of this coin, of course, is that now you have to support those other numeric types :) My usual languages of choice somehow don't support "negative integers in the range -2^64..-1 inclusive".
I mean, you don't have to support those? You still would need something on the other end to produce that type of datatype, which can be documented that it will never happen: you're making an interface anyways. The problem is if you literally don't have the option to represent common datatypes it will be a problem, not a hypothetical one just because the encoding layer can support it. Those are different problems.
And JSON, technically, allows use of unlimited-precision fractions, but also allows implementations to set arbitrary limits (it actually does, you're not required to parse JSON numbers as doubles). So the situation is not really different from CBOR, isn't it? Just™ make both sides to agree to stick to some common subset (e.g. integers-in-int64_t-range-only for some fields) and you're done; no need to support double-precision floating point numbers.
Huh, I went and referenced the ECMA JSON spec and you're right that it treats numbers only as sequences of digits which would make these effectively the same problem
The marshalling cost might be negligible for come use cases, but the bandwidth usage definitely is not. I think the best interface description protocol is one where the serialization format is unspecified. Instead, the protocol describes how to specify the structure, exchange sequences, and pre/post-conditions. A separate document describes how to implement that specification with a certain over the wire format. That way the JSON folks can use JSON when they want (unless they are using large longs), and other folks can use what they want (I like CBOR).
I read the slides, and I found it refreshing that you said at the end: don't create per-language bindings for the libraries shipped with systemd, but simply use a JSON parser for your language. That underlined that you've specified a simple protocol.
Also, there have clearly also been several attempts over the years to make a faster D-Bus implementation (kdbus, BUS1), which were never accepted into the kernel. It makes a lot of sense to instead design a simpler protocol.
There is clearly also a cautionary take about how microbenchmarks (here, for serialisation) can mask systemic flaws (lots of context switches with D-Bus, especially once polkit had to be involved).
The danger I see is that JSON has lots of edge behavior around deserialization, and some languages will deserialize a 100 digit number differently. If the main benefit is removing the broker and the need for rate limiting - it could have been accomplished without using JSON.
You are writing this as if JSON was a newly invented thing, and not a language that has become the lingua franca of the Internet when it comes to encoding structured data. Well understood, and universally handled, since 1997.
A 100 digit number cannot be encoded losslessly in D-Bus btw, nor in the far majority of IPC marshallings on this word.
Having done systems-level OS development since 25y or so I never felt the burning urge to send a 100 digit number over local IPC.
Not that 100 digit numbers aren't useful, even in IPC, but typically, that's a cryptography thing, and they generally use their own serializations anyway.
You are writing this as if security was a newly invented thing. Having done systems level security development for 12 years, anything that can be produced maliciously will be. By using JSON, you've invented a new vulnerability class for malicious deserialization attacks.
Actually, not new. Earliest CVE I found was from 2017, which feels a decade later than it should be. I guess no one thought of pushing JSON over trusted interfaces, and probably for good reason.
> A 100 digit number cannot be encoded losslessly in D-Bus btw
I think the concern is that large numbers can in fact be encoded in JSON, but there is no guarantee that they will be decoded correctly by a receiver as the format is underspecified. So you have to cater for the ill defined common denominator.
You should probably encode large numbers as strings.
The format is properly specified; its mapping onto actual hardware people use is not.
Honestly, the only thing that surprises me is you're being pedantic, and encoding int64s as strings.
I know you know JSON is nominally only 53-bit safe, because JS numbers are doubles. But in practice I'd wager most JSON libraries can handle 64-bit integers.
What if varlink supported both JSON and a subset of cbor with the "cbor data follows" tag at the beginning (so the server can determine if it is json or cbor based on the beginning of the message)?
It would add a little complexity to the server, but then clients can choose if they want to use a human readable format that has more available libraries or a binary format.
As for strace, tooling could probably be added to automatically decode cbor to json, either as part of strace, or in a wrapper.
There could also be a varlink proxy (similar to varlink bridge) that could log or otherwise capture requests in a human readable format.
But why though? Is this really a performance critical bus?
Yes. I run shared desktop login server clusters for students, DBUS is a bottle neck.
>With a simple "strace" I can now reasonably trace my programs, which a binary serialization will never allow you
Doesn't systemd use binary logging?
Not really. We use two text based formats for logging: BSD syslog, and systemd's structured logging (which is basically an env block, i.e. a key-value set, with some tweaks). Programs generate text logs, journald reads text logs hence. Programs that read from the journal get the text-based key/value stuff back, usually.
(Yes, we then store the structure log data on disk in a binary format. Lookup indexes are just nasty in text based formats).
Hence, not sure what the Journal has to do with Varlink, but any IPC that the journal does is text-based, and very nicely strace'able in fact, I do that all the time.
[Maybe, when trying to be a smartass, try to be "smart", and not just an "ass"?]
Sure the interface with the log might be text based, but my understanding is that the at rest format is binary and you need specialized tools to read it, standard unix grep is not going to cut it.
Although I use strace all the time, I hardly ever look at the payload of read and write calls, although I could see why it would be useful. But given a binary protocol it wouldn't be terribly hard to build a tool that parses the output of strace.
> [Maybe, when trying to be a smartass, try to be "smart", and not just an "ass"?]
thanks for the kind words and elevating the tone of the discussion.
> ...it wouldn't be terribly hard to build a tool that parses the output of strace.
Nah, it'd be a rage-inducing nightmare.
> With a simple "strace" I can now reasonably trace my programs, which a binary serialization will never allow you.
Doesn't strace already come with desetializers for many common data structures?
is this true of future desktop uses cases where every basic function will cause a torrent of traffic on that? or you're talking from a server start/stopping services only point of view?
i honestly don't really get the angle of debugging via strace - i'd much rather prefer something more wireshark-like, where I can see all messages processes are sending to each other, since that would make it easier to decipher cases where sending a message to a service causes it to send other messages to its backends
“ The marshalling cost for JSON is negligible”
I’ve worked with profiling code where the marshaling cost for JSON was the biggest cost. Namely it involved a heap allocation and copying a ton more data than was actually needed, and I ended up fixing it by turning the JSON into a static string and dropping the values in manually.
The systemd maintainers have probably done their due diligence and concluded that it isn’t an issue for their forseeable use cases, but it does lock everything in to doing string processing when interfacing with systemd, which is probably unnecessary. And you can’t trivially swap systemd out for something else.
systemd is so pervasive that it would be fine to add a binary-format-to-JSON translation ability into strace. That shifts the cost of debugging to the debug tools, rather than slowing down production code.
Doing any string processing tends to require a lot of branching, and branch mispredictions are most likely to slow down code. It also turns every 1-cyle load/store instruction into N-cycles.
String processing in C, which is what systemd and a lot of system tools are written, is pretty abysmal.
systemd is also non-optional, so if it turns out that it’s causing cache thrashing by dint of something generating a lot of small events, it’s not something you can do something about without digging into the details of your lowlevel system software or getting rid of systemd.
And it’s potentially just that much more waste on old or low-power hardware. Sure, it’s probably “negligible”, but the effort required to do anything more efficient is probably trivial compared to the aggregate cost.
And yeah, it may be better than D-Bus, but “it’s not as bad as the thing that it replaced” is pretty much the bare minimum expectation for such a change. I mean, if you’re swapping out things for something that’s even worse, what are you even doing?
I see there’s a TCP sidechannel, but why increase the complexity of the overall system by having two different channels when you could use one?
Dunno. This isn’t really an area that I work in, so I can’t say for sure it was the wrong decision, but the arguments I hear being made for it don’t seem great. For something fundamental like systemd, I’d expect it to use a serialization format that prioritizes being efficient and strongly-typed with minimal dependencies, rather than interoperability within the application layer with weakly-typed interpreted languages. This feels like a case of people choosing something they’re more personally familiar with than what’s actually optimal (and again, the reason I’d consider being optimal in this case being worth it is because this is a mandatory part of so many devices).
EDIT: Also, the reason that binary serialization is more efficient is because it’s simpler - for machines. JSON looks simpler to humans, but it’s actually a lot more complex under the hood, and for something fundamental having something simple tends to be better. Just because there’s an RFC out there that answers every question you could possibly have about JSON still doesn’t mean it’s as good as something for which the spec is much, much smaller.
JSON’s deceptive simplicity also results in people trying to handroll their own parsing or serialization, which then breaks in edge cases or doesn’t quite 100% follow the spec.
And Just because you’re using JSON doesn’t force C/++ developers to validate it, someone can still use an atoi() on an incoming string because “we only need one thing and it avoids pulling in an extra dependency for a proper json parser”, then breaks when a subsequent version of systemd changes the message. Etc. If the goal is to avoid memory safety issues in C/++, using more strings is not the answer.
But .. how can anyone use strace? That's not JSON. And serialization is cheap!
Do you know what strace is?
It's a command that prints the system calls (done via library). So you see write(55, "[1,2,3,4]\n")
Let me try again. You do see
and not which would obviously be much better!It can be parsed by any modern language, and deserialization is cheap.
Can we please rewrite strace with this in mind? Preferably in Rust.
How is the 2nd much better? Considering that a trace contains more than 1 single line?
Anyway they'd probably accept a patch for json output format. I don't think it's so difficult to do.
You can ask them in advance if they'd be willing to accept it, before starting to write it.
[dead]
Unfortunately I concur.
There are a lot of problems with D-Bus, but that it wasn't using JSON wasn't one of them.
The tooling for D-Bus is terrible. It has composite types but they are very primitive, e.g. you can't name fields in a structure without extensions (there's like three different extensions for this.) A lot of the code generators can't actually handle real-world schemas.
Now I understand that D-Bus itself is also a problem (as in, the protocol design beyond serialization) but varlink looks like a very solid step backwards. In any single individual use case JSON serialization is unlikely to be a huge bottleneck, but in practice if this will be used for all kinds of things it will add up.
I really wish the Linux desktop would coalesce around something like Cap'n'proto for serialization.
P.S.: It'd probably be possible to shoehorn in multiple formats but I think that's a mistake too. Something like capnp will have a strong impedance mismatch with JSON; mixing formats with different degrees of self-describability is unwise. But then formats like MsgPack and BSON which are direct JSON analogues miss out on the best benefits of using binary serialization formats, which makes them almost the worst of both worlds, and you'd probably be better off with using a highly-optimized JSON library like simdjson.
CBOR is the obvious choice here.
I've suggested it before and the systemd folks didn't seem completely opposed to it. Also because cbor parser is already in the dependencies due to FIDO2 disk unlocking
I genuinely dislike CBOR. Formats which require me to calculate the length of the message before sending it lead to bad library design which often leads to easily compromised code. Add in an "indefinite length" option and you've got potentially unbounded client memory usage to watch out for. As if that wasn't enough you get extensible tags so the meaning of any message is entirely dependent on the context it was sent in.
It gives you a lot of decent protocol wire design and then flatly ignores everything we've learned about these types of designs in the last 3 decades. Be on the lookout for client libraries to slowly add in all of these checks as the vulnerabilities are discovered in them.
JSON's everything is "indefinite length". Also its implementations are wildly inconsistent [1] (is anyone going to read the code to figure out which is the blessed systemd parser?). Also it doesn't have integers. A lot of things with JSON will deteriorate into stringifying anyway, for example dates.
[1]: https://seriot.ch/projects/parsing_json.html
CBOR does have indefinite length arrays, maps, byte strings and text strings : https://www.rfc-editor.org/rfc/rfc8949.html#name-indefinite-...
I'm a fan of DAG-CBOR, which is a deterministic subset of CBOR with most of the weird bits excluded (like indefinite lengths, multiple floating point formats, etc.), but it can still represent any sensible JSON object.
FWIW, ASN.1 BER supports an indefinite length encoding (though DER does not; ASN.1 JSON and ASN.1 XML do as well)
ASN suffers from some of the other problems regardless of the encoding.
I was thinking about this more and I think FTP actually had a lot of the right ideas here. You want two channels. One for fast interactive command messaging and a second one coordinate especially just for bulk transfers arranged over the command channel. The design flaw in FTP of putting these on two separate ports put it immediately in conflict with firewall best practices so I think it went mostly unnoticed that it fundamentally is a good arrangement.
What you want is one channel, sequenced packets, with a defined maximum message length that is negotiated at startup and never changes during the life of the channel. This should probably never be more than 65k. There should be a known length packet type, and an unknown length packet type, with any attempt to send more than the negotiated maximum triggering an error and disconnect.
If you do need to send more than the negotiated amount you should open a new connection, in a bulk transfer mode, that after the initial handshake, has no protocol and is merely an associated stream of bytes that are wholly uninterpreted by the middleware layer other than to help you associate it with the sequenced packet that requested its initiation.
You'd actually be able to use TCP (with DSCP even), mostly avoid head of line blocking and multiplexer latencies, and have a reasonable security guarantee in the sequenced packet mode, and never have a protocol which pretends that 4GB strings in the middle of a packet are necessary or even a good idea to "support."
The downfall of this is that it would be much harder to implement it on a serverless architecture and would be nearly as complicated as a protocol as WebSocket ends up being. It might be worth playing with as a concept anyways.
You would need a pretty intense QoS system built for this if you are using something like TCP for this using two different connections. Underneath it would still use IPv4/6 packets and you only have so much bandwidth. The bulk transfer channel might stall the transfer of control packets temporarily.
You would need to tightly control and multiplex the messages yourself, needing to do this in one connection or using something like TCP priority flag.
Personally I just think that TCP is a shitty protocol for building applications. In almost every use case either UDP or SCTP are a better choice. With raw UDP datagrams you aren’t guaranteed delivery but it’s great for the kind of telemetry where the last message in is what matters. You can also build quite flexible stuff on top of it.
SCTP gives you in sequence reliable datagrams and congestion control. This means you don’t have to devise message length communication into your application layer protocol and can rely on your transport. It also has multiplexing built right in. If it had a built in checksum it would truly be ideal. Not sure if something like secure communication really belongs at this level but if it had that I double we would ever use anything else. From SCTP’s Wikipedia page:
> SCTP applications submit data for transmission in messages (groups of bytes) to the SCTP transport layer. SCTP places messages and control information into separate chunks (data chunks and control chunks), each identified by a chunk header. The protocol can fragment a message into multiple data chunks, but each data chunk contains data from only one user message. SCTP bundles the chunks into SCTP packets. The SCTP packet, which is submitted to the Internet Protocol, consists of a packet header, SCTP control chunks (when necessary), followed by SCTP data chunks (when available).
I think the protocol on which PDUs are exchanged on is a bit orthogonal to the PDU protocol itself here, but the split-mode networking protocols (FTP, IRC DCC, mosh, SIP, etc) always do end up getting less used because establishing two different connections is a bigger burden and there really needs to be a great advantage to doing so, and often require other considerations (better connection tracking, TURN/ICE servers, etc).
For a localhost/UNIX domain protocol, it might work since most of these considerations are significantly reduced.
As to DSCP, it might be useful across controlled networks but in general, in my experience it's not really ever honored broadly.
Back on the original topic, when I was building this for my Linux distribution I ended up just using basic JSON with TLS (for client certificate authentication) though since having authentication for remote management was a goal, and once I was already having to perform a TLS negotiation then the PDU consideration for performance wasn't something to really spend too much time on.
One channel, multiplexing, sequenced packets. It looks like you are describing SSH.
"lead to bad library design"
how?
More complex buffer management in non-garbage collected languages.
man 3 cmsg
For a glimpse into the darker corners of hell. At least you get to grip it there. In GCed languages you have to hope the library designer gave you reasonable controls for timeouts, aborting large data transfers, and releasing buffers efficiently in a bursty network environment.
See all the controls any HTTP server gives you for this. "Maximum header size", "Maximum header(s) size", "Maximum Body Size", "Request Header Timeout", "Total Request Timeout."
None of those were added by default or by tasteful library authors. They were all added to answer to specific emergent security vulnerabilities that were discovered and then were demanded by the users.
That'd be "I can't believe it's not msgpack with extra design flaws and the name changed to my own name" CBOR?
What is the data rate (messages, bytes) that you expect for this "D-Bus replacement protocol"?
Which fraction of CPU will JSON ser/des will take to justify using CBOR?
I mean conversely every second week on HN someone complains about "inefficient modern development practices" and this is the sort of minor thing which would be trotted out as an example.
So I'd argue it's not an unreasonable question (although I lean closer in: better types then JSON is the problem, since a decent text serialization format is always going to be needed for debugging and development).
This sounds like it wants to be a fundamental IPC facility in Linux, which seems like a major thing.
Any binary format can be dumped to a text when viewed for debugging. That’s really a nonissue. Have the tools dump to JSON if you like.
It's not the data format that is important, but the tooling, code generation and binding quality.
Data format is absolutely important for operating system IPC. JSON is raw text. Messages occupy more RAM than a binary format, which increases your working set, and it requires at least some nontrivial parsing. This ultimately limits the number of messages you can send per second.
It's not that simple:
is 13 bytes, the equivalent binary storage is 3*sizeof(double) (24 bytes) + whatever overhead exists for representing a dynamic array in your languagePersonal opinion: don't put floating point numbers into an API. If you really want real numbers passing them as decimal string is actually sensible. This can be done without using JSON, of course. Can also do fixed precision as well, which will store reasonably well into a VLQ.
OTOH this is pretty cherry picked. It is questionable if you really need floating point numbers if your numbers are all this small and low precision, but in actuality unless this comprises your entire message, you'll probably still lose to JSON overhead eventually anyways, whereas non-self-describing serialization like capnp has effectively no overhead by default. The advantage of encoding as doubles is that it is predictable and won't explode into taking massively more, its always the same size per number. If you want it to be smaller for some reason, compression is always an option, though I suspect for IPC it's the wrong thing to be concerned about.
What's the chance the data you want to transmit is exactly 1.2 rather than 1.205338843375?
If you expect to routinely send small numbers your serialization protocol can use floats or some small fix point format.
Variable length binary number encodings are also an option.
The dynamic array overhead in a language is immaterial when discussion serialization formats.
Or an ASN.1 DER encoded sequence -- ASN.1 is more common than CBOR (being in a lot of PKI stuff) and there are already generators for DER (or JSON or XML...) for a given schema.
Which makes me wonder how performance sensitive is the task D-Bus handles?
Most task done over it seems to either fall into the "from time to time but not really performance sensitive at all" category or the "a mass of system events which can become huge" category.
> miss out on the best benefits the best benefits of using binary serialization format
I disagree (wrt. MsgPack), the biggest benefit is of binary formats is that you can ship binary blobs without any encoding (mainly space) overhead while for JSON this can easily be ~33%+ space overhead. And even for just e.g. numbers the space overhead can be bad. I'm not sure what you see as the biggest benefit but not having a minimalist structure isn't a property specific to binary formats (through more common there, and MsgPack doesn't require field names to be included). And bit fiddling micro space optimizations aren't that grate/as much a benefit as drawback. And as a system bus handles (emits) potentially a _ton_ of events saving 33% can matter I think.
A lot, D-Bus in a way has the same role as XPC, COM and Binder on other platforms, so many desktop applications use D-Bus for their plugins, now imagine the traffic of something like Thunar doing D-Bus calls for generating image previews for the new vacations album someone just created.
The obvious choice is ASN1 /s
I've done similar things in the past (json over udp-to-localhost, with a schema description used to generate the code for both parsing and generating on each end). It's a totally reasonable point in the design space, and I literally never saw a reason to revisit the decision to use json for that system.
You'd do that because everything under the sun knows how to generate and parse json. Performance looks like a total non-issue for this use case. The messages are tiny, local and infrequent. You wouldn't want to do this on some kind of a high performance application on the data plane, but on the control plane it's totally fine.
Even if you expect that all production use cases will use some kind of higher level library that does schema validation, it can still be quite nice during debugging to a) be able to inspect the messages on the wire when debugging, b) to be able to hand-write and inject messages.
> Performance looks like a total non-issue for this use case.
The presentation brings up performance as one of the problems with the D-Bus though, so it seems like it is an issue.
Fair. What I meant is that the performance delta between JSON and any other interchange data format should be a non-issue in this context. I would be quite surprised to hear that any performance problems systemd has with D-Bus was with serialization/de-serialization.
> performance delta between JSON and any other interchange data format should be a non-issue
it's hard to say, for most D-Bus taks perf. is an absolute non issues, but still you have bus-1 and all kind of tries to make D-Bus perform faster, reason is there are edge cases where it matters
and even if the marshaling speed might not matter the size can, like if something goes really wrong and tens of thousand of system events get spammed the difference between the memory usage of in-flight messages of a compact format vs. json can make the difference between memory pressure effectively killing your server and you recovering reliable
though what is an open question is how relevant that is in pracive
and that is quite hard to say without inside information for companies running systems where such a thing could happen on a scale where such a thing does happen (e.g. google)
Yeah... it's easy to say JSON won't be an issue. It's difficult to switch from JSON when it turns out you were wrong.
https://raphlinus.github.io/xi/2020/06/27/xi-retrospective.h...
Given how central this is going to be why would you skip the tiny bit of extra work needed to use a binary format?
The performance problem with D-Bus is the increased number of context switches over Varlink, not serialization.
The problem isn't JSON per se. Parsing JSON is fast. The problem is the simplistic interaction and data model that's just going to move complexity from universal IPC layers to ad hoc application logic that every person has to learn individually. Terrible.
Lennart pointed out the fact you can see readable messages via strace to be a benefit of json. If their average message size is small and they don't expect to ever need high volume or large messages, then it's not really likely to be a problem in practice. He also pointed out that they waste far more cycles today on context switches (he suggested something on the order of 6 per IPC message).
Even if they eventually come to the conclusion they need something more performant in the future, it's probably still a net win to make the switch today. It's also possible it turns out fine for their use case.
I personally would think they would want to benefit from an IPC system that includes provisions for sending file descriptors.
> Lennart pointed out the fact you can see readable messages via strace to be a benefit of json.
from the guy who brought you binary logfiles!
One can view the binary log files using journalctl.
Per https://systemd.io/JOURNAL_FILE_FORMAT/, the benefits of the binary format are:
The systemd journal stores log data in a binary format with several features:
Fully indexed by all fields
Can store binary data, up to 2^64-1 in size
Seekable
Primarily append-based, hence robust to corruption
Support for in-line compression
Support for in-line Forward Secure Sealing
As a user system-administrator, I see the cryptographic checksum as a benefit of being able to show tampering evidence of on-system log files.
> Primarily append-based, hence robust to corruption
It's so robust, it doesn't even let you modify the journal if you want to (e.g. https://github.com/systemd/systemd/issues/20673).
> Support for in-line compression
Mind that journald only supports compressing single lines, but not the whole journal (https://github.com/systemd/systemd/issues/31358), which is pretty limiting.
Modifying the existing journal really sounds like the wrong solution. Just "journalctl --rotate" the file and throw out the one with accidental PII. Journal files are not great for long-term storage or search. You can export the old file and filter out manually if you really want to preserve that one https://www.freedesktop.org/wiki/Software/systemd/export/
In what situations is it a harder problem than this?
In sicily we'd call this "vuliri a vutti china e a mugghieri 'mbriaca".
It's a tradeoff, if you do full compression clearly it won't be fast.
You're free to compress it again before archiving it.
> Primarily append-based > Seekable
Text logs are append-based and seekable as well.
> Support for in-line Forward Secure Sealing
I once typed an SSH password in the username field, and the only way to erase that was to erase all the logs. So this has some significant downsides.
Also, I am tired of waiting minutes for a journalctl query to load.
> I once typed an SSH password in the username field, and the only way to erase that was to erase all the logs. So this has some significant downsides.
I hope this was a personal system. Changing logs in this manner would have almost certainly led to your dismissal anywhere I ever worked. This anecdote just re-enforces the need for Forward Secure Sealing.
No, I just left it there.
Can't win huh
Sigh.
Varlink is designed for IPC, so the expected data rate is maybe hundreds of messages per second in the worst case.
JSON can be parsed at 3 Gigabytes/second [0]. Even unoptimized stdlib parsers in scripting languages get 50 MB/sec.
That's more than enough, standard library support is much more important for a project like this.
And if there is ever a reason to send tar archive or video data, there is always "upgrade" option to switch to raw TCP sockets, with no JSON overhead at all.
[0] https://news.ycombinator.com/item?id=19214387
[1] https://github.com/TeskaLabs/cysimdjson
It doesn't matter what the rate is.
Of all the things they could do, they decided to choose one of the worst possible formats for IPC data[1], maybe to pander to the web crowd.
[1] Maybe XML would be a close competitor.
It's the lingua-franca serialization format that has bindings to every programming language already, doesn't require agreeing on the payload layout beforehand, is beyond overkill for the task being asked of it, and is human debuggable over the wire.
I'm begging you to read the format it's replacing. https://dbus.freedesktop.org/doc/dbus-specification.html Ctrl+f Type System. JSON is beautiful, elegant, and minimal by comparison. yyyyuua(yv) It's like the printf message protocol but worse.
> lingua-franca serialization format
it isn't, fundamentally can't be as it has no support for binary blobs
and base64 encoded strings are a pretty terrible solution for this
and the space overhead of many small number can also be pretty bad
and while a lot of IPC use cases are so little performance sensitive that JSON is more then good enough and large blobs are normally not send over messages (instead you e.g. send a fd) there are still some use-cases where you use IPC to listen to system events and you have a situation where a pretty insane amount of them is emitted, in which case JSON might come back and bite you (less due to marshaling performance and more due to the space overhead which depending on the kind of events can easily be 33%+).
But what I do not know is how relevant such edge cases are.
Probably the biggest issue might be IPC system mainly used on 64bit systems not being able to properly represent all 64 bit integers .... but probably that is fine too.
The protocol supports “upgrading” requests. If your service relies on sending large binary blobs (over this? Why?), then it doesn’t have to be done with JSON.
For example, the metadata of the blob could be returned via JSON, then the request is “upgraded” to a pure binary pipe and the results read as-is.
not one large blob, many small events
binary blobs is just the biggest example and was only mentioned in relation to the "lingua franca" argument, many other common things are also larger in JSON. Only if you have many larger not escaped utf-8 strings does this overheads amortize. E.g. uuids are something not uncommonly send around and it's 17 bytes in msgpack as a bin value and 38 bytes in json (not inlcuding `:` and `,` ). That 211% the storage cost. Multiply it with something going on and producing endless amounts of events (e.g. some unmount/mount loop) and that difference can matter.
Through yes for most use cases this will never matter.
I get your point, but you have to understand that for every second you’ve spent writing that comment, globally hundreds of millions of HTTP responses have been processed that contain UUIDs of some kind.
Yes, there’s a more optimal format than hex-encoding UUID values. However it simply does not matter for any use case this targets.
16 bytes vs 38 bytes is completely meaningless in the context of a local process sending a request to a local daemon. It’s meaningless when making a HTTP request as well, unfortunately.
I’d have loved Arrow to be the format chosen, but that’s not lowering the barrier to entry much.
it really isn't irrelevant in edge cases
sure most people don't write code for such edge cases, like ever
but e.g. systemd does sometimes because they reliable appear when running things at e.g. googles scale
> JSON is beautiful, elegant, and minimal by comparison.
True, but binary formats like CBOR or MessagePack are also that and efficient and have support for binary strings. This JSON is human readable thing is the nerverending argument of high level programmers. Anyone can read hex given enough time.
JSON isn't simple. It's simplistic. It's too simple. By refusing to acknowledge the complexity of the real world at the protocol layer, varlink shifts the burden of dealing with the complexity from one transport layer to N ad hoc application layers, making us all worse off.
Do you mean https://varlink.org/Interface-Definition? JSON is just the marshaling format. All the language bindings don't need a fancy parser outside of json.loads. I wish the interface definitions themselves had been specified in JSON as well.
Peep the Python interface parser: https://github.com/varlink/python/blob/master/varlink/scanne.... This is the quality you can expect for an ad-hoc format, which is to say, pretty bad. Making the hardest part farming out to a parsing library you already have and is a built-in in your language is a no-brainer choice.
Being simplistic is a leitmotiv of the project, at least someone in the post-GNOME cabal stopped reinventing the wheel.
Hundreds of messages per second sounds like the performance I would expect from a 1980s computer. Why so incredibly slow?
Read the comment you’re replying to.
In it OP says it’s not slow, and the “hundreds of messages per second” you are referring to is not regarding the performance but the expected rate.
It’s possible for something to be fast, but only happen infrequently.
It just seems strange that the message rate is assumed to be in some very specific range where the performance of D-Bus is a problem (mentioned in the slides) yet the performance overhead of JSON is not a problem.
There was a Varlink talk[0] a few days ago at All Systems Go, in that talk, Lennart mentioned that JSON is unfortunate (primarily due to no 64 bit ints) but it has the surprising benefit of being able to understand the bus messages when using `strace` for debugging.
[0]: https://www.youtube.com/watch?v=emaVH2oWs2Y&list=PLWYdJViL9E...
With this logic (oft repeated) we should be sending TCP as JSON. { sourcePort: 443, destinationPort: 12345,...}
Debugability is important but the answer is to build debugging tools, not to lobotomise the protocol for the vanishingly tiny fraction of packets that are ultimately subject to debugging.
I love your TCP over JSON idea. Maybe we can implement layer 2 that way as well, with CSMA/CD: {startTime:2024-09-29T13:56:21.06551, duration:00:00:00.0024, packet: …}
You forgot quotes. JSON has no date or time types :)
Oh my God. I'm genuinely struggling to avoid using an endless stream of profanity here. If our problem is that our observability tools suck, the solution is to improve these tools, not mutilate our IPC mechanism to accommodate the limitations of these tools.
Christ almighty, this is a terrible proposal.
The observability tools in question (strace) follow the UNIX tradition of displaying and processing data as text by default. I don’t think that means that they suck.
I’ll go even stronger than that: IPC and RPC should prefer plaintext-representable forms by default and only abandon them for binary once the real world costs of the textually-representable protocol are found to be unacceptable and unmitigateable.
The benefit of being able to use pre-existing introspection tools that were not designed with your protocol in mind—-and use those tools without extensive configuration—-is huge.
I think the existence of bad textual formats (e.g. JSON) and the presence of largely-orthogonal-to-binaryness useful affordances in popular binary formats (e.g. schema validation and strong typing in Protobuf) muddies the underlying truth: textual-representability-by-default is rarely costly and often a huge boon to protocol implementors and protocol debuggers.
Cool, readable messages in strace but still some odd binary log format?
He explained it here
https://news.ycombinator.com/item?id=41694711
I do not understand where the 64bit integers not working with JSON comes from.
JSON the format had no limit on integer size. And all Java JSON libraries I know can handle arbitrary prevsion integers (BigInt) and 32/64bit int/long types when serializing and deserializing.
Quick googling shows that also JavaScript has proper support for 64bit integers with their BigInt type, and it can be used to deserialize incoming data correctly with the default parser, albeit requiring a bit more work to annotate the fields.
Personally I often explicitly make sure that the integers I return in trust environments as identifiers in REST APIs are by default longer than 52bits so that buggy parser libraries are caught early.
The number type in JSON is 64 bit float, limiting integers without loss of precision to 2⁵³-1.
BigInt is a new concept and not technically supported. So whether it works in your random library of choice is probably a potshoot. "Use within JSON: Using JSON.stringify() with any BigInt value will raise a TypeError, as BigInt values aren't serialized in JSON by default. " https://developer.mozilla.org/en-US/docs/Web/JavaScript/Refe...
That's not true. The number type in JSON is an arbitrary sized number. You're thinking of JavaScript which is not the same thing.
https://datatracker.ietf.org/doc/html/rfc7493#section-2.2 No.
(which expands on the slightly fluffier text in https://datatracker.ietf.org/doc/html/rfc7159#section-6 which broadly says the same thing... and https://datatracker.ietf.org/doc/html/rfc8259#section-6)
If you look at a parser matrix, flouting this rule is asking for trouble. Which is why the recommendations on MDN for bigint recommend an explicit parsing from a string as needed.
(I'm genuinely curious why this is being downvoted. Does someone out there feel so strongly that because an arbitrary string of characters technically can be parsed as any infinitely large number, that it's wise to ignore RFCs and dozens of existing implementations when making use of JSON as a common interchange format?)
You are mixing up two different standards here, JSON and I-JSON. I-JSON is a subset of JSON with stricter requirements.
JSON recommends (for interoperability) that numbers be limited to those representable as IEEE 64-bit binary floats-but does not require that. A document which ignores that recommendation is still officially valid JSON. By contrast, I-JSON, as a subset of JSON, upgrades that recommendation to a requirement, so documents which ignore it are not valid I-JSON
Yes, they are different standards, but the purpose of I-JSON was specifically to solve interoperability problems. Those interoperability problems are due to the looseness of the original spec.
The others do not forbid larger numbers but note that doing that will make for bad times, and if you look at JSON parsing matrices (say, the "Parsing JSON is a Minefield" one), you can see that the majority of implementations do not do allow larger. So. float64 is a defacto standard and I-JSON is simply clarifying this. Given the main purpose of JSON is an interoperable data exchange format it would be a very bad idea to do otherwise.
https://en.wikipedia.org/wiki/JSON?useskin=vector#Interopera... (essentially same point made here)
yes but no
there is more then one JSON standard and while the JS in JSON stands for JavaScript it doesn't directly means it is using JS types as confusing as it might seem.
the initial definition of JSON by itself has no limit on integer size at all, JSON with number which can only be represented with BigInt _totally is a thing_ just not so much in the Js ecosystem. Furthermore systems accidental expecting full i64 or u64 number ranges are as much a thing, and this leading to subtle bugs the moment a different (de-)serializer is used is another issues.
Qt refused for almost a decade to support deserializing 64-bit integers from JSON because of compatibility concerns.
jq couldn't handle them until relatively recently. This isn't a few bad parsers. You can't assume a json parser will handle bigints correctly and when you're angling to be low-level plumbing and work with every language and software that hasn't been recompiled in years you have to be pretty conservative in what you send.
Yeah, and it isn't like jq is being incorrect in this, the JSON RFCs recommend not using values that can't be represented as a 64 bit float (pasted links to a few RFCs in another response). So if you want to represent a large number safely, better to put it in a string, where special handling can be done after the initial JSON parse without loss of information.
If you fully work within a trusted environment, why bother with JSON? Use your favorite binary serialization with codegen.
The whole point of JSON is almost every programming language can read and write it - and if you want this to be the case, stringify anything unusual, like large integers.
Not a fan of JSON per se for this type of messaging, but I don't think binary parser+validator combinations are that much faster. Plus, this forces C(++) developers to actually validate the data they're exchanging rather than casting raw structs from data buffers, which solves a huge vulnerability. Then again, the lack of 64 bit integers makes the protocol design comically unsuited.
I don't think the serialization/deserialisation overhead matters much when also taking into account the bus connection overhead. I would've preferred a better data exchange format but out of all problems I have with Linux IPC, the performance of JSON is the least of my worries.
The 64bit issue is certainly an issue, but very much overblown.
First of all, in systemd, which is a heavy D-Bus user, we effectively IRL only send integers > 2^53 when we use UINT64_MAX as a special "niche" marker meaning "unlimited"/"unset"/"undefined". But the thing is that JSON has a proper concept for this: the "null" value (or simply not specifying a specific JSON field). Hence, in reasonably clean APIs this is mostly a non-issue.
That said, sd-varlink/sd-json (i.e. systemd's implementation of it), of course is 64bit signed and unsigned integer clean when it processes JSON. More-over it automatically handles if you do what the various specs on the internet suggest you do if you have an integer > 2^53: you encode it as decimal value as a string.
Would it be better if JSON would have been more precise on this, yes. Is it a big issue? No, not at all.
Are strings used for timestamps? Those are usually why I'm sending 64-bit ints over IPC.
To be honest I don't see why they don't use the existing IPC mechanism created for Android - binder and ashmem. It has been tested and developed in real world scenarios for over a decade by now.
Same reason Android (or PalmOS) didn't decide to use any of the existing IPC mechanisms, and why then Luna/webOS also didn't decide to use Binder even though they had experience with it and reinvent something else.
That urge that forces every developer to reinvent RPC every other week.
Not really.
Binder solves a real-world problem — priority inversion. It is also the only IPC on Linux that allows to perform a blocking call from process A to process B (and have process B synchronously call back to process A if needed).
D-Bus, Varlink and other "IPC solutions" on top of sockets can not do those things. Android uses synchronous Binder calls for 90% of it's application APIs, so there is clearly a real use case for it.
Incidentally, Binder itself does not specify message format. A message is a simple memory buffer. Android runtime uses a custom binary serialization format, but you can use JSON or whatever. Binder is an alternative to sockets/pipes, not D-Bus or other wrappers on top of them.
Binder was not even designed for Linux. The primary reason it is used is that (at the time) they wanted to abstract from Linux. "They" here is not Android/Google.
Binder also specifies a message format -- even if not fully, since the kernel is going to peek into your message for things like FDs, binder objects, etc. Or your userspace is going to need to "special treat" these fields somehow.
It competes in the same space as D-Bus no doubt. If the story of Android had been different, with the companies spinning off some years later, it may have very well ended up using Luna RPC (https://www.webosbrew.org/pages/luna-service-bus.html ) (which is practically indistinguishable from D-Bus, bar the JSON) or anything else, really.
> D-Bus, Varlink and other "IPC solutions" on top of sockets can not do those things.
WTF? Sure you can do on top of even pipes. Even XDR could...
"Synchronous" is a very loose word here if you mean to be interrupted in the middle of a RPC call, anyway.
Binder was invented at Be, Inc. for BeOS, and many Be refugees joined Android in the early days. (I just learned this recently on HN, so congratulations!)
> WTF? Sure you can do on top of even pipes. Even XDR could...
Of course, you "can". Implement message-based communication on top of streaming. Emulate blocking calls on top of non-blocking socket API. Implement authentication via ancillary data (try not to throw up in process). Use some clever tricks to solve priority inversion. Somehow distinguish your fds from all others file descriptors to ensure that no confused deputy problems arise.
Congratulations! You have reached feature parity with Binder.
> Binder was not even designed for Linux.
Neither are Berkeley sockets.
(You cannot actually fully solve priority inversion _and priority propagation_ on a fully async system. There are ultimately no tricks clever enough.)
> Of course, you "can".
What is your point here? You claimed that "other IPC solutions" _could not_ do this on top of sockets, but as I've shown and now you admit, you definitely _can_. Obviously I'm not suggesting you roll this by hand, nor claiming that a plain UNIX pipe is a fully featured RPC system; just that there's a million RPC systems out there that do it and don't need anything else than sockets.
I don't know what binder does exactly, but rendez-vous synchronization is surprisingly hard to do efficiently on POSIX systems without excessive context switches.
Varlink was specifically designed to be available as early as possible when the kernel is booting and requires virtually nothing apart from good old sockets. But yes, definitely getting yet another one vibe from this.
Then again, Dbus is not really what I would call an incredible piece of software - we are very much in the adequate territory if even - so anything shaking up the status quo can’t be too bad.
It's probably the first time in 14 years of systemd I appreciate a technical choice coming from the systemd team.
It's a small format, it has limited pitfalls, it's fast to marshal and unmarshal (and with simd gets even more fast), it's deflate-friendly, would performance be an issue they can pack that through CBOR or msgpack or what-have-you.
How many messages is this actually gonna handle? If it's tens per second, should you maybe be using something else?
JSON is so prevalent I'm pretty sure it's a lot of people's mental model of what non relational structured data looks like, if it's not just a file.
It's nice from a dev perspective to always work directly with the same high level concepts that you use when thinking about this stuff.
Other data formats are often advertised in terms of JSON, either subsets or supersets, or equivalent representations.
I'd use something else for high performance stuff.... But for everything else it's pretty awesome.
Lets take it a level up. I'm still not sure why we even need a message bus in there in the first place. The whole Linux boot, init, systemd pile is a Rube Goldberg machine that is very difficult to understand at the moment. The only reason I suspect most people aren't complaining is that our abstraction level is currently a lot higher (docker / kubernetes etc) and machines are mostly ephemeral cattle and few people go near it.
As for JSON, please don't even start me on that. It is one of the worst serialisation decisions we ever made as a society. Poorly defined and unreliable primitive types, terrible schema support and expensive to parse. In a kernel, it does not belong! Give it a few months and varlink will be YAML over the top of that.
I think it doesn't have to be a message bus per-se, that design decision is mostly just because it's convenient. Even D-Bus can actually be used without the bus, if you really want to.
D-Bus is just a bus so it can solve a bunch of different IPC problems at the same time. e.g. D-Bus can handle ACLs/permissions on RPC methods, D-Bus can handle multiple producers and multiple consumers, and so on. I think ultimately all that's really needed is IPC, and having a unified bus is just to allow for some additional use cases that are harder if you're only using UNIX domain sockets directly.
If there are going to be systemd components that have IPC, then I'd argue they should probably use something standard rather than something bespoke. It's good to not re-invent the wheel.
Not that I think Varlink is any better. It seems like at best it's probably a lateral move. I hope this does not go forward.
> If there are going to be systemd components that have IPC, then I'd argue they should probably use something standard rather than something bespoke. It's good to not re-invent the wheel.
This is my point.
My favourite DBus situation a number of years ago was a CentOS 7 box that reboot stopped working on with a cryptic DBus error that no one has ever seen before. I had to sync it, power cycle the node from the ILO card and cross my fingers.
I really don't give a shit about this. I just wanted to run my jobs on the node, not turn into a sysadmin due to someone else's dubious architectural decisions.
Yes but the systemd developers don't want to implement their own protocols with e.g. ACL checking, and given some of their track record I kind of think you don't want them to, either. I'm pretty sure the error conditions would be even more bespoke if they "just" used UNIX domain sockets directly. Don't get me wrong, there's nothing particularly wrong with UNIX domain sockets, but there's no "standard" protocols for communicating over UDS.
This is systemd we’re talking about. A service manager that already mucks with mount namespaces.
It would be quite straightforward to map a capability-like UNIX socket into each service’s filesystem and give it a private view of the world. But instead…
> Public varlink interfaces are registered system-wide by their well-known address, by default /run/org.varlink.resolver. The resolver translates a given varlink interface to the service address which provides this interface.
…we have well known names, and sandboxing, or replacing a service for just one client, remains a mess. Sigh.
Please, your trolling is not really welcome.
> It would be quite straightforward to map a capability-like UNIX socket into each service’s filesystem and give it a private view of the world. But instead…
Can you link to your PR where you solved the problem?
Well there sort of is but people don't tend to know or use it. If it's within the same machine and architecture, which should be the case for an init system, then a fixed size struct can be written and read trivially.
C structs are a terrible serialization format, since they are not a serialization format at all. Nothing guarantees that you will get consistent struct behavior on the same machine, but also, it only really solves the problem for C. For everything else, you have to duplicate the C structure exactly, including however it may vary per architecture (e.g. due to alignment.)
And OK fine. It's not that bad, most C ABIs are able to work around this reasonably OK (not all of them but sure, let's just call it a skill issue.) But then what do you do when you want anything more complicated than a completely fixed-size type? Like for example... a string. Or an array. Now we can't just use a struct, a single request will need to be split into multiple different structures at the bare minimum.
And plus, there's no real reason to limit this all to the same machine. Tunneling UNIX domain sockets over the network is perfectly reasonable behavior and most* SSH implementations these days support this. So I think scoping the interoperability to "same machine" is unnecessarily limiting, especially when it's not actually hard to write consistent de/serialization in any language.
* At least the ones I can think of, like OpenSSH[1], Go's x/crypto/ssh[2], and libssh2[3].
[1]: https://www.openssh.com/txt/release-6.7
[2]: https://pkg.go.dev/golang.org/x/crypto/ssh#Client.ListenUnix
[3]: https://github.com/libssh2/libssh2/pull/945
BTW, Lustre's RPC system's serialization is very much based on C structs. It's receiver-makes-right to deal with endianness, for example. It's a pain, but it's also fast.
Making an RPC serialization system that is zero-overhead i.e. can use the same format on the wire as it does on disk is not a terrible idea. Capnp is the serialization format that I've been suggesting as a potential candidate and it is basically just taking the idea of C structs, dumping it into it's own schema language, and adding the bare minimum to get some protobuf-like semantics.
Well, Lustre RPC doesn't use on-disk data structures on the wire, though that is indeed an interesting idea.
In Lustre RPC _control_ messages go over one channel and they're all a C structure(s) with sender encoding hints so the receiver can make it right, and any variable-length payloads go in separate chunks trailing the C structures.
Whereas bulk _data_ is done with RDMA, and there's no C structures in sight for that.
Capnp sounds about right for encoding rules. The way I'd do it:
(That's roughly what Lustre RPC does.)As for syntax, I'd build an "ASN.2" that has a syntax that's parseable with LALR(1), dammit, and which is more like what today's devs are used to, but which is otherwise 100% equivalent to ASN.1.
Out of curiosity, why not use offsets instead of pointers? That's what capnp does. I assume offset calculation is going to be efficient on most platforms. This removes the need for fixing up pointers; instead you just need to check bounds.
It's more work for the sender, but the receiver still has to do the same amount of work as before to get back to actual pointers. So it seems like pointless work.
Having actual interior pointers means not having to deal with pointers as offsets when using these objects. Now the programming language could hide those details, but that means knowing or keeping track of the root object whenever traversing those interior pointers, which could be annoying, or else encoding an offset to the root and an offset to the pointed-to-item, which would be ok, and then the programming language can totally hide the fact that interior pointers are offset pairs.
I've a feeling that fixing up pointers is the more interoperable approach, but it's true that it does more memory writes. In any case all interior pointers have to be validated on receiving -- I don't see how to avoid that (bummer).
What a fun sub-thread.
Note within the domain of this problem was the point. Which means on the same machine, with the same architecture and both ends being C which is what the init system is written in.
You are adding more problems that don't exist to the specification.
As for strings, just shove a char[4096] in there. Use a bit of memory to save a lot of parsing.
> You are adding more problems that don't exist to the specification.
D-Bus does in fact already have support for remoting, and like I said, you can tunnel it today. I'm only suggesting it because I have in fact tunneled D-Bus over the network to call into systemd specifically, already!
> As for strings, just shove a char[4096] in there. Use a bit of memory to save a lot of parsing.
OK. So... waste an entire page of memory for each string. And then we avoid all of that parsing, but the resulting code is horribly error-prone. And then it still doesn't work if you actually want really large strings, and it also doesn't do much to answer arrays of other things like structures.
Can you maybe see why this is compelling to virtually nobody?
Even being run on the same machine doesn't guarantee two independent processes agree on C struct layout compiled from the same source. For one, you could have something as simple as one compiled for 32bit, one 64, but even then compiler flags can impact struct layout.
> As for strings, just shove a char[4096] in there.
For the love of God, use a proper slice/fat pointer, please.
Switching over to slices eliminates 90%+ of the issues with using C. Carrying around the base and the length eliminates a huge number of the overrun issues (especially if you don't store them consecutively).
Splitting the base and the offset gives a huge amount of semantic information and makes serialization vastly easier.
Broadly, I agree with you. C strings were a mistake. The standard library is full of broken shit that nobody should use, and worse, due to the myriad of slightly different "safe" string library functions, (a completely different subset of which is supported on any given platform,) which all have different edge cases, many people are over-confident that their C string code is actually correct. But is it? Is it checking errors? Does your function ensure that the destination buffer is null-terminated when it fails? Are you sure you don't have any off-by-one issues anywhere?
Correct as you may be though the argument here is that you should just write raw structs into Unix sockets. In this case you can't really use pointers. So, realistically, no slices either. In this context a fixed-size buffer is quite literally the only sensible thing you can do, but also, I think it's a great demonstration of why you absolutely shouldn't do this.
That said, if we're willing to get rid of the constraint of using only one plain C struct, you could use offsets instead of pointers. Allocate some contiguous chunk of memory for your entire request, place struct/strings/etc. in it, and use relative offsets. Then on the receiver side you just need some fairly basic validation checks to ensure none of the offsets go out of bounds. But at that point, you've basically invented part of Cap'n'proto, which begs the question... Why not just use something like that instead. It's pretty much the entire reason they were invented.
Oh well. Unfortunately the unforced errors of D-Bus seem like they will lead to an overcorrection in the other direction, turning the core of our operating system into something that I suspect nobody will love in the long term.
> But at that point, you've basically invented part of Cap'n'proto
The only problem I have with Cap'n Proto is that the description is external to the serialization. Ideally I'd like the binary format to have a small description of what it is at the message head so that people can process messages from future versions.
ie. Something like: "Hmm, I recognize your MSG1 hash/UUID/descriptor so I can do a fast path and just map you directly into memory and grab field FD. Erm, I don't recognize MSG2, so I need to read the description and figure out if it even has field FD and then where FD is in the message."
I thought about this for a bit. I think largely to do things with messages you don't know about is probably a bad idea in general; writing code that works this way is bound to create a lot of trouble in the future, and it's hard to always reason about from every PoV. However, there are some use cases where dealing with types not known at compile-time is useful, obviously debugging tools. In that case I think the right thing to do is just have a way to look up schemas based on some sort of identity. Cap'n'proto is not necessarily the greatest here: It relies on a randomly-generated 64-bit file identifier. I would prefer a URL or perhaps a UUID instead. Either way, carrying a tiny bit of identity information means that the relatively-niche users who need to introspect an unknown message don't cause everyone else to need to pay up-front for describability, and those users that do need introspection can get the entire schema rather than just whatever is described in the serialized form.
It's better to design APIs to be extensible in ways that doesn't require dynamic introspection. It's always possible to have a "generic" header message that contains a more specific message inside of it, so that some consumers of an API can operate on messages even when they contain some data that they don't understand, but I think this still warrants some care to make sure it's definitely the right API design. Maybe in the future you'll come to the conclusion it would actually be better if consumers don't even try to process things they're not aware of as the semantics they implement may some day be wrong for a new type of message.
> I think largely to do things with messages you don't know about is probably a bad idea in general
Versioning, at the least, is extremely difficult without this.
Look at the Vulkan API for an example of what they have to do in C to manage this. They have both an sType tag and a pNext extension pointer in order for past APIs to be able to consume future versions.
But how do you know that the field called "FD" is meaningful if the message is a totally different schema than the one you were expecting?
In general there's very little you can really do with a dynamic schema. Perhaps you can convert the message to JSON or something. But if your code doesn't understand the schema it received, then it can't possibly understand what the fields mean...
Varlink is not a message bus. Hence you should be happy?
Yeah, how will number/float serialization go? Are we going to serialize them as strings and parse them? That abstraction isn't handled the same way across multiple languages.
> I'm still not sure why we even need a message bus in there in the first place.
Because traditional POSIX IPC mechanisms are absolute unworkable dogshit.
> It is one of the worst serialisation decisions we ever made as a society.
There isn't really any alternative. It's either JSON or "JSON but in binary". (Like CBOR.) Anything else is not interoperable.
There are a world of serialization formats that can offer a similar interoperability story to JSON or JSON-but-binary formats. And sure, implementing them in every language that someone might be interested in using them in might require complication, but:
- Whatever: people in more niche languages are pretty used to needing to do FFI for things like this anyhow.
- Many of them already have a better ecosystem than D-Bus. e.g. interoperability between Protobuf and Cap'n'proto implementations is good. Protobuf in most (all?) runtimes supports dynamically reading a schema and parsing binary wire format with it, as well as code generation. You can also maintain backwards compatibility in these formats by following relatively simple rules that can be statically-enforced.
- JSON and JSON-but-binary have some annoying downsides. I really don't think field names of composite types belong as part of the ABI. JSON-like formats also often have to try to deal with the fact that JSON doesn't strictly define all semantics. Some of them differ from JSON is subtle ways, so supporting both JSON and sorta-JSON can lead to nasty side-effects.
Maybe most importantly, since we're not writing software that's speaking to web browsers, JSON isn't even particularly convenient to begin with. A lot of the software will be in C and Rust most likely. It helps a bit for scripting languages like Python, but I'm not convinced it's worth the downsides.
Sorry, but bash isn't a "niche language" and it doesn't have an FFI story.
Behold, FFI for bash: https://github.com/taviso/ctypes.sh
I don't know how to tell you this, but, you don't need to implement an RPC protocol in bash, nor do you need FFI. You can use CLI tools like `dbus-send`.
I pray to God nothing meaningful is actually doing what you are insinuating in any serious environment.
I'm trying to tell you that something that isn't straceable and greappable shouldn't belong in your system services stack.
FFI is the shell's only job.
This is a quite frankly ridiculous point. Most of that garb came from the HPC people who built loads of stuff on top of it in the first place. It's absolutely fine for this sort of stuff. It's sending the odd little thing here and there, not on a complex HPC cluster.
As for JSON, are you really that short sighted that it's the only method of encoding something? Is "oh well it doesn't fit the primitive types, so just shove it in a string and add another layer of parsing" acceptable? Hell no.
> ...it's the only method of encoding something?
If you want something on the system level parsable by anything? Yes it is.
protobufs / asn.1 / structs ...
Edit: hell even XML is better than this!
Structs are a part of C semantic. They are not an ipc format. You can somewhat use them like one if you take a lot of precaution about how they are laid out in memory including padding and packing but it’s very brittle.
Asn.1 is both quite complicated and not very efficient.
They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice?
I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part.
ASN.1 BER/DER is more or less the same thing as CBOR. The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”).
That "convoluted telco/ITU-T style" yields amazingly high quality specifications. I'll take X.68x / X.69x any day over most Internet RFCs (and I've written a number of Internet RFCs). The ITU-T puts a great deal of effort into its specs, or at least the ASN.1 working group did.
ASN.1 is not that complicated. Pity that fools who thought ASN.1 was complicated re-invented the wheel quite poorly (Protocol Buffers I'm looking at you).
For our sins, our industry is doomed to suffer under unbearable weight of endless reinvented wheels. Of course it would have been better to stick with ASN.1. Of course we didn't, because inexperience and hubris. We'll never learn.
It sure seems that way. Sad. It's not just hubris nor inexperience -- it's cognitive load. It's often easier to wing something that later grows a lot than it is to go find a suitable technology that already exists.
One thing I liked about a Vernor Vinge sci-fi novel I read once was the concept of "computer archeologist". Spool the evolution of software forwards a few centuries, and we'll have layers upon layers of software where instead of solving problems with existing tooling, we just plaster on yet another NIH layer. Rinse and repeat, and soon enough we'll need a separate profession of people who are capable of digging down into those old layers and figure out how they work.
> The perceived complexity of ASN.1 comes from the schema language and specifications written in the convoluted telco/ITU-T style (and well, the 80's type system that has ~8 times two different types for “human readable string”).
I can’t resist pointing that it’s basically a longer way of saying quite complicated and not very efficient.
> I can’t resist pointing that it’s basically a longer way of saying quite complicated and not very efficient.
That's very wrong. ASN.1 is complicated because it's quite complete by comparison to other syntaxes, but it's absolutely not inefficient unless you mean BER/DER/CER, but those are just _some_ of the encoding rules available for use with ASN.1.
To give just one example of "complicated", ASN.1 lets you specify default values for optional members (fields) of SEQUENCEs and SETs (structures), whereas Protocol Buffers and XDR (to give some examples) only let you specify optional fields but not default values.
Another example of "complicated" is that ASN.1 has extensibility rules because the whole "oh TLV encodings are inherently extensible" thing turned out to be a Bad Idea (tm) when people decided that TLV encodings were unnecessarily inefficient (true!) so they designed efficient, non-TLV encodings. Well guess what: Protocol Buffers suffers from extensibility issues that ASN.1 does not, and that is a serious problem.
Basically, with a subset of ASN.1 you can do everything that you can do with MSFT RPC's IDL, with XDR, with Protocol Buffers, etc. But if you stick to a simple subset of ASN.1, or to any of those other IDLs, then you end up having to write _normative_ natural language text (typically English) in specifications to cover all the things not stated in the IDL part of the spec. The problem with that is that it's easy to miss things or get them wrong, or to be ambiguous. ASN.1 in its full flower of "complexity" (all of X.680 plus all of X.681, X.682, and X.683) lets you express much more of your protocols in a _formal_ language.
I maintain an ASN.1 compiler. I've implemented parts of X.681, X.682, and X.683 so that I could have the compiler generate code for the sorts of typed holes you see in PKI -all the extensions, all the SANs, and so on- so that the programmer can do much less of the work of having to invoke a codec for each of those extensions.
A lot of the complexity in ASN.1 is optional, but it's very much worth at least knowing about it. Certainly it's worth not repeating mistakes of the past. Protocol Buffers is infuriating. Not only is PB a TLV encoding (why? probably because "extensibility is easy with TLV!!1!, but that's not quite true), but the IDL requires manual assignment of tag values, which makes uses of the PB IDL very ugly. ASN.1 originally also had the manual assignment of tags problem, but eventually ASN.1 was extended to not require that anymore.
Cavalier attitudes like "ASN.1 is too complicated" lead to bad results.
> That's very wrong. ASN.1 is complicated because it's quite complete by comparison to other syntaxes
So, it's quite complicated. Yes, what I have been saying from the start. If you start the conversation by "you can define a small subset of this terrible piece of technology which is bearable", it's going to be hard convincing people it's a good idea.
> Cavalier attitudes like "ASN.1 is too complicated" lead to bad results.
I merely say quite complicated not too complicated.
Still, ASN.1 is a telco protocol through and through. It shows everywhere: syntax, tooling. Honestly, I don't see any point in using it unless it's required by law or by contract (I had to, I will never again).
> but it's absolutely not inefficient unless you mean BER/DER/CER, but those are just _some_ of the encoding rules available for use with ASN.1.
Sorry, I'm glade to learn you can make ASN.1 efficient if you are a specialist and now what you are doing with the myriad available encodings. It's only inefficient in the way everyone use it.
> So, it's quite complicated.
Subsets of ASN.1 that match the functionality of Protocol Buffers are not "quite complicated" -- they are no more complicated than PB.
> Still, ASN.1 is a telco protocol through and through.
Not really. The ITU-T developed it, so it gets used a lot in telco protocols, but the IETF also makes a lot of use of it. It's just a syntax and set of encoding rules.
And so what if it were "a telco protocol through and through" anyways? Where's the problem?
> It shows everywhere: syntax, tooling.
The syntax is very much a 1980s syntax. It is ugly syntax, and it is hard to write a parser for using LALR(1) because there are cases where the same definition means different things depending on what kinds of things are used in the definition. But this can be fixed by using an alternate syntax, or by not using LALR(1), or by hacking it.
The tooling? There's open source tooling that generates code like any XDR tooling and like PB tooling and like MSFT RPC tooling.
> Sorry, I'm glade to learn you can make ASN.1 efficient if you are a specialist and now what you are doing with the myriad available encodings. It's only inefficient in the way everyone use it.
No, you don't have to be a specialist. The complaint about inefficiency is about the choice of encoding rules made by whatever protocol spec you're targeting. E.g., PKI uses DER, so a TLV encoding, thus it's inefficient. Ditto Kerberos. These choices are hard to change ex-post, so they don't change.
"[T]he way everyone use it" is the way the application protocol specs say you have to. But that's not ASN.1 -- that's the application protocol.
> Structs are a part of C semantic.
Uh, no, structs, records, whatever you want to call them, are in many, if not most programming languages. "Structs" is not just "C structs" -- it's just shorthand for "structured data types" (same as in C!).
> Asn.1 is both quite complicated and not very efficient.
Every rich encoding system is complicated. As for efficiency, ASN.1 has many encoding rules, some of which are quite bad (BER/DER/CER, which are the first family of ERs, and so many thing ASN.1 == BER/DER/CER, but that's not the case), and some of which are very efficient (PER, OER). Heck, you can use XML and JSON as ASN.1 encoding rules (XER, JER).
> They could certainly have gone with protobufs or another binary serialisation format but would it really be better than the current choice?
Protocol buffers is a tag-length-value (TLV) encoding, same as BER/DER/CER. Having to have a tag and length for every value encoded is very inefficient, both in terms of encoding size as well as in terms of computation.
The better ASN.1 ERs -PER and OER- are much more akin to XDR and Flat buffers than to protobufs.
> I don’t think the issue they are trying to solve is related to serialisation anyway. Seems to me they are unhappy about the bus part not the message format part.
This definitely seems to be be the case.
> Uh, no, structs, records, whatever you want to call them
It's plenty clear from discussion context that OP is talking about C struct but yes, replace C with any languages which suit you. It will still be part of the language semantic and not an IPC specification.
The point is you can't generally use memory layout as an IPC protocol because you generally have no guarantee that it will be the same for all architectures.
If it's IPC, it's the same architecture (mostly; typically there's at most 3 local architectures). The receiver can always make right. If there's hidden remoting going on, the proxies can make things right.
> The receiver can always make right.
Certainly but that’s hardly structs anymore. You are implicitly defining a binary format which is aligned on the sender memory layout then.
It's "structs" when the sender and receiver are using the same architecture, and if they're using the same int/long/pointer sizes then the only work to do is swabbing and pointer validation / fixups. That's a lot less work than is needed to do just about any encoding like protocol buffers, but it's not far from flat buffers and capnp.
Thank goodness they didn’t pick YAML though.
Yet!
You don't quite understand how this works.
One requirement is being able to strace a misbehaving service and figure out quickly what it's sending and receiving.
This is a system-level protocol, not just a random user app.
I think it is primarily for compatibility. Practically every language has a readily available library, often part of the standard library, for json. That is less true for something like Msgpack or cbor, and even less so if make a new binary format.
Also, there are benefits to the format being human readable.
As a little bit of perspective, I thought along the same lines when I saw the first web browser. Text was good, but why did they have all these inline images. It was so wasteful of bandwidth.
(also, unix uses files for everything, including /proc talking to drivers!)
This looks like a joke, naturally D-BUS without JSON is what is preventing Linux on Desktop to take off.
You surely are aware that systemd and dbus aren't only used on desktop, right?
A side effect of its adoption, D-BUS was created as replacement for Bonobo and DCOP.
That was 15 years ago? I was talking about the usage today.
And I was being sarcastic, as there are things much more relevant to spend brain cells on.
Wasn't sarcasm frowned upon here?
[dead]
Sigh. Cap'n Proto already exists. Reinventing the wheel yet again because NIH.
(Varlink also isn't something new)
CapnProto is kind of awful though. I had to use it for several years and the API is extremely unfriendly.
Which implementation were you using? (I.e. which programming language?)
C++
> So Varlink requires a proper specification of message types, but then uses god damn JSON to serialize messages? Why would you do that?
Someone took a look at D-Bus and decided "You know what? It just isn't shitty enough. It's 2024 and D-Bus runs halfway decently on the average laptop. Let's fix that."
I wouldn't be surprised if the Scooby Gang pulled the mask off the head dev, revealing ThePrimeagen, in a scheme of long-form comedy that's a cross between one of his "let's write a network protocol" streams and his new enterprise development satire series.
Why don't they just adopt the same fundamental IPC design as Wayland? Swap out the root wl_display singleton object for something dbus appropriate and away you go?
More or less all the display specific stuff is an extension on top of the core design.
https://wayland-book.com/protocol-design.html
I often wonder why that protocol isn’t used in other fields. It has clear schemas with documentation, and existing codegen implementations.
OTOH, it’s too complex for simple protocols with a few messages.
IIRC, Pipewire uses a variation of the Wayland protocol.
because systemd does what redhat clients ask for, not matter how obnoxious.
That would be going full circle, given that D-Bus was predated by Bonono on GNOME side, and DCOP on KDE, which was based on X11 Inter-client communication.
D-Bus evolved from merging their needs into a single approach.
>reverse-domain name
Why do people keep replicating this terrible design decision? It puts the lowest-entropy part of the name at the beginning, so that you have to parse (or type) the maximum number of characters before you get a unique specification. Try tab-completing a "flatpak run" command sometime.
You might be interested in https://github.com/Aloxaf/fzf-tab
Setting the serialization format issue aside, why is the D-Bus broker an issue?
Why was it designed that way in the first place if all we need (supposedly) is a socket and a simple request/response protocol? If it wasn't needed, why does D-Bus have it? And if it is needed, what will replace it when systemd switches to Varlink?
Accompanying talk from Lennart Poettering a few days ago:
https://media.ccc.de/v/all-systems-go-2024-276-varlink-now-
Related. Others?
AllSystemsGo: Varlink Now [video] - https://news.ycombinator.com/item?id=41670431 - Sept 2024 (2 comments)
Varlink: Interface description format and protocol for humans and machines - https://news.ycombinator.com/item?id=25621936 - Jan 2021 (5 comments)
Varlink – A plain-text, type-safe, discoverable, self-documenting interface - https://news.ycombinator.com/item?id=20950146 - Sept 2019 (10 comments)
https://hn.garglet.com/similar/stories/41687413
Does this mean Lunux will need both Dbus and varlink if using systemd ? I ask because I believe Firefox uses Dbus.
https://www.phoronix.com/news/Systemd-Varlink-D-Bus-Future
systemd has been using varlink for a while now along side dbus, its not something just now being introduced. They don't have a problem living side-by-side.
It's more code, more bloat, more complexity, and more attack surface area.
Ok? I don't see how that is relavant to the question at hand (to be fair, I did assume the parent meant "will this cause any problems since firefox relies on dbus?").
Anything that add new features (early boot IPC in this case) is going to require "more code, more bloat, more complexity, and more attack surface area" unless you rip other features out at the same time.
[dead]
Hello Nth system syndrome.
That looks interesting, I especially appreciate the intentional design decision for documentation and schemas. While it didn't work quite well with dbus Ime it is still a good idea to make APIs more accessible.
There is something impressive about Lennart’s work, in that treads a thin line between “completely triggering” to some people and “genuinely quite a good idea” to others.
Lots of discussion here has been around the choice of JSON as a format, with some valid points being brought up about specific limitations and possible footguns.
But these miss the point: this massively moves the needle with regard to lowering the barrier to entry when writing services. I’m confident I can write a simple Varlink service in about 20 lines of Python, with no dependencies after reading the linked docs. You cannot in good faith say the same thing about dbus.
That’s nuts. Yeah - if my service is for some reason sending/receiving high-precision 128 bit numbers or huge binary files then it’s not perfect.
But nothing really does that. It’s not worth making the entire stack more complex and less performant for a possible use case that doesn’t really fit.
He's living churn. It's just constant change often for gains that don't matter to a lot of people and losees that do. Linux is radically different because of him. People were deeply invested in how things were. Think about how hard it is to learn a laguage as an adult compared to when you're a kid. Even when some of his stuff is amazing, it's still going to cause stress for a lot of people due to its radical nature.
D-Bus is 17 years old by now, introducing a replacement is hardly "living churn", especially since D-Bus will still be supported, probably for years.
If that were all it is, I'd agree but you may have missed the constant stream of other radical changes over the years.
But what is the excuse to start from scratch? Rather than a compat layer that convinces the world that raw your interfaces, libraries and clients are so much nicer to work with that the other thing should be demoted to compat layer eventually?
>But what is the excuse to start from scratch?
Red Hat FOSS system software is imposing foss-ified client work to the general public. The cloud industry benefited greatly from their work, the other users are involved in a multi-decade flamewar.
[dead]
Someone explain to me please, how type-safe and JSON can even exist in the same sentence? JSON has so many edge cases for handling numbers, floats, bools etc. Why would you not use gRPC or even something like CapnProto if you want to maintain that type-safety-ness?
Which edge cases do you mean? Are you confusing it with YAML? I know YAML has some serious footguns [1]. In JSON I'm not aware of any.
[1] https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-fr...
I think that there are problems with both Varlink and D-Bus. JSON is one of them but not only one. One of the problems with JSON is the data types it uses. There is no support for character codes other than Unicode (I think that "Unicode string types considered harmful"; they are not needed for handling text (even Unicode text)), and binary data must be encoded as hex or base64, which just makes it inefficient. There is other problems too, including problems with implementations (including JavaScript, which has a big integer type now but it is not the same type as numbers in JSON). I also think there are problems with the way that the message buses are working. D-Bus has some problems and Varlink has some other problems. And, it isn't true that "the whole world speaks JSON".
Yeah, using serialization format that doesn't even have obvious way of representing every possible Linux filename (nothing mandates filenames to be valid UTF-8, only thing they can't contain is NUL and '/') seems bad fit for low-level system services.
I can't wait to have apps and services crash and burn left and right due to accidentally (or maliciously) created non-UTF8 filenames in world readable directories.
> Yeah, using serialization format that doesn't even have obvious way of representing every possible Linux filename
That is hardly the only problem with such a serialization format, although it is one of them.
> I can't wait to have apps and services crash and burn left and right due to accidentally (or maliciously) created non-UTF8 filenames in world readable directories.
It is not only accidentally or maliciously. Sometimes it may also be done deliberately because it is a character set other than Unicode, without the intention to be malicious.
(A user might also deliberately want to display file names using a different character set, perhaps in order to ensure that all characters can be displayed unambiguously. Of course, this is not necessarily what everyone intends, but some people will.)
Cramming more NIH garbage down GNU/Linux users' throats.
Nanomsg ng and cap'n proto already exist.
Looks like the transformation of RHEL into a Windows clone is nearly complete. I'm all for it, as I expect a modern fork without systemd will coalesce for those of us that prefer Linux over Windows.
This is JSON-RPC basically. Microkernel architecture personified where everything is a service. Unix was all about files. The emerging next standard is probably an OS built entirely of services.
This is overblown reaction t
Next time someone asks you why computers are getting slower despite hardware being faster, everyone who thinks this is a good idea, please remember to blame yourselves.
All the the hate for json here, but at the bottom there is a bridge screenshot. Varlink is designed such that it can trivially be piped across different systems and be maximally interoperable due simple to parse and encode protocol. We might end up with some IPC interface that a lot me tooling can understand. Eg imagine that instead of some daemons supporting sighup to reload, they could also confirm that they reloaded, provided Prometheus style metrics etc
Who is behind this?
I just happened to read Lennart Pottering's original post on sd-dbus yesterday actually. It gives a good introduction to D-Bus for those not up to date.
https://0pointer.net/blog/the-new-sd-bus-api-of-systemd.html
Christ almighty, I have seldom seen a technical proposal as bad as this one. I want off Mr. Poettering's wild ride. No 64 bit integers? No file descriptor transport? No message sequence numbers? Just cut it out already. Dbus doesn't do something? Extend dbus. Don't make some other random and worse things.
> Extend dbus.
How would you address the issue of dbus unavailability in early boot listed in the presentation via extension?
How have we survived without such a feature up to now?
It is a common occurrence that whatever Lennart comes up with fulfills several, erm, novel requirements while not fulfilling some old ones that aren't even mentioned. The parallels to binary logging are there. "Fancy" features that no one asked for - check. Simplicity and thoughtful use of existing functionality (the "full exploitation" principle of the Unix design philosophy) - nope.
>No 64 bit integers? No file descriptor transport? No message sequence numbers?
Are these required or useful in this context?
>Dbus doesn't do something? Extend dbus.
How are you so certain that this is the best use of resources, do you have personal XP in the problem domain?
> Don't make some other random and worse things.
How did you determine that these changes are "worse" or "random?"
In json, with their baked in schema, I think that someone may encode a DEC64[1] number as... string, array, or whatever.
[1] https://www.crockford.com/dec64.html
Hold on, (even though kicking JSON while it's down is funny) how do you make sure that the reply you've got is to the call you issued with this thing? Surely not pinky promise?
They finished varlink in 2018 already https://github.com/varlink/libvarlink
Is there also a varlink dump utility?
[dead]
[flagged]
[flagged]
Is modularity an unalloyed good? Modularity comes with tradeoffs that mean you end up with, well, a bunch of discrete modules rather than something that works cohesively. That's why systemd was adopted pretty much everywhere, ignoring the arbitrary modular boundaries results in a more useful tool.
Some analogs: modern filesystems like ZFS and BTRFS that combine volume management with the filesystem, every service that's consciously chosen to deploy a monolith instead of microservices, and so on, every deployment that chooses to statically link, etc.
Poettering does work for microsoft.
“but its just an init system, you don’t have to use these pieces!“
/s
Here’s a vote for Devuan - you don’t have to accept systemD if you don’t want to. I didn’t.
So does this mean I can shove SQLite databases into the file structure, abstract away an entire SQLite engine into a single header that you are forced to use, and declare that IPC? And you know, since we _are_ already hiding databases within the file structure, why not use it to also unify configuration and installation files too? What could go wrong?
I’m having trouble following what you mean.
Yes, you can certainly send a whole SQLite database using any binary communication protocol as a way to exchange data today. You can even compress it before if you feel like it.
It will not make for a good ipc format because, well, it’s not designed to be one but it would most definitely work.
What’s your point?
There is this one somewhat widely deployed niche system that uses sqlite databases as RPC serialization format. The original idea behind that was limited virtual address space of the CE-based embedded devices, but the design got stuck.
Naming the system would be useful context to add to this point.
Could you name it?
My point is, what's stopping me from creating yet another IPC standard?
Nothing. It might not see adoption, but you can create as many IPC standards as you desire.