Chicken and Egg
For a long time we tried to solve the early boot IPC problem. IPC problem you ask? Well, in the early boot phase we cannot talk D-BUS, because the daemon is not running yet, but to bring up the system, so that the D-BUS daemon can run (mount stuff, load modules, start other services), we need some kind of IPC. Therefore all the early boot daemons have a fallback IPC via unix domain sockets with its own homegrown protocol.
Moving the D-BUS daemon in the kernel was our first approach, but with all the functionality the D-BUS daemon has (signal broadcasting, etc.), there was no clean solution possible, which satisfied all kernel devs.
BUS1 was written, but turned out overly complex and was just a transport without a protocol.
While BUS1 was developed as a transport, our team was looking for a protocol and tried simplifying the D-BUS protocol. After a lot of iterations not even the variant data protocol was left. Instead Kay Sievers and Lars Karlitski came up with a super simple JSON based IPC protocol, which only needs a point-to-point connection, so BUS1 was not even needed.
The varlink protocol is so simple, that language bindings can be implemented easily and using varlink does not result into spending more time coding for IPC than for the actual problem.
A service for resolving interface names into connection URLs and starting the executable has been written, which resembles some of the D-BUS functionality of resolving and activating services.
The most important feature of varlink though, is in the interface definition files. They are human readable and can be even discussed amongst people, which are not developing the implementation. They enable a “checks and balance” system for product management, customers, quality engineering and software developers. Interface stability and backwards compatibility can be enforced easily.
To quote the varlink Ideals page:
Varlink aims to be as simple as possible. It is not specifically optimized for anything else but ease-of-use and maintainability.
Varlink services describe themselves; with a machine-readable interface definition, and human-readable documentation.
It should be easy to forward, proxy, redirect varlink interfaces over any connection-oriented transport. Varlink should be free of any side-effects of local APIs. All interactions need to be simple messages on a network, not carrying things like file descriptors or references to locally stored files.
Varlink is the protocol and definition of interfaces, but it does not define or provide any significant functionality itself.
Varlink errors should carry enough information to be consumed by machines and automated rules.
Making it easy to run automated tests against varlink interfaces should be considered a necessity when designing interfaces.
Right now the API to a Linux system consists of:
- kernel: ioctls, syscalls, procfs, sysfs
- CLI: options, stdout output to be parsed
- D-BUS, Protobuf, various other IPC, homegrown IPC protocols, …
If the adoption of varlink takes off, then the collection of interfaces could form a common Linux System API.
These interface definitions can be inspected even at runtime via a common interface named org.varlink.service, which every service provides. With this mechanism it is very easy for interpreter languanges to create bindings at runtime, as we will see in the next chapter.
Varlink for Python
Ready for some example on how to use varlink with python?
First we need some packages to install on Fedora:
$ sudo dnf copr enable "@varlink/varlink" $ sudo dnf install fedora-varlink $ sudo setenforce 0 # needed until systemd is able to create sockets in /run $ sudo systemctl enable --now org.varlink.resolver.socket $ varlink help
In this example, we will call an interface called com.redhat.system.accounts which has the following varlink interface:
interface com.redhat.accounts type Account ( name: string, uid: int, gid: int, full_name: string, home: string, shell: string ) # Retrieve a list of account information for all known accounts method GetAll() -> (accounts: Account) # Retrieve the account information for a specific user ID method GetByUid(uid: int) -> (account: Account) # Retrieve the account information method GetByName(name: string) -> (account: Account) # Add new account method Add(account: Account) -> (account: Account) error NotFound () error CreationFailed (field: string)
From reading the interface definition, the API should be understandable, even for people, who have never seen a varlink interface file.
Now, let’s call the service from some python code:
from varlink import Client client = Client(resolve_interface='com.redhat.accounts') accounts = client.open('com.redhat.accounts', namespaced=True) ret = accounts.GetByName("root") print(ret)
namespace(account=namespace(full_name='root', gid=0, home='/root', name='root', shell='/bin/bash', uid=0))
print(ret.account.full_name, ret.account.uid, ret.account.shell)
root 0 /bin/bash
That’s it! And the varlink python module is also very small.
Stay tuned for more and go read Jens Reinmann’s Java Varlink blog post.