Varlink

Harald Hoyer December 18, 2017

For a long time we tried to solve the early boot IPC problem. IPC problem you ask? Well, in the early boot phase we cannot talk D-BUS, because the daemon is not running yet, but to bring up the system, so that the D-BUS daemon can run (mount stuff, load modules, start other services), we need some kind of IPC. Therefore all the early boot daemons have a fallback IPC via unix domain sockets with its own homegrown protocol.

K-DBUS

Moving the D-BUS daemon in the kernel was our first approach, but with all the functionality the D-BUS daemon has (signal broadcasting, etc.), there was no clean solution possible, which satisfied all kernel devs.

BUS1

BUS1 was written, but turned out overly complex and was just a transport without a protocol.

While BUS1 was developed as a transport, our team was looking for a protocol and tried simplifying the D-BUS protocol. After a lot of iterations not even the variant data protocol was left. Instead Kay Sievers and Lars Karlitski came up with a super simple JSON based IPC protocol, which only needs a point-to-point connection, so BUS1 was not even needed.

KISS

The varlink protocol is so simple, that language bindings can be implemented easily and using varlink does not result into spending more time coding for IPC than for the actual problem. Bindings for C, Javascript, Go, Java, Python and Rust were quickly implemented (some still only as a proof of concept). A service for resolving interface names into connection URLs and starting the executable has been written, which resembles some of the D-BUS functionality of resolving and activating services.

APIs

The most important feature of varlink though, is in the interface definition files. They are human readable and can be even discussed amongst people, which are not developing the implementation. They enable a “checks and balance” system for product management, customers, quality engineering and software developers. Interface stability and backwards compatibility can be enforced easily. To quote the varlink Ideals page:

Simplicity

Varlink aims to be as simple as possible. It is not specifically optimized for anything else but ease-of-use and maintainability.

Discoverability

Varlink services describe themselves; with a machine-readable interface definition, and human-readable documentation.

Remotability

It should be easy to forward, proxy, redirect varlink interfaces over any connection-oriented transport. Varlink should be free of any side-effects of local APIs. All interactions need to be simple messages on a network, not carrying things like file descriptors or references to locally stored files.

Focus

Varlink is the protocol and definition of interfaces, but it does not define or provide any significant functionality itself.

Errors

Varlink errors should carry enough information to be consumed by machines and automated rules.

Testing

Making it easy to run automated tests against varlink interfaces should be considered a necessity when designing interfaces.

Right now the API to a Linux system consists of:

All these can be replaced by varlink interfaces (yes, even the kernel interfaces). Of course varlink is the 15th xkcd standard here :-P If the adoption of varlink takes off, then the collection of interfaces could form a common Linux System API.

These interface definitions can be inspected even at runtime via a common interface named org.varlink.service, which every service provides. With this mechanism it is very easy for interpreter languanges to create bindings at runtime, as we will see in the next chapter.

Ready for some example on how to use varlink with python? First we need some packages to install on Fedora: 

$ sudo dnf copr enable "@varlink/varlink"
$ sudo dnf install fedora-varlink
$ sudo setenforce 0 # needed until systemd is able to create sockets in /run
$ sudo systemctl enable --now org.varlink.resolver.socket
$ varlink help

In this example, we will call an interface called com.redhat.system.accounts which has the following varlink interface: 

interface com.redhat.accounts

type Account (
    name: string,
    uid: int,
    gid: int,
    full_name: string,
    home: string,
    shell: string
)

# Retrieve a list of account information for all known accounts
method GetAll() -> (accounts: []Account)
# Retrieve the account information for a specific user ID
method GetByUid(uid: int) -> (account: Account)
# Retrieve the account information
method GetByName(name: string) -> (account: Account)
# Add new account
method Add(account: Account) -> (account: Account)

error NotFound ()
error CreationFailed (field: string)

From reading the interface definition, the API should be understandable, even for people, who have never seen a varlink interface file.

Now, let’s call the service from some python code:

from varlink import Client
client = Client(resolve_interface='com.redhat.accounts')
accounts = client.open('com.redhat.accounts', namespaced=True)
ret = accounts.GetByName("root")
print(ret)

will output:

namespace(account=namespace(full_name='root', gid=0, home='/root', name='root', shell='/bin/bash', uid=0))

and

>> print(ret.account.full_name, ret.account.uid, ret.account.shell)

will output:

root 0 /bin/bash

That’s it! And the varlink python module is also very small. Stay tuned for more and go read Jens Reinmann’s Java Varlink blog post.