Dozens of services, developed by many people
Each service has its own way of communicating with the rest of the world
Meaning that a service that needs to interact (yep, that's all of them) will need to implement a bunch of communication systems
dak, our archive software, uses emails and databases to communicate. Metadata is available in a RFC822 format, no real API (the DB is not public either)
wanna-build, our build queue management software, polls a database every few minutes to know what needs to get built. No API outside of its database
debbugs, our bug tracking system, works via email, stores its data in flat files (for now), and exposes a read-only SOAP API
SCM pushes in the distro-provided repos (on alioth) can trigger an IRC bot (KGB), some mails. No central notification mechanism
Contains a snapshot of a lot of the databases undelying the Debian infrastructure at a given time (even some Ubuntu bits)
A lot of cron-triggered importers
Useful for distro-wide QA purposes, not so much for real-time updates. Furthermore, consistency is not guaranteed
https://packages.qa.debian.org/ (and https://tracker.debian.org)
Cron-triggered updates on source packages
Subscribe for email updates on a given packages
Messages not uniform, contain some headers for filtering and machine parsing
Not real-time
Jesse Keating's talk in 2009 at NorthWest Linux Fest
A unified message bus, reducing the coupling of interdependent services
Services subscribe to one (or several) message topic(s), register callbacks, and respond to events
First candidate: AMQP as implemented by qpid
Issues:
Actual message bus: 0mq
No broker, each service binds to a port and starts publishing messages
Other services connect to those ports and start consuming messages
Advantages
Main issue with a brokerless system: service discovery
Three options
Simon Chopin implemented the DNS solution during his GSoC.
Event topics follow the rule:
org.distribution.ENV.SERVICE.OBJECT[.SUBOBJECT].EVENT
Where:
From python:
import fedmsg fedmsg.publish(topic='testing', modname='test', msg={ 'test': "Hello World", })
From the shell:
$ echo "Hello World." | fedmsg-logger --modname=git --topic=repo.update $ echo '{"a": 1}' | fedmsg-logger --json-input $ fedmsg-logger --message="This is a message." $ fedmsg-logger --message='{"a": 1}' --json-input
From python:
import fedmsg config = fedmsg.config.load_config([], None) for name, endpoint, topic, msg in fedmsg.tail_messages(**config): print fedmsg.encoding.pretty_dumps(msg)
In the shell, you can use the fedmsg-tail command
A lot happened during GSoC 2013, thanks to Simon Chopin's involvement:
Since July 14th 2013:
Datanommer (database component) has been packaged and is being deployed.
Debian services are widely distributed, firewall restrictions are sometimes our of our control: we need a way for services to push their messages instead of having the fedmsg-gateway pull them.
fedmsg-relay: non-standard port, non-standard protocol.
Currently pondering a way for services to push their messages through https.
olasd@debian.org
If there's interest, I'd be glad to hold an ad-hoc hacking session!