Dozens of services, developed by many people
Each service has its own way of communicating with the rest of the world
Meaning that a service that needs to interact (yep, that's all of them) will need to implement a bunch of communication systems
dak, our archive software, uses emails and databases to communicate. Metadata is available in a RFC822 format, no real API (the DB is not public either)
wanna-build, our build queue management software, polls a database every few minutes to know what needs to get built. No API outside of its database
debbugs, our bug tracking system, works via email, stores its data in flat files, and exposes a read-only SOAP API
SCM pushes in the distro-provided repos (on alioth) can trigger an IRC bot (KGB), some mails. No central notification mechanism
In the packaging pipeline: transition from service to service:
…
Not maintainable
Adding new services is annoying (you need to touch everything)
Contains a snapshot of a lot of the databases undelying the Debian infrastructure at a given time (even some Ubuntu bits)
A lot of cron-triggered importers
Useful for distro-wide QA purposes, not so much for real-time updates
http://packages.qa.debian.org/ (and http://pts.debian.net)
Cron-triggered updates on source packages
Subscribe for email updates on a given packages
Messages not uniform, contain some headers for filtering and machine parsing
Not real-time
Jesse Keating's talk in 2009 at NorthWest Linux Fest
A unified message bus, reducing the coupling of interdependent services
Services subscribe to one (or several) message topic(s), register callbacks, and respond to events
First candidate: AMQP
Issues:
Actual message bus: 0mq
No broker, each service binds to a port and starts publishing messages
Other services connect to those ports and start consuming messages
Advantages
Main issue with a brokerless system: service discovery
Three options
Fedora uses option #3, and the Debian GSoC student implemented option #2
Event topics follow the rule:
org.distribution.ENV.SERVICE.OBJECT[.SUBOBJECT].EVENT
Where:
From python:
import fedmsg fedmsg.publish(topic='testing', modname='test', msg={ 'test': "Hello World", })
From the shell:
$ echo "Hello World." | fedmsg-logger --modname=git --topic=repo.update $ echo '{"a": 1}' | fedmsg-logger --json-input $ fedmsg-logger --message="This is a message." $ fedmsg-logger --message='{"a": 1}' --json-input
From python:
import fedmsg # Read in the config from /etc/fedmsg.d/ config = fedmsg.config.load_config([], None) for name, endpoint, topic, msg in fedmsg.tail_messages(**config): print topic, msg # or use fedmsg.encoding.pretty_dumps(msg)
In the shell, you can use the fedmsg-tail command (demo time)
More than 7000 users registered!!
Distro agnostic