The default SELinux policy for *collectd* does not allow it all the
necessary access for the way we use it. Notably, it cannot bind to the
HTTP port to export Prometheus metrics, and it is not allowed to use
netlink to read interface statistics. The latter is not a huge deal, as
it can fall back to the legacy procfs interface, but the former is a
nonstarter.
Eventually, I should write an SELinux module with the correct
permissions (and submit the changes upstream), but for now, we'll just
make the `collectd_t` domain permissive.
Unfortunately, running *collectd* in a container is not going to work.
Although containers can be configured to share some of the host's
namespaces, one notable exception is the mount namespace. Naturally,
containers must have their own mount namespace, which prevents them from
seeing filesystems that are actually mounted on the host. For
*collectd*, this effectively makes the `df` plugin useless, which
ultimately prevents us from monitoring disk space.
This reverts commit 4048e5cc0a.
Installing packages with `rpm-ostree` is somewhat problematic. Notably,
if a new package needs an update of an already-installed package (e.g.
shared library), the new package cannot be installed until a new version
of CoreOS is published with the updated dependency.
In order for collectd to be effective, the container it runs in has to
have most isolation features disabled. Most importantly, the PID, UTS,
and network namespaces need to be shared with the host, so that
*collectd* can "see" the actual values. Additionally, the default
SELinux policy for containerized processes denies practically all of the
instrumentation syscalls *collectd* needs, so it needs to run in the
unconfined `spc_t` domain. Finally, the `/run` directory needs to be
shared with the host, so *collectd* can communicate with various daemons
via UNIX sockets.
The `collectd.yaml` Butane configuration fragment configures the machine
to install *collectd* and its various plugin packages directly on the
host using `rpm-ostree` (via *install-packages.service*).
Dustin