In order to manage servers that are not members of the
_pyrocufflink.blue_ AD domain, Jenkins needs a user certificate signed
by the SSH CA. Unfortunately, there is not really a good way to get a
certificate issued on demand in a non-interactive way, as SSHCA relies
on OIDC ID tokens which are issued by Authelia, and Authelica requires
browser-based interactive login and consent. Until I can come up with a
better option, I've manually signed a certificate for Jenkins to use.
The Jenkins SSH Credentials plugin does not support certificates
directly, so in order to use one, we have to explicitly configure `ssh`
to load it via the `CertificateFile` option.
This doesn't actually work, because the shell expands `~` to the value
of the `HOME` environment variable, but `ssh` ignores that variable and
reads from the path from the user's `passwd` entry. Since managed hosts
all have certificates now, and the CA key is included in the global host
key database, individual host keys are not needed anymore anyway.
Now that servers are configured to use *pam_ssh_agent_auth* for `sudo`
authentication, the Jenkins CI pipelines no longer need to manage the
raw password for the *jenkins* user. A special SSH public key for
Jenkins is listed in `/etc/security/sudo.authorized_keys`, so as long
as a) the corresponding private key is in the SSH agent and b) SSH
agent forwarding is enabled, Ansible will be able to perform privileged
operations without a password.
The latest version of the *ansible* container runs processes as the
unprivileged *jenkins* user, provides its own "sleep forever" default
command, and sets the correct LANG environment variable. Since it runs
processes as *jenkins*, we need to override HOME and set it to the
WORKSPACE to ensure Jenkins has a writable path for arbitrary files.
Hopefully this will fix this warning from Ansible:
> [WARNING]: An error occurred while calling
> ansible.utils.display.initialize_locale (unsupported locale setting).
> This may result in incorrectly calculated text widths that can cause
> Display to print incorrect line lengths
I don't know why I didn't think of this before! There's no reason to
have to have already copied the `ssh_known_hosts` file from to
`/etc/ssh` before running `ansible-playbook`. In fact, keys just end up
getting copied from `/etc/ssh/ssh_known_hosts` into `~/.ssh/known_hosts`
anyway. So let's just make it so that step isn't necessary: copy the
host key database directly to `~/.ssh` and avoid the trouble.
We'll use the `podTemplate` block to define an ephemeral agent running in
a Kubernetes pod as the node for this pipeline. This takes the place of
the Docker container we used previously.
In order for Jenkins to apply configuration policy on machines that are
not members of the *pyrocufflink.blue* domain, it needs to use an SSH
private key for authentication.
Since the `remount.yml` playbook now tries to avoid redundant remounts,
it needs up-to-date facts about mounted filesystems and their options.
If the cached facts are out of date, it may incorrectly skip remounting
a filesystem. Besides, having up-to-date facts in the CI pipeline is
probably a good thing anyway.
Some playbooks apply only to hosts that do not have read-only root
filesystems. For these, the `rw_limit` pattern will be empty. The
*Remount R/W* and *Remount R/O* stages should be skipped when this is
the case.
Some hosts, such as the Raspberry Pis built using default Fedora images,
do not have proper filesystem separation, but use a single volume for
the entire filesystem. These hosts cannot have the root filesystem
mounted read-only, since all the writable data are also stored there.
When Jenkins runs configuration policy jobs, it always tries to remount
the root filesystem as read-only on every machine that it configured.
For these hosts with a single volume, this step fails, causing the job
to be marked as failed. To avoid this, I have added a new group,
*rw-root*; hosts in this group will be omitted from the final remount
step.
Software should never be installed or updated by the continuous
enforcement jobs. This can cause unexpected outages or other problems
if applications or libraries unexpectedly. Everything should already be
installed and in production before continuous enforcement begins, so
skipping install steps should not matter.
Most tasks that install software are tagged with the `install` tag.
When Jenkins runs `ansible-playbook` to apply configuration policy, it
will now skip any task that includes this tag.
The Jenkins pipeline definition files are highly redundant. Each one
implements almost the same stages, with only a few variations. Whenever
a new pipeline is added, it's copied from the most recent file and
modified. If any improvements are made to it, they do not usually get
implemented in any of the existing pipelines.
To address this, the `applyConfigPolicy` pipeline library function is
now available. This function generates the full pipeline for a
particular application, including stages for setup, each individual
playbook, and cleanup. Using this function, pipeline files can be as
simple as:
@Library('cfgpol')_
applyConfigPolicy(
'gitea',
[
'Gitea': [
'gitea.yml',
],
]
)
This will create a pipeline that mounts the root filesystem read-write
on all hosts in the "gitea" group (any Ansible host pattern is allowed),
applies the `gitea.yml` playbook (in a stage named "Gitea"), and then
remounts the filesystems read-only.
Since this "library" is so simple, containing only a single function in
a single file, and since it will not be used by any pipelines outside
this repository, it makes sense to keep it in this repository, instead
of a separate repository as is customary for Jenkins pipeline shared
libraries.
Dustin