MinIO/S3 clients generate a _lot_ of requests. It's also not
particularly useful to have these stored in Loki anyway. As such, we'll
stop routing them to syslog/journal.
Having access logs is somewhat useful for troubleshooting, but really
for only live requests (i.e. what's happening right now). We therefore
keep the access logs around in a file, but only for one day, so as not
to fill up the filesystem with logs we'll never see.
There may be cases where we want either error logs or access logs to be
sent to syslog, but not both. To support these, there are now two
variables: `nginx_access_log_syslog` and `nginx_error_log_syslog`.
Both use the value of the `nginx_log_syslog` variable by default, so
existing users of the _nginx_ role will continue to work as before.
If _nginx_ is configured to send error/access log messages to syslog, it
may not make sense to _also_ send messages to log files as well. The
`nginx_error_log_file` and `nginx_access_log_file` variables are now
available to control whether/where to send log messages. Setting either
of these to a falsy value will disable logging to a file. A non-empty
string value is interpreted as the path to a log file. By default, the
existing behavior of logging to `/var/log/nginx/error.log` and
`/var/log/nginx/access.log` is preserved.
_wal-g_ can send StatsD metrics when it completes an upload/backup/etc.
task. Using the `statsd_exporter`, we can capture these metrics and
make them available to Victoria Metrics.
The *statsd exporter* is a Prometheus exporter that converts statistics
from StatsD format into Prometheus metrics. It is generally useful as a
bridge between processes that emit event-based statistics, turning them
into Prometheus counters and gauges.
The [Memories] app for Nextcloud provides a better user interface and
more features than the built-in Photos app. The latter seems to be
somewhat broken recently (timeline stops in June 2024, even though there
are more recent photos available), so we're trying out Memories (and
Recognize for facial recognition).
[Memories]: https://memories.gallery
Nextcloud 28+ uses JavaScript modules (`.mjs` files). These need to be
served from the filesystem like other static files, so the *mod_rewrite*
configuration needs to be updated as such.
When a VM uses a serial port for its default console, kernel messages
(e.g. panics) are lost if no console client is connected at the time.
This is a major disadvantage when compared to a graphical console, which
usually at least keeps a "screenshot" of the console when the kernel
crashes.
While researching the available console device types to determine how
best to implement a tool that would both log the output from the serial
console at all times, while still allowing interactive connections to
it, I discovered that _libvirt_ actually already has this exact
functionality built-in:
https://libvirt.org/formatdomain.html#consoles-serial-parallel-channel-devices
Nextcloud writes JSON-structured logs to
`/var/lib/nextcloud/data/nextcloud.log`. These logs contain errors,
etc. from the Nextcloud server, which are useful for troubleshooting.
Having them in Loki will allow us to view them in Grafan as well as
generate alerts for certain events.
_nginx_ access logs are typically either very small or very large. For
small log files, it's fast enough to decompress them on the fly if
necessary. For large files, they may take up so much space in
uncompressed form that the log volume fills too quickly. In either
case, compressing the files as soon as they are rotated is a good
option, especially since their contents should already be sent to Loki.
_WAL-G_ and _restic_ both generate a lot of HTTP traffic, which fills up
the log volume pretty quickly. Let's reduce the number of days logs are
kept on the file system. Logs are shipped to Loki anyway, so there's
not much need to have them local very long.
The default `logrotate` configuration for _nginx_ may not be appropriate
for high-volume servers. The `nginx_keep_num_logs` variable is now
available to control how many days of logs are kept.
Invoice Ninja needs to be accessible from the Internet in order to
receive webhooks from Stripe. Additionally, Apple Pay requires
contacting Invoice Ninja for domain verification.
Gitea and Vaultwarden both have SQLite databases. We'll need to add
some logic to ensure these are in a consistent state before beginning
the backup. Fortunately, neither of them are very busy databases, so
the likelihood of an issue is pretty low. It's definitely more
important to get backups going again sooner, and we can deal with that
later.
Since `restic` needs to run as root in order to back up files regardless
of their permissions, we need to restrict it to doing only that. Using
systemd sandbox features, especially the capability bounding set, we can
remove all of _root_'s powers except the ability to read all files.
The `restic.yml` playbook applies the _restic_ role to hosts in the
_restic_ group. The _restic_ role installs `restic` and creates a
systemd timer and service unit to run `restic backup` every day.
Restic doesn't really have a configuration file; all its settings are
controlled either by environment variables or command-line options. Some
options, such as the list of files to include in or exclude from
backups, take paths to files containing the values. We can make use of
these to provide some configurability via Ansible variables. The
`restic_env` variable is a map of environment variables and values to
set for `restic`. The `restic_include` and `restic_exclude` variables
are lists of paths/patterns to include and exclude, respectively.
Finally, the `restic_password` variable contains the password to decrypt
the repository contents. The password is written to a file and exposed
to the _restic-backup.service_ unit using [systemd credentials][0].
When using S3 or a compatible service for respository storage, Restic of
course needs authentication credentials. These can be set using the
`restic_aws_credentials` variable. If this variable is defined, it
should be a map containing the`aws_access_key_id` and
`aws_secret_access_key` keys, which will be written to an AWS shared
credentials file. This file is then exposed to the
_restic-backup.service_ unit using [systemd credentials][0].
[0]: https://systemd.io/CREDENTIALS/
Since LAN clients have IPv6 addresses now, some may try to connect to
the database over IPv6, so we need to allow this in the host-based
authentication rules.
It turns out, having the exporter connect to the _template1_ database is
not a great idea. PostgreSQL does not allow creating a new database if
the template database is currently being accessed by any clients. Since
_template1_ is the default choice, the `createdb` command will probably
fail.
It doesn't specifically matter which database the exporter connects to,
since it reads most (all?) of its data from the PostgreSQL catalog,
which isn't database-specific.
Moving the Nextcloud database to the central PostgreSQL server will
allow it to take advantage of the monitoring and backups in place there.
For backups specifically, this will make it easier to switch from BURP
to Restic, since now only the contents of the filesystem need backed up.
The PostgreSQL server on _db0_ requires certificate authentication for
all clients. The certificate for Nextcloud is stored in a Secret in
Kubernetes, so we need to use the _nextcloud-db-cert_ role to install
the script to fetch it. Nextcloud configuration doesn't expose the
parameters for selecting the certificate and private key files, but
fortunately, they can be encoded in the value provided to the `host`
parameter, though it makes for a rather cumbersome value.
Currently, the certificate authority that issues certificates for
PostgreSQL clients is hosted in Kubernetes and managed by
_cert-manager_. Certificates it issues are stored in Kubernetes Secret
resources, making them easy to consume by applications running in the
cluster, but not for anything outside. Since Nextcloud runs on its own
VM, we need a way to get the certificate out of the Secret and into a
file on that machine. To that end, I've written the
`nextcloud-fetch-cert.py` script. This script uses a Kubernetes Service
Account token to authenticate to the Kubernetes API and download the
contents of the Secret. It runs periodically, triggered by a systemd
timer unit, to ensure the certificate is always up-to-date.
The obvious drawback to this approach is the requirement for a static
token. Since there's not really a way to "renew" Service Account
tokens, it needs to be issued with a fairly long duration, to mitigate
the risk of being unable to fetch a new certificate once it has expired
because the token has also expired. This somewhat negates the advantage
of using certificates for authentication, since now the machine needs a
static, pre-defined secret.
At some point, I may deploy another instance of _step-ca_ to manage the
PostgreSQL client CA. Clients can then use e.g. `certbot` or `step ca
certificate` to obtain their certificates. I chose not to implement
this yet, though for a couple of reasons. First, I need to move the
Nextcloud database very soon, so we switch to using `restic` for backups
without having to deal with the database. Second, I am still
considering moving Nextcloud into Kubernetes eventually, where it will
be able to get the Secret directly; since Nextcloud is the only client
outside the cluster, it may not be worth setting up _step-ca_ in that
case.
The _nextcloud_ role originally handled setting up the PostgreSQL
database and assumed that it was running on the same server as Nextcloud
itself. I have factored out those tasks into their own role,
_nextcloud-db_, which can be applied to a separate host.
I have also introduced some new variables (`nextcloud_db_host`,
`nextcloud_db_name`, `nextcloud_db_user`, and `nextcloud_db_password`),
which can be used to specify how to connect to the database, if it is
hosted remotely. Since these variables are used by both the _nextcloud_
and _nextcloud-db_ roles, they are actually defined in a separate role,
_nextcloud-base_, upon which both depend.
When HAProxy binds to the IPv6 socket, it can handle both IPv6 and IPv4
clients. IPv4 clients are handled as IPv4-mapped IPv6 addresses, which
some backends (i.e. Apache) cannot support. To avoid this, we configure
HAProxy to bind to the IPv4 and IPv6 sockets separately, so that IPv4
addresses are handled as IPv4 addresses.
Expose a virtual host on a separate TCP port that uses the PROXY
protocol. This way, HAProxy can pass the original client IP address to
Jellyfin without terminating the TLS connection.
In order to enable authentication using LDAP over TLS in Jellyfin, we
need to expose the CA certificate that issues the LDAP server
certificates to the container.
*chromie.pyrocufflink.blue* will replace *burp1.pyrocufflink.blue* as
the backup server. It is running on the hardware that was originally
*nvr1.pyrocufflink.blue*: a 1U Jetway server with an Intel Celeron N3160
CPU and 4 GB of RAM.
The `raid-array.yml` playbook can create Linux *md* software RAID arrays
using the `mdadm` command. Two variables are required: `md_name` and
`raid_disks`. The former is a string name for the array. The latter is
an array of paths of block devices to add to the array.
This playbook uses the *minio-nginx* and *minio-backups-cert* role to
deploy MinIO with nginx.
The S3 API server is *s3.backups.pyrocufflink.blue*, and buckets can be
accessed as subdomains of this name.
The Admin Console is *minio.backups.pyrocufflink.blue*.
Certificates are issued by DCH CA via ACME using `certbot`.
The MinIO server for backups has special requirements for HTTPS. I want
to use subdomains for bucket names, so the certificate must have a
wildcard name, which requires using the DNS-01 challenge. Fortunately,
it is actually pretty easy to use `nsupdate` with GSS-TSIG
authentication to automate DNS record creation, and by default, all
domain-member machines can create any records. Thus, using the `manual`
auth plugin for `certbot` and a script to run `nsupdate`, obtaining the
wildcard certificate is fairly straightforward.
The biggest issue I encountered while developing this feature was
caching of NXDOMAIN responses. There doesn't seem to be a way to change
the TTL of the SOA record of the Active Directory DNS domain, which
defaults to 3600, meaning NXDOMAIN responses are always cached for an
hour. When adding a record using `nsupdate -g`, the tool always
performs a SOA lookup of new name to find the target zone for it. Since
the name does not exist yet, the domain controller responds with
NXDOMAIN, which gets cached by the main DNS server. Thus, even after
adding the record, the ACME server will not be able to resolve the
name for up to an hour. We can a void this by explicitly setting the
target zone. That would not work in a multi-domain forest, but
fortunately, we do not have to worry about that.
This role borrows some logic from the *postgresql-cert* role.
Eventually, I probably want to combine some of the steps from both of
these roles, possibly replacing the old *certbot* role.
The *minio-nginx* role configures nginx to proxy for MinIO. It uses the
"subdomain" pattern, as described in [Configure NGINX Proxy for MinIO
Server][0]; the S3 API and the console UI are accessible through
different domain names.
[0]: https://min.io/docs/minio/linux/integrations/setup-nginx-proxy-with-minio.html
Modern versions of Podman use Netavark, which needs to write various
files on the host file system (even when the container uses the
host's network namespace).
If the `minio_address` variable is specified, it will be passed with the
`--address` argument to `minio server`. This allows controlling the
socket the server binds to and listens on.
The `minio_browser_redirect_url` can be specified to populate the
similarly-named environment variable, which configures how MinIO serves
the web UI.
The `minio_domain` variable sets the `MINIO_DOMAIN` environment
variable, which enables DNS names (subdomains) for buckets, i.e.
`{bucket_name}.{MINIO_DOMAIN}`.
Dustin