BIND response policy zones (RPZ) support provides a mechanism for
overriding the responses to DNS queries based on a wide range of
criteria. In the simplest form, a response policy zone can be used to
provide different responses to different clients, or "block" some DNS
names.
For the Pyrocufflink and related networks, I plan to use an RPZ to
implement ad/tracker blocking. The goal will be to generate an RPZ
definition from a collection of host lists (e.g. those used by uBlock
Origin) periodically.
This commit introduces basic support for RPZ configuration in the
*named* role. It can be activated by providing a list of "response
policy" definitions (e.g. `zone "name"`) in the `named_response_policy`
variable, and defining the corresponding zones in `named_zones`.
Normally, Home Assistant uses a SQLite database for storing state
history. On a Raspberry Pi with only an SD card for storage like
*hass1.pyrocufflink.blue*, this can become extremely slow, especially
for large data sets. To speed up features like history and logbook,
Home Assistant supports using an external database engine such as
PostgreSQL or MariaDB.
The *hassdb* role and corresponding `hassdb.yml` playbook deploys a
PostgreSQL server for Home Assistant to use. It needs only to create
the role and database, as Home Assistant manages its own schema.
The *postgresql-setup* service is no longer necessary, as upstream has
fixed the SELinux policy to allow root to invoke the `postgresql-setup`
command directly.
This commit adds a task to generate a PostgreSQL configuration file from
a template. Previously, the default configuration file generated by
`initdb` was sufficient, but in order to enable SSL connections, some
changes to it are required.
Naturally, SSL connections require a server certificate, so the
*postgresql-server* role will now also copy certificate files to the
managed node, if any.
Fedora has renamed the *strongswan* service to *strongswan-starter*.
The *strongswan* service now controls strongSwan via Vici, which uses a
different configuration format and is not compatible with the files in
`/etc/strongswan/ipsec.d`. As I am migrating everything to Wireguard
now, it does not make sense to rewrite all of the IPsec configuration in
this new format, so using the legacy format with the renamed service
makes more sense.
Because the Home Assistant user's home directory is on `/var`, Python
packages installed in the "user site" do not get the correct SELinux
labels and thus run in the wrong domain. This causes a lot of AVC
denials and other issues that prevent Home Assistant from working
correctly.
To resolve this issue, Home Assistant is now installed in a virtual
environment at `/usr/local/homeassistant`. This directory is still
owned by the Home Assistant user, allowing Home Assistant to manage
packages installed there. Since it is rooted under `/usr`, files are
labelled correctly and processes launched from executables there will
run in the correct domain.
The main network, *pyrocufflink.blue* (172.30.0.0/26) is now on VLAN 1
instead of VLAN 30. This changed when I replaced the Cisco SG200-26
with the UniFI Switch 48, to simplify configuration of all of the
Ubiquiti devices.
For some time, I have been trying to design a new configuration for the
reverse proxy on port 443 to correctly handle all the types of traffic
on that port. In the original implementation, all traffic on port 443
was forwarded by the gateway to HAProxy. HAproxy then used TLS SNI to
route connections to the correct backend server based the requested host
name. This allowed both HTTPS and OpenVPN-over-TLS to use the same
port, however it was not without issues. A layer 4 (TCP) proxy like
this "hides" the real source address of clients connecting to the
backend, which makes IP-based security (e.g. rate limiting, blacklists,
etc.) impossible at the application level. In particular, Nextcloud,
which implements rate limiting was constantly imposing login delays on
all users, because legitimate traffic was indistinguishable from
Internet background noise.
To alleviate these issues, I needed to change the proxy to operate in
layer 7 (HTTP) mode, so that headers like *X-Forwarded-For* and
*X-Forwarded-Host* could be added. Unfortunately, this was not easy,
because of the simultaneous requirement to forward OpenVPN traffic.
HAProxy can only do SNI inspection in TCP mode. So, I began looking for
an alternate way to proxy both HTTP and non-HTTP traffic on the same
port.
The HTTP protocol defines the `CONNECT` method, which is used by forward
proxies to tunnel HTTPS over plain HTTP. OpenVPN clients support
tunneling OpenVPN over HTTP using this method as well. HAProxy has
limited support for the CONNECT method (i.e. it doesn't do DNS
resolution, and I could find no way of restricting the destination) with
the `http_proxy` option, so I looked for alternate proxy servers that
had more complete support. Unsurprisingly, Apache HTTPD has the most
complete implementation of the `CONNECT` method (Nginx doesn't support
it at all). Using a name-based virtual host on port 443, Apache will
accept requests for *vpn.pyrocufflink.net* (using TLS SNI) and allow the
clients to use the `CONNECT` method to create a tunnel to the OpenVPN
server. This requires OpenVPN clients to a) use *stunnel* to wrap plain
HTTP proxy connections in TLS and b) configure OpenVPN to use the
TLS-wrapped HTTP proxy.
With Apache accepting all incoming connections, it was trivial to also
configure it as a layer 7 forward proxy for Bitwarden, Gitea, Jenkins,
and Nextcloud. Unfortunately, proxying for the other websites
(darkchestofwonders.us, chmod777.sh, dustin.hatch.name) was not quite as
straightforward. These websites would need to have an internal name
that differed from their external name, and thus a certificate valid for
that name. Rather than reconfigure all of these sites and set all of
that up, I decided to just move the responsibility for handling direct
connections from outside to the *web0* and eliminate the dedicated
reverse proxy. This was not possible before, because Apache could not
forward the OpenVPN traffic directly, but now with the forward proxy
configuration, there is no reason to have a separate server for these
connections.
Overall, I am pleased with how this turned out. It makes the OpenVPN
configuration simpler (*stunnel* no longer needs to run on the OpenVPN
server itself, since Apache is handling TLS termination), eliminates a
network hop for the websites, makes the reverse proxy configuration for
the other web applications much easier to understand, and resolves the
original issue of losing client connection information.
A name-based HTTP (not HTTPS) virtual host for *pyrocufflink.net* is
necessary to ensure requests are handled properly, now that there is
another HTTP virtual host (chmod777.sh) defined on the same server.
This commit updates the configuration for *pyrocufflink.net* to use the
wildcard certificate managed by *lego* instead of an unique certificate
managed by *certbot*.
*chmod777.sh* is a simple static website, generated by Hugo. It is
built and published from a Jenkins pipeline, which runs automatically
when new commits are pushed to Gitea.
The HTTPS certificate for this site is signed by Let's Encrypt and
managed by `lego` in the `certs` submodule.
This commit adds front-end and back-end configuration for HAProxy to
proxy HTTP/HTTPS for
*nextcloud.pyrocufflink.net*/*nextcloud.pyrocufflink.blue* to
*cloud0.pyrocufflink.blue*.
The *nextcloud* role installs Nextcloud from the specified release
archive, downloading it to the control machine first if necessary, and
configures Apache and PHP-FPM to serve it.
The `nextcloud.yml` playbook uses the *cert* role to install the X.509
certificate for the Nextcloud server, sets up Apache HTTPD with the
*apache* role, and installs Nextcloud using the *nextcloud* role.
The host *cloud0.pyrocufflink.blue* is the Nextcloud server for
Pyrocufflink.
The *cert* role is intended to be a generic, reusable role to copy an
X.509 certificate and/or private key file to managed nodes. It is
intended to be included in a playbook with at least the `cert_src` and
`cert_dest` variables defined, e.g.:
```
- hosts: whatever
roles:
- role: cert
cert_src: whatever.cer
cert_dest: /path/to/whatever.cer
```
the `haproxy_ssl_default_bind_options` variable is not defined for
machines running Fedora, because this parameter is not used in the
default configuration file there.
I seem to have forgotten how I got the RPM for Gitea. I think I built
it, but I cannot find the spec file, nor the RPM package. Since this is
clearly not reproducible, I decided to switch to using the binary
provided by upstream for now, until either I or Fedora get around to
making a better RPM.
Installing Gitea from the upstream binary is simple: just download it
and copy it to `/usr/local/bin`. Of course, the OS user and systemd
unit have to be managed by configuration policy when it's installed this
way.
*burp1.pyrocufflink.blue* will replace *burp0.pyrocufflink.blue* as the
BURP server for Pyrocufflink. It is a physical machine (Fitlet), making
it simpler to manage the USB drives. The old virtual machine will be
decommissioned soon.
Ansible replaced the `version_compare` filter with a `version_compare`
test that does the same thing. The former is completely gone now,
causing the template to fail to render, so its usage of that filter
needs to be updated.
Using the generic *burp.pyrocufflink.blue* name will allow easier
transition to a new BURP server. However, since this is not the actual
name, it cannot be used for task delegation, so a separate variable is
required to store the real name of the BURP server. This is only used
during client deployment, and not by BURP itself.
The *graylog* role installs Graylog from the *graylog2.org* Yum
repository and manages basic server configuration. It augments the
default systemd unit to provide the `CAP_NET_BIND_SERVICE` capability to
the Graylog server process via ambient capabilities, thereby allowing
the server to bind to the privileged Syslog UDP port.
The `Alias` configuration for Certbot needs to be configured before any
other locations, to ensure the `/.well-known` path is always served from
the local filesystem. If another drop-in configuration file (e.g.
`bitwarden.conf`) is ordered before it, it may override this
configuration and prevent Let's Encrypt from working.
In order to allow Jenkins to connect to the Docker daemon socket, the
socket must be owned by the *docker* group, and the *jenkins* user must
be a member of it.
This commit adds an HAProxy backend for Bitwarden, and adds ACL rules to
the frontend to proxy traffic to *bitwarden.pyrocufflink.blue* or
*bitwarden.pyrocufflink.net* to it.
Since the same certificate is used for LDAPS and RADIUS (EAP-TLS), it
makes more sense to store it only once, with the later file as a symlink
to the former.
This commit configures *bw0.pyrocufflink.blue* as a BURP client, so that
the Bitwarden data can be backed up. A pre-backup script is used to
take a consistent snapshot of the SQLite database before copying it to
the BURP server.
The BURP server runs as user *burp*, and nas such, requires that the
client-specific configuration files be owned by that user so they can be
read when a client connects.
Newer versions of the BURP client require `status_port` to be set. This
commit updates the `burp.conf.j2` template to more closely match the
default configuration shipped with the *burp* package, including setting
this new value.
Newer versions of Gitea need a JWT secret for Oauth2. Gitea will
attempt to generate one at startup if it is not already specified in the
configuration file, but this will fail since the file is not writable by
the user running the service. As such, it must be set via configuration
policy.
The point of the "wheel host" is to serve as a repository of Python
packages (wheels) built by Jenkins for consumption by `pip` et al. For
applications and libraries that do not provide all of their dependencies
as binary packages, this makes a convenient way to install them without
requiring all of the build tools and dependencies on the destination
machine.
The idea here is that a Jenkins job runs `pip wheel` for a distribution
package name or `requirements.txt` file and then uploads the resulting
wheel files using `rsync`. Apache is configured to serve the upload
directory with an index compatible with `pip`'s `--find-links`.
The *hass-dhcp* role installs dnsmasq and configures it to serve DHCP
requests on the Home Assistant network. Since this network is not
routed, the regular DHCP relay/server setup will not work.
This commit adds a systemd unit to enable the Kernel Same-page Merging
daemon on VM hosts. This allows much greater virtual machine density,
especially when many VMs are running the same guest OS.
Debian does not support system-wide SSL cipher suite profiles of course,
so these options need to be specified explicitly when deploying HAProxy
on Debian-based machines.
This commit updates the net-ifaces scripts for both *vmhost0* and
*vmhost1* to create VLAN and bridge interfaces for the Management and
Home Assistant networks.
The *taiga* role installs the three components of Taiga:
* taiga-back
* taiga-events
* taiga-front
*taiga-back* is a Python application. Its dependencies are installed via
`pip` in the *taiga* user's site-packages, and the application itself is
installed by unpacking the archive. *taiga-events* is a Node.js
application. Its dependencies are installed by `npm`, and is itself
installed by unpacking the archive. Finally, *taiga-front* is a
single-page browser application that is installed by unpacking the
archive, and served by Apache.
Taiga requires PostgreSQL and RabbitMQ.
The *websites/pyrocufflink.net* role configures the public web server to
host *pyrocufflink.net*. This site has two functions:
* It redirects `/` to http://dustin.hatch.name/
* It proxies user home directories (i.e. /~dustin/) to the file server
The `ServerName` directive needs to be set inside the default SSL vhost,
as this property does not get inherited from the global configuration,
and it is needs to be set in order for SNI to work correctly.
By default, per-user directories (i.e. `/~username/`) are disabled in
Fedora's configuration of Apache. This commit introduces a new variable,
`apache_userdir`, which can be used to enable this feature. It should be
set to a string other than *disabled*, which is the path under users'
home directories that will be served, if it is accessible. Normally, the
value would be `public_html`.
To support multiple websites with separate Let's Encrypt certificates,
the *certbot* role needs to be applied as a dependency of each
individual website role. This will allow each application to specify a
different value for `certbot_domains`.
The default systemd unit configuration for *certbot-renew.service* runs
the `certbot renew …` command as root. This can cause permissions
issues, since this Ansible role expects the *certbot* user to be able to
access all configuration, data, and log files. As such, this commit adds
a systemd unit extension for *certbot-renew.service* to run the command
as *certbot*.
Moving the route definitions to global scope, and defining an address
pool, will allow other clients besides *dhatch-d4b* to connect to and
use the OpenVPN tunnel service. This may be useful in situations where
IPsec is blocked
The VPN capability of the UniFi Security Gateway is extremely limited.
It does not support road-warrior IPsec/IKEv2 configuration, and its
OpenVPN configuration is inflexible. As with DHCP, the best solution is
to simply move service to another machine.
To that end, I created a new VM, *vpn0.pyrocufflink.blue*, to host both
strongSwan and OpenVPN. For this to work, the necessary TCP/UDP ports
need to be forwarded, of course, and all of the remote subnets need
static routes on the gateway, specifying this machine as the next hop.
Additionally, ICMP redirects need to be disabled, to prevent confusing
the routing tables of devices on the same subnet as the VPN gateway.
The routes to FireMon networks are now defined using the
`firemon_networks` Ansible variable. The global `iroute` and
client-specific `route` options are generated from the CIDR blocks
specified in this list.
The *aria2* role installs the *aria2* download manager and sets it up to
run as a system service with RPC enabled. It also sets up the web UI,
though that must be installed manually from an archive, for now.
For hosts that do not have any TSIG keys, the `named_keys` variable
still must be defined (to an empty iterable) in order for the template
to expand properly.
To avoid having a single point of failure, a second recursive DNS server
is necessary. This will be useful in cases where the VM hosts must both
be taken offline, but Internet access is still required.
The new server, *dns1.pyrocufflink.blue*, has all the same zones defined
as the original. It forwards the *pyrocufflink.blue* zone and
corresponding reverse zones to the domain controllers, and acts as a
slave for the *pyrocufflink.red* zone.
*smtp1.pyrocufflink.blue* is a VM that will replace
*smtp0.pyrocufflink.blue*, a Raspberry Pi.
I decided that there is little use in having the availability guarantee of
a discreet machine for the SMTP relay. The only system that would NEED
to send mail if the VM host fails is Zabbix, which operates as its own
relay anyway. As such, the main relay can be a VM, and the Raspberry Pi
can be repurposed as a recursive DNS server.
The *koji-web* role installs and configures the Koji Web GUI front-end
for Koji. It requires Apache and mod_wsgi. A client certificate is
required for authentication to the hub, and must be placed in the
host-specific subdirectory of `certs/koji`.
The *koji-client* role is a generic role that can be used to configure
the Koji client library/`koji` CLI tool. By default, it manages the
default configuration at `/etc/koji`, but by using the
`koji_client_dir`, `koji_client_user`, and `koji_client_id` variables,
it can be used to configure per-user client configuration as well.
The *kojira* role sets up the Koji repository agent to manage
repository metadata for build tags. It runs as a daemon, usually on the
same machine as the Koji hub. A client certificate is required for
authentication, and must be supplied by placing it in the
`certs/koji/{{ inventory_hostname }}` directory.
The *koji-gc* role sets up the Koji garbage collector utility to run
periodically. It uses cron for scheduling. A client certificate is
required for authentication, and must be supplied by placing it in the
`certs/koji/{{ inventory_hostname }}` directory.
The minimal Fedora installation does not include a cron implementation.
The *cronie* role can be applied to hosts installed in this way to
ensure that cron is available for task scheduling.
The *burp-client* role installs and configures a BURP client. It should
support RHEL/CentOS/Fedora and Gentoo.
To manage the client password and other server-mandated configuration,
the role uses Ansible's delegation feature to generate a configuration
file in the "clientconfdir" on the BURP server.
An hourly cron task is scheduled that runs `burp -a t` every hour. This
allows the server to configure backup timebands and intervals.
The *burp-server* role installs and configures a BURP server. It is
adapted from a previous iteration, and should support CentOS/RHEL/Fedora
and Gentoo, as well as both BURP 1.x and 2.x (depending on which version
gets installed by the system package manager).
To manage the certificate authority, the *burp-server* role uses the
`burp_ca` command. This has the advantage of not requiring any external
certificate management, but effectively binds the CA to a specific
machine.
The value of the `shlib_directory` is dependent the system architecture.
Specifically, x86_64 machines use `/usr/lib64/postfix`, while everything
else uses `/usr/lib/postfix`. This role was originally deployed on a
Raspberry Pi, so the original path was correct. Attempting to deploy it
on an x86_64 machine revealed the error.
This commit adds a new task that loads a variables file based on the
architecture. Each option defines an `arch_libdir` variable, which can
be expanded in the `postfix_shlib_directory` variable as needed.
*myala.pyrocufflink.jazz* no longer hosts any public-facing websites,
and is in fact shut down. To prevent HAproxy from failing to start
because it cannot resolve the name, this backend needs to be removed.
The *fileserver* role configures Samba as a file sharing server. It uses
the *samba* role to handle cross-distribution installation of Samba
itself, and is focused primarily on configuring shared folders.
This commit adds an *after* ordering dependency on the network device
unit to the *wait-global-address@.service* template unit. Without this
dependency, the service will wait forever for a global address if the
device does not exist. With the dependency, though, if the device does
not appear within the default timeout, the wait service will never
start, causing all dependent services to fail, but allowing the boot
process to continue.
Dustin