r/nextcloud-db-cert: Fetch client cert from k8s

Currently, the certificate authority that issues certificates for
PostgreSQL clients is hosted in Kubernetes and managed by
_cert-manager_.  Certificates it issues are stored in Kubernetes Secret
resources, making them easy to consume by applications running in the
cluster, but not for anything outside.  Since Nextcloud runs on its own
VM, we need a way to get the certificate out of the Secret and into a
file on that machine.  To that end, I've written the
`nextcloud-fetch-cert.py` script.  This script uses a Kubernetes Service
Account token to authenticate to the Kubernetes API and download the
contents of the Secret.  It runs periodically, triggered by a systemd
timer unit, to ensure the certificate is always up-to-date.

The obvious drawback to this approach is the requirement for a static
token.  Since there's not really a way to "renew" Service Account
tokens, it needs to be issued with a fairly long duration, to mitigate
the risk of being unable to fetch a new certificate once it has expired
because the token has also expired.  This somewhat negates the advantage
of using certificates for authentication, since now the machine needs a
static, pre-defined secret.

At some point, I may deploy another instance of _step-ca_ to manage the
PostgreSQL client CA.  Clients can then use e.g. `certbot` or `step ca
certificate` to obtain their certificates.  I chose not to implement
this yet, though for a couple of reasons.  First, I need to move the
Nextcloud database very soon, so we switch to using `restic` for backups
without having to deal with the database.  Second, I am still
considering moving Nextcloud into Kubernetes eventually, where it will
be able to get the Secret directly; since Nextcloud is the only client
outside the cluster, it may not be worth setting up _step-ca_ in that
case.

roles/nextcloud-db-cert/files/fetch-cert.py

@@ -0,0 +1,83 @@
+#!/usr/bin/env python3
+import base64
+import json
+import logging
+import os
+import ssl
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+try:
+    from systemd.journal import JournalHandler as LogHandler
+except ModuleNotFoundError:
+    LogHandler = logging.StreamHandler
+
+
+log = logging.getLogger('fetchcert')
+
+
+CREDENTIALS_DIRECTORY = Path(os.environ['CREDENTIALS_DIRECTORY'])
+
+CA_FILE = Path('/etc/pki/ca-trust/kube-root-ca.crt')
+CERT_OUT = Path('/etc/nextcloud/postgresql.cer')
+KEY_OUT = Path('/etc/nextcloud/postgresql.key')
+
+BASE_URL = 'https://kubernetes.pyrocufflink.blue:6443'
+NAMESPACE = 'postgresql-ca'
+SECRET = 'nextcloud-client'
+
+
+def main():
+    if 'LOG_LEVEL' in os.environ:
+        logging.root.setLevel(os.environ['LOG_LEVEL'].upper())
+    logging.root.addHandler(LogHandler())
+
+    token_path = CREDENTIALS_DIRECTORY / 'nextcloud.fetchcert.token'
+    log.debug('Reading token from %s', token_path)
+    token = token_path.read_text().rstrip()
+
+    ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
+    ctx.load_verify_locations(cafile=CA_FILE)
+
+    url = f'{BASE_URL}/api/v1/namespaces/{NAMESPACE}/secrets/{SECRET}'
+    headers = {
+        'Authorization': f'Bearer {token}'
+    }
+    req = urllib.request.Request(url, headers=headers)
+    log.info('Fetching Secret from %s', url)
+    try:
+        res = urllib.request.urlopen(req, context=ctx)
+    except urllib.error.HTTPError as e:
+        log.error('%s %s', e, e.read())
+        raise SystemExit(1)
+    with res:
+        body = json.load(res)
+    log.info('Received HTTP reponse %d %s', res.status, res.reason)
+    cert = base64.b64decode(body['data']['tls.crt'].encode('ascii'))
+    key = base64.b64decode(body['data']['tls.key'].encode('ascii'))
+    cert_new = CERT_OUT.with_suffix(f'{CERT_OUT.suffix}.new')
+    key_new = KEY_OUT.with_suffix(f'{KEY_OUT.suffix}.new')
+    if not CERT_OUT.exists() or CERT_OUT.read_bytes() != cert:
+        log.debug('Writing certificate to %s', cert_new)
+        with cert_new.open('wb') as f:
+            os.fchown(f.fileno(), 0, 48)
+            os.fchmod(f.fileno(), 0o0444)
+            f.write(cert)
+    if not KEY_OUT.exists() or KEY_OUT.read_bytes() != key:
+        log.debug('Writing private key to %s', key_new)
+        with key_new.open('wb') as f:
+            os.fchown(f.fileno(), 0, 48)
+            os.fchmod(f.fileno(), 0o0440)
+            f.write(key)
+    if cert_new.exists():
+        log.debug('Renaming %s to %s', cert_new, CERT_OUT)
+        cert_new.rename(CERT_OUT)
+    if key_new.exists():
+        log.debug('Renaming %s to %s', key_new, KEY_OUT)
+        key_new.rename(KEY_OUT)
+    log.info('Successfully fetched certificate from Kubernetes Secret')
+
+
+if __name__ == '__main__':
+    main()

roles/nextcloud-db-cert/files/nextcloud-fetch-cert.service

@@ -0,0 +1,17 @@
+[Unit]
+Description=Fetch Nextcloud database client certificate
+Wants=network-online.target
+After=network-online.target
+
+[Service]
+Type=oneshot
+Environment=LOG_LEVEL=debug
+ExecStart=/usr/local/libexec/nextcloud-fetch-cert.py
+LoadCredential=nextcloud.fetchcert.token
+CapabilityBoundingSet=CAP_DAC_OVERRIDE CAP_CHOWN
+PrivateTmp=yes
+ProtectHome=yes
+ProtectKernelTunables=yes
+ProtectProc=invisible
+ProtectSystem=full
+ReadWritePaths=/etc/nextcloud

roles/nextcloud-db-cert/handlers/main.yml

@@ -0,0 +1,8 @@
+- name: reload systemd
+  systemd:
+    daemon_reload: true
+
+- name: restart nextcloud-fetch-cert.timer
+  systemd:
+    name: nextcloud-fetch-cert.timer
+    state: restarted

roles/nextcloud-db-cert/tasks/main.yml

@@ -0,0 +1,71 @@
+- name: ensure nextcloud db cert fetch script is installed
+  copy:
+    src: fetch-cert.py
+    dest: /usr/local/libexec/nextcloud-fetch-cert.py
+    owner: root
+    group: root
+    mode: u=rwx,go=rx
+  notify:
+  - restart nextcloud-fetch-cert.timer
+  tags:
+  - copy-script
+
+- name: ensure nextcloud db cert fetch token credential exists
+  copy:
+    dest: /etc/credstore/nextcloud.fetchcert.token
+    content: |+
+      {{ nextcloud_fetchcert_token }}
+    owner: root
+    group: root
+    mode: u=rw,go=
+  diff: false
+  tags:
+  - credentials
+
+- name: ensure kubernetes ca certificate is installed
+  copy:
+    src: kube-root-ca.crt
+    dest: /etc/pki/ca-trust/kube-root-ca.crt
+    owner: root
+    group: root
+    mode: u=rw,go=r
+  tags:
+  - cacert
+
+- name: ensure nextcloud cert fetch timer unit is installed
+  template:
+    src: nextcloud-fetch-cert.timer.j2
+    dest: /etc/systemd/system/nextcloud-fetch-cert.timer
+    owner: root
+    group: root
+    mode: u=rw,go=r
+  notify:
+  - reload systemd
+  - restart nextcloud-fetch-cert.timer
+  tags:
+  - systemd
+- name: ensure nextcloud cert fetch service unit is installed
+  copy:
+    src: nextcloud-fetch-cert.service
+    dest: /etc/systemd/system/nextcloud-fetch-cert.service
+    owner: root
+    group: root
+    mode: u=rw,go=r
+  notify:
+  - reload systemd
+  - restart nextcloud-fetch-cert.timer
+  tags:
+  - systemd
+
+- name: ensure nextcloud cert fetch timer is enabled
+  systemd:
+    name: nextcloud-fetch-cert.timer
+    enabled: true
+  tags:
+  - service
+- name: ensure nextcloud cert fetch timer is started
+  systemd:
+    name: nextcloud-fetch-cert.timer
+    state: started
+  tags:
+  - service

roles/nextcloud-db-cert/templates/nextcloud-fetch-cert.timer.j2

@@ -0,0 +1,9 @@
+[Unit]
+Description=Periodically fetch Nextcloud database client certificate
+
+[Timer]
+OnActiveSec=1s
+OnUnitInactiveSec=12h
+
+[Install]
+WantedBy=timers.target