Grafana Loki explicitly eschews built-in authentication. In fact, its
[documentation][0] states:
> Operators are expected to run an authenticating reverse proxy in front
> of your services.
While I don't really want to require authentication for agents sending
logs, I definitely want to restrict querying and viewing logs to trusted
users.
There are _many_ reverse proxy servers available, and normally I would
choose _nginx_. In this case, though, I decided to try Caddy, mostly
because of its built-in ACME support. I wasn't really happy with how
the `fetchcert` system turned out, particularly using the Kubernetes API
token for authentication. Since the token will eventually expire, it
will require manual intervention to renew, thus mostly defeating the
purpose of having an auto-renewing certificate. So instead of using
_cert-manager_ to issue the certificate and store it in Kubernetes, and
then having `fetchcert` download it via the Kubernetes API, I set up
_step-ca_ to handle issuing the certificate directly to the server. When
Caddy starts up, it contacts _step-ca_ via ACME and handles the
challenge verification automatically. Further, it will automatically
renew the certificate as necessary, again using ACME.
I didn't spend a lot of time optimizing the Caddy configuration, so
there's some duplication there (i.e. the multiple `reverse_proxy`
statements), but the configuration works as desired. Clients may
provide a certificate, which will be verified against the trusted issuer
CA. If the certificate is valid, the client may access any Loki
resource. Clients that do not provide a certificate can only access the
ingestion path, as well as the "ready" and "metrics" resources.
[0]: https://grafana.com/docs/loki/latest/operations/authentication/
According to the [Grafana Loki documentation][0], sending SIGHUP to the
Loki process will instruct it to reload its configuration. This is
necessary in order for it to re-read its server certificate after it has
been renewed.
[0]: https://grafana.com/docs/loki/latest/configure/#reload-at-runtime
Before going into production with Grafana Loki, I want to set it up to
use TLS. To that end, I have configured _cert-manager_ to issue it a
certificate, signed by _DCH CA_. In order to use said certificate,
we need to configure `fetchcert` to run on the Loki server.
Dustin