Having name overrides for in-cluster services breaks ACME challenges,
because the server tries to connect to the Service instead of the
Ingress. To fix this, we need to configure both _cert-manager_ and
_step-ca_ to *only* resolve names using the network-wide DNS server.
I never ended up using _Step CA_ for anything, since I was initially
focused on the SSH CA feature and I was unhappy with how it worked
(which led me to write _SSHCA_). I didn't think about it much until I
was working on deploying Grafana Loki. For that project, I wanted to
use a certificate signed by a private CA instead of the wildcard
certificate for _pyrocufflink.blue_. So, I created *DCH CA R3* for that
purpose. Then, for some reason, I used the exact same procedure to
fetch the certificate from Kubernetes as I had set up for the
_pyrocufflink.blue_ wildcard certificate, as used by Frigate. This of
course defeated the purpose, since I could have just as easily used
the wildcard certificate in that case.
When I discovered that Grafana Loki expects to be deployed behind a
reverse proxy in order to implement access control, I took the
opportunity to reevaluate the certificate issuance process. Since a
reverse proxy is required to implement the access control I want (anyone
can push logs but only authenticated users can query them), it made
sense to choose one with native support for requesting certificates via
ACME. This would eliminate the need for `fetchcert` and the
corresponding Kubernetes API token. Thus, I ended up deciding to
redeploy _Step CA_ with the new _DCH CA R3_ for this purpose.
[Step CA] is an open-source online X.509 and SSH certificate authority
service. It supports issuing certificates via various protocols,
including ACME and its own HTTP API via the `step` command-line utility.
Clients can authenticate using a variety of methods, such as JWK, Open
ID Connect, or mTLS. This makes it very flexible and easy to introduce
to an existing ecosystem.
Although the CA service is mostly stateless, it does have an on-disk
database where stores some information, notably the list of SSH hosts
for which it has signed certificates. Most other operations, though, do
not require any persistent state; the service does not keep track of
every single certificate it signed, for example. It can be configured
to store authentication information (referred to as "provisioners") in
the database instead of the configuration file, by enabling the "remote
provisioner management" feature. This has the advantage of being able
to modify authentication configuration without updating a Kubernetes
ConfigMap and restarting the service.
The official Step CA documentation recommends using the `step ca init`
command initialize a new certificate authority. This command performs a
few steps:
* Generates an ECDSA key pair and uses it to create a self-signed root
certificate
* Generates a second ECDSA key pair and signs an intermediate CA
certificate using the root CA key
* Generates an ECDSA key pair and SSH root certificate
* Creates a `ca.json` configuration file
These steps can be performed separately, and in fact, I created the
intermediate CA certificate and signed it with the (offline) *dch Root
CA* certificate.
When the service starts for the first time, because
`authority/enableAdmin` is `true` and `authority/provisioners` is empty,
a new "Admin JWK" provisioner will be created automatically. This key
will be encrypted with the same password used to encrypt the
intermediate CA certificate private key, and can be used to create other
provisioners.
[Step CA]: https://smallstep.com/docs/step-ca/
Dustin