Now that Victoria Metrics is hosted in Kubernetes, it only makes sense
to host Grafana there as well. I chose to use a single-instance
deployment for simplicity; I don't really need high availability for
Grafana. Its configuration does not change enough to worry about the
downtime associated with restarting it. Migrating the existing data
from SQLite to PostgreSQL, while possible, is just not worth the hassle.
Invoice Ninja is a small business management tool. Tabitha wants to
use it for HLC.
I am a bit concerned about the code quality of this application, and
definitely alarmed at the data it send upstream, so I have tried to be
extra careful with it. All privileges are revoked, including access to
the Internet.
Since *mtrcs0.pyrocufflink.blue* (the Metrics Pi) seems to be dying,
I decided to move monitoring and alerting into Kubernetes.
I was originally planning to have a single, dedicated virtual machine
for Victoria Metrics and Grafana, similar to how the Metrics Pi was set
up, but running Fedora CoreOS instead of a custom Buildroot-based OS.
While I was working on the Ignition configuration for the VM, it
occurred to me that monitoring would be interrupted frequently, since
FCOS updates weekly and all updates require a reboot. I would rather
not have that many gaps in the data. Ultimately I decided that
deploying a cluster with Kubernetes would probably be more robust and
reliable, as updates can be performed without any downtime at all.
I chose not to use the Victoria Metrics Operator, but rather handle
the resource definitions myself. Victoria Metrics components are not
particularly difficult to deploy, so the overhead of running the
operator and using its custom resources would not be worth the minor
convenience it provides.
[sshca] is a simple web service I wrote to automatically create signed
SSH certificates for hosts' public keys. It authenticates hosts by
their machine UUID, which it can find using the libvirt API.
[sshca]: https://git.pyrocufflink.net/dustin/sshca
The `argocd` command needs to have its own OIDC client configuration,
since it works like a "public" client. To log in, run
```sh
argocd login argocd.pyrocufflink.blue --sso
```
[Argo CD] is a Kubernetes-native GitOps/continuous deployment manager.
It monitors the state of Kubnernetes resources, such as Pods,
Deployments, ConfigMaps, Secrets, and Custom Resources, and synchronizes
them with their canonical definitions from a Git repository.
*Argo CD* consists of various components, including a Repository
Service, an Application Controller, a Notification Controller, and an
API server/Web UI. It also has some optional components, such as a
bundled Dex server for authentication/authorization, and an
ApplicationSet controller, which we will not be using.
[Argo CD]: https://argo-cd.readthedocs.io/
Dustin