The *cri-o* package has moved from its own module into the base Fedora
repository, as Fedora is [eliminating modules][0]. The last modular
version was 1.25, which is too old to run pods with user namespaces.
Version 1.26 is available in the base repository, which does support
user namespaces.
[0]: https://fedoraproject.org/wiki/Changes/RetireModularity
Lately, cloud nodes seem to be failing to come up more frequently. I
traced this down to the fact that `/etc/resolv.conf` in the `kube-proxy`
container contains both the AWS-provided DNS server and the on-premises
server set by Wireguard. This evidently "works" correctly sometimes,
but not always. When it doesn't, the `kube-proxy` cannot resolve the
Kubernetes API server address, and thus cannot create the necessary
netfilter rules to forward traffic correctly. This causes pods to be
unable to communicate.
I am not entirely sure what the "correct" solution to this problem would
be, since there are various issues in play here. Fortunately, cloud
nodes are only ever around for a short time, and never need to be
rebooted. As such, we can use a "quick fix" and simply remove the
AWS-provided DNS configuration.
The default configuration for the *kubelet.service* unit does not
specify the path to the `config.yml` generated by `kubeadm`. Thus, any
settings defined in the `kublet-config` ConfigMap do not take effect.
To resolve this, we have to explicitly set the path in the `config`
property of the `kubeletExtraArgs` object in the join configuration.
The Cluser Autoscaler uses EC2 Auto-Scaling Groups to configure the
instances it launches when it determines additional worker nodes are
necessary. Auto-Scaling Groups have an associated Launch Template,
which describes the properties of the instances, such as AMI ID,
instance type, security groups, etc.
When instances are first launched, they need to be configured to join
the on-premises Kubernetes cluster. This is handled by *cloud-init*
using the configuration in the instance user data. The configuration
supplied here specifies the Fedora packages that need to be installed on
a Kubernetes worker node, plus some additional configuration required by
`kubeadm`, `kubelet`, and/or `cri-o`. It also includes a script that
fetches the WireGuard client configuration and connects to the VPN,
finalizes the setup process, and joins the cluster.
Dustin