dustin/dynk8s-provisioner/pipeline/head This commit looks goodDetails
Jenkins jobs that build container images in user namespaces need access
to `/dev/fuse`, which is provided by the [fuse-device-plugin][0]. This
plugin runs as a DaemonSet, which updates the status of the node it's
running on when it starts to indicate that the FUSE device is available.
When scaling up from zero nodes, Cluster Autoscaler has no way to know
that this will occur, and therefore cannot determine that scaling up the
ASG will create a node with the required resources. Thus, the ASG needs
a tag to inform CA that the nodes it creates will indeed have the
resources and scaling it up will allow the pod to be scheduled.
Although this feature of CA was added in 1.14, it apparently got broken
at some point and no longer works in 1.22. It works again in 1.26,
though.
[0]: https://github.com/kuberenetes-learning-group/fuse-device-plugin/tree/master
Instead of hard-coding the AMI ID of the Fedora build we want, we can
use the `aws_ami` data source to search for it. The Fedora release team
has a consistent naming scheme for AMIs, so finding the correct one is
straightforward.
The Cluser Autoscaler uses EC2 Auto-Scaling Groups to configure the
instances it launches when it determines additional worker nodes are
necessary. Auto-Scaling Groups have an associated Launch Template,
which describes the properties of the instances, such as AMI ID,
instance type, security groups, etc.
When instances are first launched, they need to be configured to join
the on-premises Kubernetes cluster. This is handled by *cloud-init*
using the configuration in the instance user data. The configuration
supplied here specifies the Fedora packages that need to be installed on
a Kubernetes worker node, plus some additional configuration required by
`kubeadm`, `kubelet`, and/or `cri-o`. It also includes a script that
fetches the WireGuard client configuration and connects to the VPN,
finalizes the setup process, and joins the cluster.
Dustin