When an instance is terminated, any bootstrap tokens assigned to it are
now deleted. Though these would expire anyway, deleting them ensures
that they cannot be used again if they happened to be leaked while the
instance was running. Further, it ensures that attempting to fetch the
`kubeadm` configuration for the instance will return an HTTP 404 Not
Found response once the instance has terminated.
The *GET /kubeadm/kubeconfig/<instance-id>* operation returns a
configuration document for `kubeadm` to add the node to the cluster as a
worker. The document is derived from the kubeconfig stored in the
`cluster-info` ConfigMap, which includes the external URL of the
Kubernetes API server and the root CA certificate used in the cluster.
The bootstrap token assigned to the specified instance is added to the
document for `kubeadm` to use for authentication. The kubeconfig is
stored in the ConfigMap as a string, so extracting data from it requires
deserializing the YAML document first.
In order to access the cluster information ConfigMap, the service
account bound to the pod running the provisioner service must have the
appropriate permissions.
The * GET /wireguard/config/<instance-id>* resource returns the
WireGuard client configuration assigned to the specified instance ID.
The resource contents are stored in the Kubernetes Secret, in a data
field named `wireguard-config`. The contents of this field are returned
directly as a string, without any transformation. Thus, the value must
be a complete, valid WireGuard configuration document. Instances will
fetch and save this configuration when they first launch, to configure
their access to the VPN.
Setting up the WireGuard client requires several pieces of information,
beyond the node private key and peer's public key. The peer endpoint
address/port, peer public key, and node IP address are also required.
As such, naming the resource a "key" is somewhat misleading.
In order to join the on-premises Kubernetes cluster, EC2 instances will
need to first connect to the WireGuard VPN. The *dynk8s* provisioner
will provide keys to instances to configure their WireGuard clients.
WireGuard keys must be pre-configured on the server and stored in
Kubernetes as *dynk8s.du5t1n.me/wireguard-key* Secret resources. They
must also have a `dynk8s.du5t1n.me/ec2-instance-id` label. If this
label is empty, the key is available to be assigned to an instance.
When an EventBridge event is received indicating an instance is now
running, a WireGuard key is assigned to that instance (by setting the
`dynk8s.du5t1n.me/ec2-instance-id` label). Conversely, when an event is
received indicating that the instance is terminated, any WireGuard keys
assigned to that instance are freed.
The lifecycle of ephemeral Kubernetes worker nodes is driven by events
emitted by Amazon EventBridge and delivered via Amazon Simple
Notification Service. These events trigger the *dynk8s* provisioner to
take the appropriate action based on the state of an EC2 instance.
In order to add a node to the cluster using `kubeadm`, a "bootstrap
token" needs to be created. When manually adding a node, this would be
done e.g. using `kubeadm token create`. Since bootstrap tokens are just
a special type of Secret, they can be easily created programmatically as
well. When a new EC2 instance enters the "running" state, the
provisioner creates a new bootstrap token and associates it with the
instance by storing the instance ID in a label in the Secret resource's
metadata.
The initial implementation of the event handler is rather naïve. It
generates a token for every instance, though some instances may not be
intended to be used as Kubernetes workers. Ideally, the provisioner
would only allocate tokens for instances matching some configurable
criteria, such as AWS tags. Further, a token is allocated every time
the instance enters the running state, even if a token already exists or
is not needed.
The `terraform` directory contains the resource descriptions for all AWS
services that need to be configured in order for the dynamic K8s
provisioner to work. Specifically, it defines the EventBridge rule and
SNS topic/subscriptions that instruct AWS to send EC2 instance state
change notifications to the *dynk8s-provisioner*'s HTTP interface.
dustin/dynk8s-provisioner/pipeline/head This commit looks goodDetails
Fedora 36 has OpenSSL 3, while the *rust* container image has OpenSSL
1.1. Since Fedora 35 is still supported, and it includes OpenSSL 1.1,
we can use it as our base for the runtime image.
Upon receipt of a notification or unsubscribe confirmation message from
SNS, after the message signature has been verified, the receiver will
now write the re-serialized contents of the message out to the
filesystem. This will allow the messages to be inspected later in order
to develop additional functionality for this service.
The messages are saved in a `messages` director within the current
working directory. This directory contains a subdirectory for each SNS
topic. Within the topic subdirectories, the each message is saved in a
file named with the message timestamp and ID.
This commit introduces the HTTP interface for the dynamic K8s node
provisioner. It will serve as the main communication point between the
ephemeral nodes in the cloud, sharing the keys and tokens they require
in order to join the Kubernetes cluster.
The initial functionality is simply an Amazon SNS notification receiver.
SNS notifications will be used to manage the lifecycle of the dynamic
nodes.
For now, the notification receiver handles subscription confirmation
messages by following the link provided to confirm the subscription.
All other messages are simply written to the filesystem; these will be
used to implement and test future functionality.
The `model::sns::Message` enumeration provides a mechanism for
deserializing a JSON document into the correct type. It will be used by
the HTTP operation that receives messages from SNS in order to determine
the correct action to take in response to the message.
In order to prevent arbitrary clients from using the provisioner to
retrieve WireGuard keys and Kubernetes bootstrap tokens, access to those
resources *must* be restricted to the EC2 machines created by the
Kubernetes Cloud Autoscaler. The key to the authentication process will
be SNS notifications from AWS to indicate when new EC2 instances are
created; everything that the provisioner does will be associated with an
instance it discovered through an SNS notification.
SNS messages are signed using PKCS#1 v1.5 RSA-SHA1, with a public key
distributed in an X.509 certificate. To ensure that messages received
are indeed from AWS, the provisioner will need to verify those
signatures. Messages with missing or invalid signatures will be
considered unsafe and ignored.
The `model::sns` module includes the data structures that represent SNS
messages. The `sns::sig` module includes the primitive operations for
implementing signature verification.
Dustin