Scheduled upgrade from November 26, 07:00 UTC to November 26, 17:00 UTC
Kindly note that during the maintenance window, app.hopsworks.ai will not be accessible.
5
View the Changes
.svg){kind=small}
For Hopsworks, installing in air-gapped environments is essential. We must deliver air-gap guarantees to our customers. An air-gapped environment refers to a computer network or system that is physically isolated from external networks, including the internet. In the context of Hopsworks, it specifically means a Kubernetes cluster that is completely disconnected from outside networks, preventing any external communication or access. However, after hundreds of lines of code, dependencies, and scripting, it's normal to expect some air-gap leaks. For instance, we might find an innocently forgotten wget command hidden within the Hopsworks code.
As coding enthusiasts, we recently asked ourselves, "Can we evaluate this automatically? And if so, can we turn it into a test?" Each run of the CI pipeline of Hopsworks involves creating a multinode Kubernetes cluster with Vagrant virtual machines. In this tutorial, we'll walk through how we extended this process to test air-gapped installations as well.
After evaluating our options, we consider using iptable rules as the best and most generic way of testing the air-gapped environment. We test the air-gapped environment by implementing fine-grained iptable rules. In the following text, we describe how this approach works.
As mentioned before the premise for this air gapped test is to block everything from the kubernetes nodes to the outside world. Therefore the very first iptable rule should be.
That effectively blocks everything. In fact, if you run such a rule inside a node your vagrant ssh session will be blocked and you will probably lose the control of the machine. Therefore, we should have an exception for host-guest communication in the network.
In a vagrant node, the host IP is usually the default network routing. The first line gets us that IP, we then allow the guest to access the host.
Our Kubernetes cluster is a multinode environment. This means that we have several vagrant machines that are communicating between each other. We have a fixed subnetwork for these nodes. We allow communication between them with the following rule.
At this point, we have isolated the communication for the nodes: they communicate between each other and with the host, everything else is blocked. Although we aim for an air-gapped cluster, our CI process requires pulling several container images. While this isn't strictly air-gapped, blocking access to all registries except our trusted one provides a good simulation. This is because companies usually allow traffic through a proxy to ensure air gap. Thus, we need to add rules that allow access only to our "approved" image registry.
However, if we set ALLOWED_DOMAINS="docker.hops.works" and then run ctr -i pull docker.hops.works/busybox, it will fail. This is because the DNS resolution for docker.hops.works is also blocked. To resolve this issue, we need to add the following rules:
We add a list of known DNS IPs and allow communication with them, but only on port 53. This enables our cluster nodes to resolve the IP address behind docker.hops.works. Consequently, running ctr i pull docker.hops.works/busybox now works effectively. The DNS resolution returns the same IP we previously added to our allowed list.
We also permit communication within the IP range 10.96.0.0/16. This specific range is designated for the Kubernetes API server, which is crucial for the proper functioning of the cluster. By allowing this communication, we ensure that the nodes can interact with the control plane components. This allowance is a necessary exception to our air-gapped setup, as it enables the internal workings of the Kubernetes cluster while still maintaining isolation from external networks.
At this stage, we can effectively test if our Kubernetes deployments exclusively use images from docker.hops.works. However, we aim to dig deeper: once we've pulled the deployment images, are the deployment services truly air-gapped?
We install some needed operators that dynamically add other images, so we can't simply assume we're air-gapped by checking for external registries. This is the case also for Kubernetes CNI operators. For example, certain Kubernetes CNI rules, like the ones from calico, can bypass our written rules for Pods, potentially compromising our isolation efforts.
[A Pod is Kubernetes' basic unit of work—it groups containers that run on the same node and collectively form services.]
Pods have their own IPs, and the CNI handles Pod-to-Pod and Pod-to-external communication routing. The catch is that CNIs also modify node routing at a low level. Consequently, our rules may become ineffective for Pod-Pod and Pod-external communication if they don't overlap with these modifications.
To address this issue, we employ the same approach used for inter-node communication. When configuring a Kubernetes cluster, we can specify the Pod IP range. This allows us to preemptively block all communication by default, just as we did for node communication. In the snippet below, the Pod IP range is set to 10.244.0.0/16.
We also need to allow communication between the nodes and any pod.
Pods need to also communicate potentially with our whitelisted IPs and the DNS navigation. We need to modify the rules above in the bash loops.
While we've effectively created an air-gapped environment with iptables rules, we'd like to know what's being blocked for debugging and auditing our services. Specifically, we want to identify which IP is blocked from which Pod. Since a Pod isolates a service, we could then say, "Aha! Webserver X is trying to access IP Y." This insight would be invaluable for troubleshooting and maintaining our air-gapped setup.
If you add the following LOG rules just before DROP actions, the packet will be caught and logged.
Iptables rules are applied in reverse order, from bottom to top. Therefore, after compiling all the rules mentioned above, we've created the following script that must be executed on each cluster node.
We have effectively created an air-gapped test for evaluating every change in Hopsworks. This test verifies that a deployment in Kubernetes can be conducted in air-gapped environments. We achieve this by implementing fine-grained iptables rules that block all external access, except for a whitelist of domain names that includes our images registry.
.svg)
