Vendor-Agnostic Reconfiguration of Kubernetes Clusters in Cloud Federations

Kubernetes (K8s) defines standardized APIs for container-based cluster orchestration such that it becomes possible for application managers to deploy their applications in a portable and interopable manner. However, a practical problem arises when the same application must be replicated in a distributed fashion across different edge, fog and cloud sites; namely, there will not exist a single K8s vendor that is able to provision and manage K8s clusters across all these sites. Hence, the problem of feature incompatibility between different K8s vendors arises. A large number of documented features in the open-source distribution of K8s are optional features that are turned off by default but can be activated by setting specific combinations of parameters and plug-in components in configuration manifests for the K8s control plane and worker node agents. However, none of these configuration manifests are standardized, giving K8s vendors the freedom to hide the manifests behind a single, more restricted, and proprietary customization interface. Therefore, some optional K8s features cannot be activated consistently across K8s vendors and applications that require these features cannot be run on those vendors. In this paper, we present a unified, vendor-agnostic feature management approach for consistently configuring optional K8s features across a federation of clusters hosted by different Kubernetes vendors. We describe vendor-agnostic reconfiguration tactics that are already applied in industry and that cover a wide range of optional K8s features. Based on these tactics, we design and implement an autonomic controller for declarative feature compatibility management across a cluster federation. We found that the features configured through our vendor-agnostic approach have no impact on application performance when compared with a cluster where the features are configured using the configuration manifests of the open-source K8s distribution. Moreover, the maximum time to complete reconfiguration of a single feature is within 100 seconds, which is 6 times faster than using proprietary customization interfaces of mainstream K8s vendors such as Google Kubernetes Engine. However, there is a non-negligible disruption to running applications when performing the reconfiguration to an existing cluster; this disruption impact does not appear using the proprietary customization methods of the K8s vendors due to the use of rolling upgrade of cluster nodes. Therefore, our approach is best applied in the following three use cases: (i) when starting up new K8s clusters, (ii) when optional K8s features of existing clusters must be activated as quickly as possibly and temporary disruption to running applications can be tolerated or (iii) when proprietary customization interfaces do not allow to activate the desired optional feature.