Kubernetes Usage

Kubernetes usage

Maestro uses kubernetes for orchestrating game room instances. It uses a unique namespace for each scheduler, and a unique pod for each game room instance.

We use client-go for communicating with kubernetes. The Runtime port is the interface used for managing resources, you can find all of the features we are using for managing k8s resources in it.

The diagram below shows how maestro components interact with kubernetes for managing resources.

flowchart BT classDef borderless stroke-width:0px classDef darkBlue fill:#00008B, color:#fff classDef brightBlue fill:#6082B6, color:#fff classDef gray fill:#62524F, color:#fff classDef gray2 fill:#4F625B, color:#fff subgraph maestroSystem[ ] subgraph k8s[ ] A3[Kubernetes] end class k8s,A3 brightBlue class A3, borderless subgraph WORKER[ ] A7[Operation Execution Worker manage kubernetes resources by creating/deleting/updating pods abnd namespaces] end class WORKER,A7 brightBlue class WORKER,A7 borderless WORKER--Create namespace HTTPS-->k8s WORKER--Delete namespace HTTPS-->k8s WORKER--Create pod HTTPS-->k8s WORKER--Delete pod HTTPS-->k8s subgraph RUNTIME_WATCHER[ ] A8[Runtime Watcher watch for change events in managed pods] end class RUNTIME_WATCHER,A8 brightBlue class RUNTIME_WATCHER,A8 borderless RUNTIME_WATCHER--List/Watch pods HTTPS-->k8s end click A3 "/csymapp/mermaid-c4-model/blob/master/AWAComponent.md" "AWA"

Runtime watcher

The runtime watcher component spawn two types of workers: one that is responsible for mitigating disruptions and another for processing change events in pods resources.

Disruption Worker

This worker consists in a single goroutine with a ticker. Each time it runs, it will check the number of occupied rooms at the time and try to mitigate disruptions. For k8s, this mitigation consists on applying a PDB to the scheduler's namespace, that has minAvailable equals to the number of occupied rooms plus a safety percentage.

One can configure the interval in which this worker runs and also the safety percentage in config/config.yaml:

runtimeWatcher:
  disruptionWorker:
    intervalSeconds: 5
    safetyPercentage: 0.05

We should always allow node consolidation to happen, thus if the total number of rooms is 1, then we set PDB's minAvailable: 0.

Pod Change Events Worker

For this type of worker, runtime watcher spawns multiple goroutines, maintaining a worker process for each scheduler that keeps watching and processing change events in pods resources. For doing that, it uses a pods informer, binding handlers for add, update and delete events for all pods managed by it.

This component is not responsible for updating/creating/deleting kubernetes resources, all it does is to watch for changes and update its game room instances internal representation using redis.

Operation execution worker

The worker uses kubernetes for managing pods and namespaces. It executes several operations that, alongside other side effects, will need to create, update, and delete namespaces and pods.

Currently, maestro does not check for HostPort conflict while creating new rooms

One important note regarding how maestro creates pods: each new requested game room instance will be assigned to a pseudo-random port to be used as HostPort.

Maestro uses the scheduler PortRange to generate the pseudo-random port. Currently, maestro does not check for HostPort conflict while creating new rooms. The final address of the game room will be composed of the Node address and the game room container assigned HostPort. That's the reason why maestro needs access for reading the Node addresses.

Configuring cluster access

Maestro needs the following permissions for managing resources in a kubernetes cluster: - nodes: read (we need to use the node address to compose the game room address); - pods: read, create, update, delete; - namespace: read, create, update, delete.

Maestro provides two ways for configuring kubernetes cluster access.

Using inCluster mode

Set adapters.runtime.kubernetes.inCluster config value to true or use its env var equivalent, the kubernetes client will be configured automatically using the same service account of the maestro component running pod.

The kubernetes client has a default rate limiter which implements a token bucket approach. The inCluster mode allows rate limit changes via adapters.runtime.kubernetes.qps and adapters.runtime.kubernetes.burst configs.

This mode is recommended to be used when running maestro components in the same cluster in which the schedulers and rooms will be managed.

Using kubeconfig mode

Populate adapters.runtime.kubernetes.kubeconfig and adapters.runtime.kubernetes.masterUrl configs or use its env var equivalent, the kubernetes client will be configured using the provided kubeconfig file and master url.