Node Pools

With the release of GSK Version 1.30, support for Node Pools has been introduced, allowing users to create dedicated groups of nodes optimized for different types of workloads. This feature provides enhanced flexibility and control over resource allocation.

For deep dives into the underlying Kubernetes concepts used in GSK Node Pools, refer to the official Kubernetes documentation:

Overview

Node Pools in GSK Version 1.30 allow for the creation and management of multiple groups of nodes, each configurable to suit specific workload requirements.

This enables fine-grained control over infrastructure resources, allowing you to:

  1. Scale worker nodes within a pool to meet workload demands.

  2. Configure resources such as CPU, memory (RAM), and storage for each node pool, including advanced storage options like Rocket Storage.

  3. Add or remove pools dynamically to optimize the cluster’s performance based on the workload.

By segmenting workloads across different node pools, you can ensure that resource-intensive tasks are isolated from lightweight processes, improving performance, resource utilization, and operational efficiency.

Support for Taints and Labels

Starting with GSK Version 1.32, taints and labels are fully supported for Node Pools.

In versions prior to 1.32 this feature was not supported, which limited granular control over workload isolation.

Now, you can apply taints directly to a node pool to “repel” pods, ensuring that only pods with a matching toleration can be scheduled on those nodes.

You can also apply labels directly to a node pool to “attract” pods ensuring that only pods with matching labels can be scheduled on these nodes

This is the primary mechanism for creating dedicated node pools and enforcing strict workload isolation.

For complete workload placement control, you can now use a combination of:

  • Taints and Tolerations: To repel and restrict pods from a node pool (e.g., PreferNoSchedule NoSchedule, NoExecute).
  • Labels (with nodeSelector or nodeAffinity): To attract and assign pods to a specific node pool.

Label and Taint Validation Rules

The labels and taints applied to node pools are subject to the following validation rules, which adhere to Kubernetes standards.

Label and Taint Keys

The key for both labels and taints follows a standard Kubernetes format, which consists of an optional prefix and a required name, separated by a slash (/).

Prefix (Optional):

  • If present, it must be a valid DNS subdomain (e.g., my-company.com, company.internal).
  • It must start and end with a lowercase alphanumeric character (a-z, 0-9).
  • It can contain hyphens (-) and dots (.) in the middle.

Name (Required):

  • This is the part after the / (or the entire string if no prefix is used).
  • It must start and end with an alphanumeric character (a-z, A-Z, 0-9).
  • It can contain hyphens (-), underscores (_), and dots (.) in the middle.

Forbidden Keys (Reserved)

You should never use these keys, as they are reserved for GSK Internal use.

  • kubernetes.io/hostname
  • kubernetes.io/os
  • kubernetes.io/arch
  • beta.kubernetes.io/os
  • node-role.kubernetes.io/master

Label and Taint Values

The value for labels and taints must adhere to the following rules:

  • It can be an empty string.
  • If not empty, it must start and end with an alphanumeric character (a-z, A-Z, 0-9).
  • It can contain hyphens (-), underscores (_), and dots (.) in the middle.

Taint Effects

Taints require an effect field, which specifies what happens to pods that do not tolerate the taint.

The allowed values are:

  • NoSchedule: Pods will not be scheduled on the node. Existing pods are not affected.
  • PreferNoSchedule: The system will try to avoid scheduling pods on the node, but it’s not a hard requirement.
  • NoExecute: Pods will not be scheduled on the node, and any existing pods on the node that do not tolerate the taint will be evicted.

Validation Examples (Labels & Taints)

Here are examples to illustrate the validation rules for GSK labels and taints.

Valid Keys (No Prefix)

These examples only use the Name component. They are simple and commonly used.

  • app
  • environment
  • tier
  • team_name (underscores are allowed in the name)
  • project.id-123 (dots and hyphens are allowed in the middle)

Valid Keys (With Prefix)

These examples use both the Prefix and Name components, separated by a /.

The prefix is often used to namespace keys for your own organization.

  • company.internal/node-type
  • my-company.com/billing-code
  • app.security/pci-scope

Invalid Keys (Will Be Rejected)

  • _my-key Reason: Invalid Name. Cannot start with an underscore (_)
  • my-key- Reason: Invalid Name. Cannot end with a hyphen (-).
  • MyCompany.com/app Reason: Invalid Prefix. Cannot contain uppercase letters (M, C). Must be a valid DNS subdomain.
  • company.internal/ Reason: Invalid Name. The name part (after the /) is required and cannot be empty.
  • company..internal/app Reason: Invalid Prefix. Cannot have consecutive dots (..).
  • -company.internal/app Reason: Invalid Prefix. Cannot start with a hyphen (-).

Valid Values

  • production
  • frontend
  • high-memory
  • A1B-456-XYZ
  • true
  • 12345
  • core.infra-team
  • "" (An empty string is perfectly valid)

Invalid Values (Will Be Rejected)

  • -my-value- Reason: Cannot start or end with a hyphen (-).
  • my_value_ Reason: Cannot end with an underscore (_).
  • .staging Reason: Cannot start with a dot (.).

Valid Key-Value Pair Examples

Here is how valid keys and valid values look when paired together.

  • app: frontend
  • environment: production
  • my-company.com/billing-code: A1B-456-XYZ
  • app.security/pci-scope: true
  • needs-cleanup: “”

Use Case Examples

Here is how to use taints and labels to control workload placement in your GSK cluster.

Example 1: Dedicating Nodes for High-Memory Apps using Taints and Tolerations

Imagine you have a node pool named high-mem-pool with high-RAM specifications. You want to prevent regular pods from being scheduled there, reserving it only for memory-hungry applications.

1. Taint the Node Pool: First, you apply a NoSchedule taint to your node pool.

In GSK: You would apply this taint when creating or updating your high-mem-pool, likely via the GSK UI or Terraform.

Key: workload-type
Value: memory-intensive
Effect: NoSchedule

Any node in this pool is now tainted with workload-type=memory-intensive:NoSchedule.

2. Create a Pod with a Toleration: Now, a normal pod (like Nginx) will not be scheduled on this pool. To schedule your memory-heavy pod, you must add a matching tolerations block to its manifest:

apiVersion: v1
kind: Pod
metadata:
  name: large-cache-job
spec:
  containers:
  - name: redis-container
    image: redis:6.2-alpine
    # Optional: Defining high memory request to justify the placement
    resources:
      requests:
        memory: "16Gi"
      limits:
        memory: "32Gi"
    command: ["redis-server"]
  
  tolerations:
  # This toleration "matches" the taint on the node pool exactly
  - key: "workload-type"
    operator: "Equal"
    value: "memory-intensive" 
    effect: "NoSchedule"

This pod can now be scheduled on the high-mem-pool.

Example 2: Dedicating Nodes for High-Memory Apps using Labels and nodeSelector

Let’s say you have a node pool named high-mem-pool for your databases. You want to force your database pods to run on this pool.

1. Label the Node Pool: You apply a label to your node pool.

In GSK: You would apply this label when creating or updating your high-mem-pool node pool.

Key: node-tier
Value: database

2. Create a Pod with a nodeSelector: Now, configure your database pod to require that label by using nodeSelector:

apiVersion: v1
kind: Pod
metadata:
  name: my-database
spec:
  containers:
  - name: redis
    image: redis
  # This pod will ONLY be scheduled on nodes with the "node-tier=database" label
  nodeSelector:
    node-tier: database

This pod will now be attracted exclusively to nodes in the high-mem-pool.