Pod-Level Scheduling Metrics

Gthulhu provides pod-level scheduling metrics powered by eBPF, allowing you to observe low-level kernel scheduling behavior for each Pod in your cluster — without modifying application code.

Overview

The metrics collection pipeline works as follows:

graph LR
    A[Kernel tracepoints
sched_switch / sched_process_exit] -->|per-PID data| B[eBPF hash map
task_metrics]
    B -->|poll every N seconds| C[Go Collector
+ PodMapper]
    C -->|aggregate per-Pod| D[Prometheus /metrics]
    D --> E[Grafana Dashboard]
    C -->|query via API| F[Web GUI]
    C -->|EWMA + clustering| G[Adaptive Classification]

1. eBPF tracepoints (tp_btf/sched_switch, tp_btf/sched_process_exit) capture per-PID scheduling events in the kernel.
2. Go Collector periodically reads the BPF hash map and maps PIDs to Pods using /proc/<pid>/cgroup.
3. Per-PID metrics are aggregated into per-Pod metrics and exposed via Prometheus and the REST API.

Available Metrics

Prometheus metric names use the gthulhu_pod_ prefix, for example:

• gthulhu_pod_voluntary_ctx_switches_total
• gthulhu_pod_involuntary_ctx_switches_total
• gthulhu_pod_cpu_time_nanoseconds_total
• gthulhu_pod_wait_time_nanoseconds_total
• gthulhu_pod_run_count_total
• gthulhu_pod_cpu_migrations_total
• gthulhu_pod_process_count

All metrics carry labels: pod_name, pod_uid, namespace, node_name.

Configuring Metrics Collection

You can create metrics collection configurations using either the Web GUI or the PodSchedulingMetrics CRD.

Option A: Web GUI

1. Log in to the Gthulhu Web GUI (see Configuring the scheduling policies for access instructions).
2. Navigate to Pod Metrics in the sidebar.
3. Click New Config to open the configuration panel.

Fill in the following fields:

After saving, the configuration takes effect immediately — the eBPF collector on each node will begin tracking the matching Pods' processes.

Option B: PodSchedulingMetrics CRD

If you prefer declarative configuration, apply a PodSchedulingMetrics custom resource:

apiVersion: gthulhu.io/v1alpha1
kind: PodSchedulingMetrics
metadata:
  name: monitor-upf
  namespace: default
spec:
  labelSelectors:
    - key: app
      value: upf
  k8sNamespaces:
    - free5gc
  commandRegex: ".*"
  collectionIntervalSeconds: 10
  enabled: true
  metrics:
    voluntaryCtxSwitches: true
    involuntaryCtxSwitches: true
    cpuTimeNs: true
    waitTimeNs: false
    runCount: false
    cpuMigrations: false

Apply it with:

kubectl apply -f pod-scheduling-metrics.yaml

The CRD Watcher on each node detects the resource and dynamically updates the eBPF monitoring scope.

Viewing Runtime Metrics

Web GUI

On the Pod Metrics page, the Latest Collected Metrics table displays real-time data:

Click Refresh at any time to fetch the latest data.

REST API

You can also query runtime metrics programmatically:

# List all metrics configurations
curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8080/api/v1/pod-scheduling-metrics

# Get latest collected runtime metrics
curl -H "Authorization: Bearer $TOKEN" \
  http://localhost:8080/api/v1/pod-scheduling-metrics/runtime

The runtime endpoint returns a JSON response like:

{
  "success": true,
  "data": {
    "items": [
      {
        "namespace": "free5gc",
        "podName": "upf-pod-abc123",
        "nodeID": "worker-1",
        "voluntaryCtxSwitches": 15234,
        "involuntaryCtxSwitches": 892,
        "cpuTimeNs": 4820000000,
        "waitTimeNs": 120000000,
        "runCount": 16126,
        "cpuMigrations": 47
      }
    ],
    "warnings": []
  }
}

Prometheus & Grafana

The metrics are exported on the monitor's Prometheus endpoint (default port 9090). You can:

1. Add the Gthulhu monitor as a Prometheus scrape target.
2. Use Grafana to build dashboards visualizing scheduling behavior across Pods.

Example PromQL queries:

# Voluntary context switches rate per Pod
rate(gthulhu_pod_voluntary_ctx_switches_total[5m])

# CPU time per Pod in the last hour
increase(gthulhu_pod_cpu_time_nanoseconds_total[1h])

# Top 10 Pods by involuntary context switches
topk(10, rate(gthulhu_pod_involuntary_ctx_switches_total[5m]))

Adaptive Classification

Adaptive Classification automatically identifies workload patterns and potential scheduling bottlenecks from each Pod's short-term and long-term scheduling behavior. It builds EWMA (exponentially weighted moving average) profiles from collected pod-level metrics, then uses adaptive clustering to label Pods with inferred workload types. This helps you decide whether to adjust CPU resources, priority, or NUMA placement.

On the Pod Metrics page, the Adaptive Classification table can be filtered by:

The table columns are:

Classifier Phases

Workload Types

Drift and Recommended Actions

The drift score is the average deviation between short-term and long-term EWMA profiles, normalized by long-term variance. A score above 1.5 enters the drifting state; if the high offset continues for 3 consecutive periods, the classifier confirms transitioning, meaning the workload type may have changed.

Prerequisites