GuidesKubernetes monitoring

Kubernetes monitoring: what to watch and what to alert on

Updated 17 October 2026 · 8 minute read

Kubernetes monitoring means answering three questions continuously. Do the nodes have the capacity and health to run work? Are workloads running the number of healthy replicas they should? Are the applications inside them serving requests well? Kubernetes heals many failures on its own, so the signals that matter most are the ones it cannot fix: pods that keep crashing, pods that cannot be scheduled, and nodes that are short of resources.

This guide covers the layers to watch, the key signals, the kubectl commands to check them by hand, what to alert on and how to run a monitoring collector safely.

The layers of a cluster

The first four tell you whether the platform is healthy. Only the last tells you whether users are affected. You need both: a cluster can look green while a service returns errors, and a service can look fine while a node is about to evict its pods.

The key signals

Node readiness and pressure

Each node reports a Ready condition and three pressure conditions: MemoryPressure, DiskPressure and PIDPressure. A node that is not ready gets no new pods, and after a timeout its pods are evicted. A node under pressure starts evicting pods to protect itself.

Requests, limits and usage

These three numbers are easy to confuse:

• Requests are what the scheduler reserves. A node is "full" when the requests of its pods reach its allocatable capacity, even if real usage is low.
• Limits are the ceiling. A container over its CPU limit is throttled; a container over its memory limit is killed.
• Usage is what the container actually consumes.

Watch usage against limits to predict throttling and OOM kills, and requests against allocatable capacity to predict pods that cannot be scheduled. Requests far above usage waste capacity; requests far below usage overcommit the node.

Restarts, CrashLoopBackOff and OOMKilled

A rising restart count is often the first sign of trouble. CrashLoopBackOff means a container keeps exiting and Kubernetes waits longer between each restart, up to five minutes. OOMKilled as the last termination reason, usually with exit code 137, means the container hit its memory limit or the node ran out of memory.

Pending pods, unavailable replicas and image pull errors

A pod stuck in Pending cannot be scheduled: not enough CPU or memory, a node selector or taint that nothing matches, or an unbound volume. Unavailable replicas mean a workload runs below its desired count. ErrImagePull and ImagePullBackOff point to a wrong tag, a missing registry credential or a registry that cannot be reached.

Warning events

Events explain most of the above in plain words: FailedScheduling, BackOff, Unhealthy (failed probes), Evicted, FailedMount. By default the API server keeps events for only one hour, so collect them if you want to see what happened overnight.

Checking a cluster with kubectl

Nodes and their conditions:

kubectl get nodes -o wide
kubectl describe node NODE_NAME

The Conditions and Allocated resources sections of describe node show pressure and how much of the node's capacity is already requested.

Current CPU and memory usage. These commands need metrics-server installed in the cluster:

kubectl top nodes
kubectl top pods -A --sort-by=memory

Pods that are not running, and workloads below their desired count:

kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded
kubectl get deployments,statefulsets,daemonsets -A

Why a pod is unhealthy. describe shows each container's state, last termination reason, restart count and recent events:

kubectl describe pod POD_NAME -n NAMESPACE

Recent warning events across the cluster:

kubectl get events -A --field-selector type=Warning --sort-by=.lastTimestamp

The logs of a container before its last restart, which usually hold the reason it crashed:

kubectl logs POD_NAME -n NAMESPACE -c CONTAINER_NAME --previous

What to alert on

Alert on symptoms that need a person, and let Kubernetes handle the rest. A single restart during a rollout is not worth a page; a pod restarting every few minutes is.

Add application alerts on error rate and latency for each service. They catch problems that leave every pod looking healthy.

Keep collectors read-only

Anything that watches a cluster runs with a service account, and that account is a target. Give it the least it needs:

• Grant get, list and watch only. No write verbs.
• List resources explicitly instead of using *.
• Leave out secrets and configmaps; monitoring rarely needs them.
• Prefer collectors that connect outbound, so nothing in the cluster listens for connections from outside.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: monitoring-read
rules:
  - apiGroups: [""]
    resources: ["nodes", "pods", "events", "namespaces"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets", "replicasets"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["metrics.k8s.io"]
    resources: ["nodes", "pods"]
    verbs: ["get", "list"]

Check what a service account can really do before you trust it:

kubectl auth can-i --list --as=system:serviceaccount:monitoring:collector
kubectl auth can-i get secrets --as=system:serviceaccount:monitoring:collector

The second command should answer no.

Monitor Kubernetes with Televane

Televane Kubernetes monitoring uses one small, read-only collector per cluster. You install it with one manifest and kubectl apply, and it runs as a single Deployment, with images for amd64 and arm64. Every 30 seconds it reports:

• Nodes: capacity, readiness, pressure, and CPU and memory use
• Workloads: desired against ready replicas for Deployments, StatefulSets and DaemonSets
• Pods: container states, restarts and owners
• Warning events from the last hour, with messages redacted

The collector never reads Secrets, ConfigMaps or pod environments, and it connects outbound only. A dry-run mode shows what it would send before you turn it on.

For alerting there are 14 ready-made Kubernetes measures, covering crash loops, OOM kills, image pull failures, restarts, pending, unready or failed pods, unavailable replicas, nodes not ready or under pressure, nodes short of CPU and memory, and a cluster that stops reporting. Alerts are evaluated every 15 seconds and related alerts are grouped into one incident.

For the application layer, send traces with OpenTelemetry: services in APM are linked to the Kubernetes workloads they run on. Televane charges for the data you send, with no per-host or per-container fees (see pricing). Create a free account to try it on your own cluster.

Frequently asked questions