Files
cluster-map/docs/README.md
T
Виталий ЧефановandClaude Opus 5.5 3cdc2583ee v0.8.0: problem badges, actions as the viewer, load, events, lock
Problems: cards, namespace and node boxes and the tree carry a yellow or
red badge, computed on the server from object state (crash loops, image
pulls, pending and unschedulable pods, readiness older than 2 minutes, a
stuck rollout, a failed CronJob run, Services without endpoints,
Ingresses to missing Services, node conditions), recent Warning events
(notable reasons at once, others after five repeats) and Deckhouse
ClusterAlerts mapped by their labels. A Problems panel lists them all.
Checked against the live cluster: upmeter's synthetic probe pods and
ingress-nginx's fakeservice Ingresses are excluded, they fail on purpose.

Actions: restart, scale, run a CronJob now, suspend/resume, delete a
pod, cordon/uncordon. They run with the viewer's bearer token from the
DexAuthenticator (sendAuthorizationHeader), buttons follow a
SelfSubjectAccessReview, the service account stays read-only. Cross-site
writes are refused by Sec-Fetch-Site/Origin/content type. Scale retries
on conflict: right after a restart a single Get+Update got 409.

Load (metrics.k8s.io against requests/limits/allocatable, per-pod table)
and an Events tab. Cards are locked until the padlock is opened; the map
exports to PNG; the page refreshes badges every minute in place.

Performance, measured on the live cluster:
- layout of big views 3-4x faster: equal-size cards with no edges are
  laid out on the grid ELK would produce (same sizes, checked against
  ELK on a frozen node view), skipping ~8 ms per ELK call;
- hubble-relay gets ~20% fewer flows: the to-overlay copy of pod-to-pod
  traffic is dropped at the agents; 331/331 directions unchanged;
- static files carry ETags, a reload revalidates instead of pulling
  ~2 MB of vendored scripts.

KUBECONFIG is honoured for running the binary on a workstation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-24 17:25:36 +03:00

99 lines
8.9 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
title: "The cluster-map module"
description: "Interactive topology map of the cluster with live traffic from Cilium Hubble."
webIfaces:
- name: cluster-map
---
The `cluster-map` module answers "where does this service live and what does it talk to" without a trip through `kubectl`. It shows the cluster as an inventory tree and a topology map: namespaces, workloads with their Services and Ingresses, the nodes they run on, real traffic between them, and the YAML and logs of each object.
## What is on the screen
- **Inventory tree** (left). Two views: *Namespaces* — namespace → object kind → objects; *Nodes* — node → namespace → workloads. Search matches names, images and Ingress hosts. The tree is keyboard-navigable.
- **Map** (centre). Only what is selected in the tree: one or several namespaces (Ctrl+click) or one node. Objects are compact cards inside namespace boxes, laid out left to right: Ingress → Service → workload.
- Solid arrows are configuration links: Ingress backends and Service selectors.
- Dashed animated arrows are real connections from Hubble over the chosen window (5 minutes, 15 minutes, 1 hour); red when Cilium dropped some of them. Peers from other namespaces appear as dashed cards in their own boxes.
- With more than 30 traffic directions on screen, lines stay faint until a card is hovered or selected.
- **Details** (right). Overview (a one-line verdict, the object's problems, its facts, CPU and memory against requests and limits, its pods), Links (configuration and traffic, clickable), live YAML, Events, container logs, and the actions the viewer is allowed to run.
- **Problems** (the button in the top bar). Everything that is wrong right now, most severe first, filterable to applications or system namespaces; a click opens the object.
Cards cannot be dragged until the padlock under the zoom buttons is opened, so a stray click does not rearrange the map. The last button saves the current map as a PNG picture.
Workloads managed by non-native controllers are included: Advanced DaemonSets of OpenKruise, database operator clusters, static pods, standalone Jobs. They are named the way Hubble names workloads, so their traffic attaches to the same cards.
## Problems and alerts
A card with a problem carries a badge: yellow `!` for a warning, red `!` for an error. Namespace boxes and cards, node boxes and the tree take the colour of the worst problem inside. Sources:
- **Object state**: containers in `CrashLoopBackOff`, `ImagePullBackOff` and similar, recent restarts and OOM kills, pods pending or unschedulable, workloads with no or not all replicas ready for more than 2 minutes, a stuck rollout, a CronJob whose last run failed, a Service with no ready pod, an Ingress routing to a Service that does not exist, a node that is not ready or is under memory, disk or PID pressure.
- **Warning events** of the last 30 minutes, folded per object and reason. Scheduling, volume and back-off failures badge the card at once; any other reason needs five repeats and is listed without a badge before that, so a single failed probe during a restart does not colour the map.
- **Prometheus alerts** that Deckhouse mirrors into `ClusterAlert` objects. An alert is attached by its labels (`deployment`, `statefulset`, `daemonset`, `pod`, `service`, `ingress`, `node`, `namespace`); one without an object or namespace goes to the cluster list. `severity_level` 1–3 is an error, 4–6 a warning, 7–9 information.
Information-level entries (a cordoned node, a single ordinary warning) are listed and never badge a card. Problems are recomputed on every map request and the open page refreshes them every minute.
## Actions
With `actions.enabled` (the default), the details panel offers:
| Object | Actions |
|---|---|
| Deployment, StatefulSet | restart (as `kubectl rollout restart`), scale to 0–100 replicas |
| DaemonSet | restart |
| CronJob | run now (as `kubectl create job --from=cronjob/…`), suspend, resume |
| Pod (in the Load table) | delete, so its controller starts a new one |
| Node (click the node box) | cordon, uncordon |
Every action asks for confirmation and runs **as the viewer**: the DexAuthenticator passes the user's own token to the module, the module sends the request to the API server with that token, and the API server checks the user's RBAC. A button is shown only when a `SelfSubjectAccessReview` says the user may do it. The module's service account never gets write permissions, and the API server's audit log records the user; the module also logs each action with the user's name.
Write requests from another site are refused: the module checks `Sec-Fetch-Site`, `Origin` and the JSON content type.
## How traffic is collected
Every Cilium agent keeps only a ring buffer of flows (4095 by default); at a few hundred flows per second this is about ten seconds of history. A one-off query "for the last 15 minutes" would silently return the last few seconds. The module therefore keeps a follow stream from Hubble Relay and aggregates it into per-minute buckets for the configured retention.
Counts are **distinct connections** (source and destination address and port), not Hubble events: one connection produces several events. For each direction the module also reports how many connections were opened (TCP SYN) inside the window and how many had a `DROPPED` verdict.
Kubelet probes and other traffic originating from node hosts, node-to-node traffic, Cilium health checks and DNS are filtered at the agents and never reach the module. On a typical cluster this is about three quarters of all flows and none of it maps to a workload. Set `traffic.collectInfraTraffic` to collect it anyway.
Pod-to-pod traffic between nodes is traced twice, as it leaves the sender's node (`to-overlay`) and as it reaches the receiving pod (`to-endpoint`). The first copy is also filtered at the agents when the receiver is a pod; this removes about a fifth of the remaining flows without changing any count.
The collector runs in one of three modes, see `traffic.mode`:
| Mode | Cost | Traffic windows |
|---|---|---|
| `Continuous` | stream always open; about 100–150m CPU on hubble-relay on a busy cluster | full as soon as the map is opened |
| `OnDemand` | stream only while someone uses the map | fill from the moment the map is opened |
| `Disabled` | none | configuration links only |
History is kept in memory: after a pod restart the windows fill again. The status line says how much of the window has actually been seen.
## Requirements
- Deckhouse Kubernetes Platform 1.72 or newer.
- For traffic: the `cni-cilium` and `cilium-hubble` modules. Without `cilium-hubble` the module works and shows configuration links only.
- For access control: the `user-authn` module, or any external authentication (`auth.externalAuthentication`).
## Security
The web interface reads the cluster through the module's own service account (`d8:cluster-map` ClusterRole): `get`, `list`, `watch` on namespaces, pods, services, nodes, events, deployments, statefulsets, daemonsets, cronjobs, ingresses and Deckhouse `clusteralerts`, `get` and `list` on `metrics.k8s.io` pods and nodes, and `get` on `pods/log`. It never reads Secrets or ConfigMaps and has no write verbs. Changes go through [actions](#actions), which use the viewer's own identity.
This means **anyone admitted to the web interface sees every namespace and can read container logs**, whatever their own RBAC permissions are; what they can change is limited by their own RBAC. Restrict access with `auth.allowedUserGroups` (and optionally `auth.allowedUserEmails` and `auth.whitelistSourceRanges`).
The frontend is served with a strict Content Security Policy: only its own scripts, styles and API, no inline code, no framing.
To verify the Hubble Relay certificate the module needs the public certificate of the hubble CA. A module hook reads it from the `hubble-ca-secret` Secret in `d8-cni-cilium`; the jq filter keeps only `ca.crt`, the CA key never reaches the module values.
## Metrics and alerts
The pod exposes Prometheus metrics on port `9090` (`cluster_map_*`: Hubble stream state, flows processed, tracked connections, API latency) and ships two alerts:
- `ClusterMapHubbleDisconnected` — no flows from Hubble Relay for 15 minutes while the collector is supposed to be running.
- `ClusterMapTrackedTuplesHigh` — the collector holds more than 2 million connections; lower the retention or switch to `OnDemand`.
## Limitations
- One replica: traffic history lives in the process. A second replica would keep its own history and double the load on hubble-relay.
- Reading is all-or-nothing, as described in [Security](#security). Showing each user only what their RBAC allows to read is planned; actions already follow the user's RBAC.
- Alerts need the `prometheus` module, which creates `ClusterAlert` objects. Without it the map shows problems from object state and events only.
- Traffic is L3/L4. HTTP-level details (paths, status codes) need Cilium L7 visibility and are not shown yet.