diff --git a/changelog.yaml b/changelog.yaml index 1ea846b..f3b7dde 100644 --- a/changelog.yaml +++ b/changelog.yaml @@ -1,3 +1,8 @@ +v0.11.0: + features: + - "HTTP details where Cilium parses L7 (a CiliumNetworkPolicy with an http rule on the port): requests, responses by class, p50 and p95 latency and the ten busiest paths on the link, IDs in paths grouped as :id. Links with 5xx responses are drawn in the warning colour. Headers are never requested from Hubble. The docs have an example policy that turns this on for one service without restricting it." + fixes: + - "Traffic through the L7 proxy is counted once: Envoy's own connection to the pod no longer shows as a link from world, and the pod's answers to Envoy no longer add one link to the node per request. A pod that reached itself through its own Service is not shown as a link from world either." v0.10.0: features: - "Per-user visibility (auth.visibility: ViewerRBAC, the default): each viewer sees the namespaces where their own RBAC lets them list pods, or the whole cluster if they can list pods everywhere. Logs also need get pods/log, node cards list nodes; traffic peers in hidden namespaces are shown as hidden. Checked with SelfSubjectAccessReviews as the viewer, cached for a minute. auth.visibility: Everything keeps the previous behaviour." diff --git a/docs/EXAMPLES.md b/docs/EXAMPLES.md index 75c4c9e..fd434a0 100644 --- a/docs/EXAMPLES.md +++ b/docs/EXAMPLES.md @@ -50,4 +50,33 @@ spec: enabled: false ``` -No action buttons, and the user's token is no longer passed to the module. +No action buttons. With `auth.visibility: Everything` as well, the user's token is no longer passed to the module. + +## HTTP details for one service + +This policy selects the pods of the `web` Deployment and sends port 8080 through Envoy without restricting anything: `enableDefaultDeny: false` keeps all other traffic allowed, and the empty `http` rule matches every request. + +```yaml +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: http-visibility + namespace: shop +spec: + endpointSelector: + matchLabels: + app: web + enableDefaultDeny: + ingress: false + egress: false + ingress: + - toPorts: + - ports: + - port: "8080" + protocol: TCP + rules: + http: + - {} +``` + +Delete the policy to go back to L3/L4. Before applying it to a service, read the notes in [HTTP details](README.md#http-details). diff --git a/docs/EXAMPLES_RU.md b/docs/EXAMPLES_RU.md index 6e9a688..4e38f18 100644 --- a/docs/EXAMPLES_RU.md +++ b/docs/EXAMPLES_RU.md @@ -50,4 +50,33 @@ spec: enabled: false ``` -Кнопок действий нет, и токен пользователя модулю больше не передаётся. +Кнопок действий нет. Если к тому же задано `auth.visibility: Everything`, токен пользователя модулю больше не передаётся. + +## Подробности HTTP для одного сервиса + +Политика выбирает поды Deployment `web` и пускает порт 8080 через Envoy, ничего не ограничивая: `enableDefaultDeny: false` оставляет весь остальной трафик разрешённым, а пустое правило `http` подходит под любой запрос. + +```yaml +apiVersion: cilium.io/v2 +kind: CiliumNetworkPolicy +metadata: + name: http-visibility + namespace: shop +spec: + endpointSelector: + matchLabels: + app: web + enableDefaultDeny: + ingress: false + egress: false + ingress: + - toPorts: + - ports: + - port: "8080" + protocol: TCP + rules: + http: + - {} +``` + +Чтобы вернуться к L3/L4, удалите политику. Прежде чем включать её для сервиса, прочитайте замечания в разделе [Подробности HTTP](README_RU.md#подробности-http). diff --git a/docs/README.md b/docs/README.md index fd9d556..79ec9ab 100644 --- a/docs/README.md +++ b/docs/README.md @@ -73,6 +73,16 @@ The collector runs in one of three modes, see `traffic.mode`: History is kept in memory: after a pod restart the windows fill again. The status line says how much of the window has actually been seen. +## HTTP details + +Where Cilium parses HTTP on a port, the map adds to that link the number of requests, responses by class (2xx, 3xx, 4xx, 5xx), p50 and p95 latency and the ten busiest paths. A link with 5xx responses is drawn in the warning colour. Paths are grouped: the query string is dropped and segments that look like IDs (numbers, UUIDs, long hex strings) become `:id`, so `/api/users/42?x=1` and `/api/users/43` are one row `/api/users/:id`. + +Cilium parses HTTP only on ports named in an L7 rule of a `CiliumNetworkPolicy`; the module does not create such policies. To enable it for one service without restricting its traffic, see [the example](EXAMPLES.md#http-details-for-one-service). Keep in mind: + +- the connections to that port go through the Envoy proxy on the node: a fraction of a millisecond per request and CPU of the Cilium agent in proportion to the request rate; +- the port must speak plain HTTP/1.x or HTTP/2 (h2c): Envoy cannot see inside TLS, so put the rule on the backend port behind the Ingress, not on 443. gRPC is parsed, but its status is in the trailers and every call shows as 200; a gRPC stream or a WebSocket is one request, not a series of messages; +- request and response headers reach Hubble, including `Authorization` and cookies. The module asks Hubble for the method, path, status and latency only; the headers are not read, stored or shown. Other Hubble clients (the Hubble UI, `hubble observe`) do see them. + ## Requirements - Deckhouse Kubernetes Platform 1.72 or newer. @@ -107,4 +117,4 @@ The pod exposes Prometheus metrics on port `9090` (`cluster_map_*`: Hubble strea - One replica: traffic history lives in the process. A second replica would keep its own history and double the load on hubble-relay. - Visibility is by namespace: a user who can list pods in a namespace sees all its cards, including the manifests of its Deployments and Services, even if their role would not let them read those objects one by one. - Alerts need the `prometheus` module, which creates `ClusterAlert` objects. Without it the map shows problems from object state and events only. -- Traffic is L3/L4. HTTP-level details (paths, status codes) need Cilium L7 visibility and are not shown yet. +- HTTP details only where a CiliumNetworkPolicy turns on L7 parsing, see [HTTP details](#http-details). Other protocols are shown at L3/L4. diff --git a/docs/README_RU.md b/docs/README_RU.md index b7ca731..72dffac 100644 --- a/docs/README_RU.md +++ b/docs/README_RU.md @@ -73,6 +73,16 @@ Workload'ы под нестандартными контроллерами то История хранится в памяти: после рестарта пода окна заполняются заново. Строка статуса пишет, сколько из окна реально видно. +## Подробности HTTP + +Там, где Cilium разбирает HTTP на порту, карта добавляет к связи число запросов, ответы по классам (2xx, 3xx, 4xx, 5xx), задержку p50 и p95 и десять самых нагруженных путей. Связь, на которой были ответы 5xx, рисуется цветом предупреждения. Пути группируются: строка запроса отбрасывается, а сегменты, похожие на идентификаторы (числа, UUID, длинные hex-строки), заменяются на `:id` — `/api/users/42?x=1` и `/api/users/43` попадают в одну строку `/api/users/:id`. + +Cilium разбирает HTTP только на портах, упомянутых в L7-правиле `CiliumNetworkPolicy`; сам модуль таких политик не создаёт. Как включить это для одного сервиса, не ограничивая его трафик, — в [примере](EXAMPLES_RU.md#подробности-http-для-одного-сервиса). Учтите: + +- соединения на этот порт идут через прокси Envoy на ноде: доли миллисекунды на запрос и CPU агента Cilium пропорционально числу запросов; +- порт должен говорить на обычном HTTP/1.x или HTTP/2 (h2c): внутрь TLS Envoy не заглядывает, поэтому правило ставится на порт бэкенда за Ingress, а не на 443. gRPC разбирается, но его статус лежит в трейлерах, и каждый вызов виден как 200; поток gRPC или WebSocket — это один запрос, а не последовательность сообщений; +- заголовки запроса и ответа попадают в Hubble, включая `Authorization` и cookie. Модуль запрашивает у Hubble только метод, путь, код и задержку; заголовки он не читает, не хранит и не показывает. Другие клиенты Hubble (Hubble UI, `hubble observe`) их видят. + ## Требования - Deckhouse Kubernetes Platform 1.72 или новее. @@ -107,4 +117,4 @@ Workload'ы под нестандартными контроллерами то - Одна реплика: история трафика живёт в процессе. Вторая реплика держала бы свою историю и удвоила бы нагрузку на hubble-relay. - Видимость — по namespace: пользователь, который может смотреть поды в namespace, видит все его карточки, включая манифесты Deployment и Service, даже если его роль не дала бы прочитать эти объекты по отдельности. - Для алертов нужен модуль `prometheus`, он создаёт объекты `ClusterAlert`. Без него карта показывает проблемы только по состоянию объектов и событиям. -- Трафик на уровне L3/L4. Подробности HTTP (пути, коды ответа) требуют L7-видимости Cilium и пока не показываются. +- Подробности HTTP — только там, где CiliumNetworkPolicy включает разбор L7, см. [Подробности HTTP](#подробности-http). Остальные протоколы показываются на уровне L3/L4. diff --git a/images/cluster-map/src/internal/traffic/collector.go b/images/cluster-map/src/internal/traffic/collector.go index f6ab293..1cf6410 100644 --- a/images/cluster-map/src/internal/traffic/collector.go +++ b/images/cluster-map/src/internal/traffic/collector.go @@ -28,8 +28,10 @@ import ( "context" "hash/fnv" "log/slog" + "slices" "sort" "strconv" + "strings" "sync" "time" @@ -71,13 +73,17 @@ var InfraFilters = []*flowpb.FlowFilter{ } // Cilium datapath codes (pkg/monitor/api): message type "trace" and its -// observation point "to-overlay". +// observation point "to-overlay", and the L7 access log type. const ( - monitorTrace = 4 - traceToOverlay = 4 + monitorTrace = 4 + traceToOverlay = 4 + monitorAccessLog = 129 ) -var flowFields = []string{"time", "verdict", "IP", "l4", "source", "destination", "source_names", "destination_names", "node_name"} +// flowFields: from L7 events only what the HTTP statistics use -- never +// headers, which carry cookies and Authorization. +var flowFields = []string{"time", "verdict", "IP", "l4", "source", "destination", "source_names", "destination_names", "node_name", + "l7.type", "l7.latency_ns", "l7.http.code", "l7.http.method", "l7.http.url"} // Endpoint is one side of a traffic edge: a workload, a pod or an address. type Endpoint struct { @@ -96,6 +102,9 @@ type Edge struct { New int `json:"new"` // of those, opened by TCP SYN inside the window Dropped int `json:"dropped"` // distinct 5-tuples with a DROPPED verdict Last string `json:"last"` + // HTTP is set where Hubble sees L7 on this port (an L7 visibility + // policy sends it through Envoy). + HTTP *HTTPStats `json:"http,omitempty"` } // Result is the /api/flows response. @@ -128,6 +137,7 @@ type bucketEdge struct { type bucket struct { minute int64 edges map[edgeKey]*bucketEdge + http map[edgeKey]*httpAgg // only where L7 visibility is on } // Collector keeps minute buckets of distinct connections for the @@ -227,8 +237,10 @@ func (c *Collector) loop(ctx context.Context) { Follow: true, // Backfill whatever the ring buffers still hold; tuples seen twice // after a reconnect collapse in the per-minute sets. - Since: timestamppb.New(c.now().Add(-time.Minute)), - Whitelist: []*flowpb.FlowFilter{{Reply: []bool{false}}}, + Since: timestamppb.New(c.now().Add(-time.Minute)), + // requests and connections without replies, plus every L7 event: + // the HTTP status and latency are in the RESPONSE, a reply + Whitelist: []*flowpb.FlowFilter{{Reply: []bool{false}}, {EventType: []*flowpb.EventTypeFilter{{Type: monitorAccessLog}}}}, } if !c.cfg.CollectInfra { req.Blacklist = InfraFilters @@ -290,12 +302,20 @@ func (c *Collector) setErr(err error) { // Add records one flow. Exported for tests and for replaying captures. func (c *Collector) Add(f *flowpb.Flow) { + if proxiedLeg(f) { + metrics.FlowsProcessed.Inc() + return + } src := c.endpoint(f.GetSource(), f.GetSourceNames(), f.GetNodeName()) dst := c.endpoint(f.GetDestination(), f.GetDestinationNames(), f.GetNodeName()) metrics.FlowsProcessed.Inc() if src.ID == dst.ID { return } + // Not f.GetL7() != nil: with the field mask asking for l7.* subfields, + // Hubble returns an empty L7 message on every flow, L4 included; only + // a set type says the event is an L7 one. + l7 := f.GetL7().GetType() != flowpb.L7FlowType_UNKNOWN_L7_TYPE port, tuple, syn := l4Info(f) ts := c.now() if t := f.GetTime(); t != nil { @@ -321,6 +341,11 @@ func (c *Collector) Add(f *flowpb.Flow) { } c.endpoints[src.ID] = src c.endpoints[dst.ID] = dst + if l7 { + // a RESPONSE goes server -> client; statistics are kept client -> server + c.addHTTP(b, f, dst, src, ts) + return + } k := edgeKey{src.ID, dst.ID, port} e := b.edges[k] if e == nil { @@ -355,6 +380,9 @@ func (c *Collector) pruneEndpoints() { used[k.src], used[k.dst] = true, true tuples += len(e.tuples) } + for k := range b.http { + used[k.src], used[k.dst] = true, true + } } for id := range c.endpoints { if !used[id] { @@ -390,10 +418,16 @@ func (c *Collector) Query(namespaces map[string]bool, window time.Duration) Resu touches := func(id string) bool { return namespaces[c.endpoints[id].Namespace] } merged := map[edgeKey]*bucketEdge{} + httpAggs := map[edgeKey][]*httpAgg{} for _, b := range c.buckets { if b == nil || b.minute < fromMinute { continue } + for k, a := range b.http { + if touches(k.src) || touches(k.dst) { + httpAggs[k] = append(httpAggs[k], a) + } + } for k, e := range b.edges { if !touches(k.src) && !touches(k.dst) { continue @@ -420,9 +454,23 @@ func (c *Collector) Query(namespaces map[string]bool, window time.Duration) Resu e.Dropped++ } } + if aggs, ok := httpAggs[k]; ok { + e.HTTP = httpStats(aggs) + delete(httpAggs, k) + } res.Edges = append(res.Edges, e) eps[k.src], eps[k.dst] = true, true } + // HTTP seen without a matching connection record (the proxy observed + // the requests, the L4 trace for that exact port did not reach us) + for k, aggs := range httpAggs { + last := int64(0) + for _, a := range aggs { + last = max(last, a.last) + } + res.Edges = append(res.Edges, Edge{Source: k.src, Target: k.dst, Port: k.port, Last: time.Unix(last, 0).UTC().Format(time.RFC3339), HTTP: httpStats(aggs)}) + eps[k.src], eps[k.dst] = true, true + } for id := range eps { res.Endpoints = append(res.Endpoints, c.endpoints[id]) } @@ -468,6 +516,44 @@ func l4Info(f *flowpb.Flow) (port, tuple string, syn bool) { return "?", "?", false } +// ephemeralPorts is where Linux picks client ports (ip_local_port_range); +// NodePorts (30000-32767) stay below it. +const ephemeralPorts = 32768 + +// proxiedLeg: with L7 visibility Envoy re-opens each connection to the pod +// from the node, and both halves of that inner connection reach us as +// non-replies: +// - Envoy -> pod carries the client's security identity (its labels) +// but no pod name, so it cannot be attributed and would show as +// "world" (a pod that reached itself through its own Service looks +// the same: Cilium rewrites the source to 169.254.42.1); +// - the pod's answers to Envoy go to the node's host on an ephemeral +// port, and would show as one "pod -> node" edge per request. +// +// The connection itself is already counted on the client -> proxy leg. +// A pod that dials a host does so on a service port, never an ephemeral +// one (checked on the live cluster: API server 6443, kubelet 10250, +// exporters 42xx, Hubble 4244 all stay). +func proxiedLeg(f *flowpb.Flow) bool { + if f.GetL7().GetType() != flowpb.L7FlowType_UNKNOWN_L7_TYPE { + return false + } + s, d := f.GetSource(), f.GetDestination() + if s.GetPodName() == "" { + for _, l := range s.GetLabels() { + if strings.HasPrefix(l, "k8s:io.kubernetes.pod.namespace=") { + return true + } + } + } + if s.GetPodName() != "" && slices.Contains(d.GetLabels(), "reserved:host") { + if t := f.GetL4().GetTCP(); t != nil && t.GetDestinationPort() >= ephemeralPorts && t.GetSourcePort() < ephemeralPorts { + return true + } + } + return false +} + // Restricted stands for every peer in a namespace the viewer cannot see: // the connection is shown, the workload behind it is not. var Restricted = Endpoint{ID: "ext/restricted", Label: "restricted", Kind: "external"} @@ -508,6 +594,7 @@ func Restrict(res Result, visible func(string) bool) Result { m.New += e.New m.Dropped += e.Dropped m.Last = max(m.Last, e.Last) + m.HTTP = mergeHTTP(m.HTTP, e.HTTP) continue } c := e diff --git a/images/cluster-map/src/internal/traffic/collector_test.go b/images/cluster-map/src/internal/traffic/collector_test.go index 0fad0a7..394494d 100644 --- a/images/cluster-map/src/internal/traffic/collector_test.go +++ b/images/cluster-map/src/internal/traffic/collector_test.go @@ -18,12 +18,14 @@ package traffic import ( "encoding/json" + "strconv" "strings" "testing" "time" flowpb "github.com/cilium/cilium/api/v1/flow" "google.golang.org/protobuf/types/known/timestamppb" + "google.golang.org/protobuf/types/known/wrapperspb" "git.project-chimera.ru/fanchji/cluster-map/internal/config" ) @@ -208,3 +210,102 @@ func TestRestrictHidesForeignPeers(t *testing.T) { t.Errorf("two hidden peers on one port merge into one restricted edge: %+v", e) } } + +func TestPathTemplate(t *testing.T) { + for in, want := range map[string]string{ + "http://web/": "/", + "http://web/api/42?token=secret": "/api/:id", + "/users/7f3a9c2e1b/avatar": "/users/:id/avatar", + "https://h/o/3fa85f64-5717-4562-b3fc-2c963f66afa6/x": "/o/:id/x", + "/static/app.js": "/static/app.js", + } { + if got := PathTemplate(in); got != want { + t.Errorf("%q: %q, want %q", in, got, want) + } + } +} + +func httpResponse(at time.Time, code uint32, url string, latency time.Duration) *flowpb.Flow { + return &flowpb.Flow{ + Time: timestamppb.New(at), + Verdict: flowpb.Verdict_FORWARDED, + IP: &flowpb.IP{Source: "10.0.0.2", Destination: "10.0.0.1"}, + L4: &flowpb.Layer4{Protocol: &flowpb.Layer4_TCP{TCP: &flowpb.TCP{SourcePort: 8080, DestinationPort: 40000}}}, + Source: &flowpb.Endpoint{Namespace: "shop", PodName: "web"}, + Destination: &flowpb.Endpoint{Namespace: "shop", PodName: "client"}, + IsReply: &wrapperspb.BoolValue{Value: true}, + L7: &flowpb.Layer7{Type: flowpb.L7FlowType_RESPONSE, LatencyNs: uint64(latency.Nanoseconds()), //nolint:gosec // test durations are small and positive + Record: &flowpb.Layer7_Http{Http: &flowpb.HTTP{Code: code, Method: "GET", Url: url}}}, + } +} + +func TestHTTPStats(t *testing.T) { + now := t0.Add(2 * time.Minute) + c := newTestCollector(&now) + for i := 0; i < 18; i++ { + c.Add(httpResponse(t0, 200, "http://web/api/"+strconv.Itoa(i), 3*time.Millisecond)) + } + c.Add(httpResponse(t0, 503, "http://web/api/1", 400*time.Millisecond)) + c.Add(httpResponse(t0, 404, "http://web/missing", time.Millisecond)) + // a REQUEST event carries no status and is not counted + req := httpResponse(t0, 0, "http://web/", 0) + req.L7.Type = flowpb.L7FlowType_REQUEST + c.Add(req) + + res := c.Query(map[string]bool{"shop": true}, 5*time.Minute) + if len(res.Edges) != 1 { + t.Fatalf("edges: %+v", res.Edges) + } + e := res.Edges[0] + if e.Source != "wl/shop/deployment/client" || e.Target != "wl/shop/deployment/web" || e.Port != "8080/TCP" { + t.Errorf("a response must be recorded in the request's direction: %+v", e) + } + h := e.HTTP + if h == nil || h.Requests != 20 || h.Status["2xx"] != 18 || h.Status["5xx"] != 1 || h.Status["4xx"] != 1 { + t.Fatalf("http: %+v", h) + } + if h.P50Ms != 5 || h.P95Ms != 5 { + t.Errorf("p50/p95: %v/%v", h.P50Ms, h.P95Ms) + } + // 19 requests on the path: the 95th percentile is the 19th, the slow one + if top := h.Paths[0]; top.Path != "/api/:id" || top.Count != 19 || top.Errors != 1 || top.P95Ms != 500 { + t.Errorf("top path: %+v", top) + } +} + +// With the field mask, Hubble sends an empty L7 message on L4 flows too; +// those must still count as connections (caught live: every edge in the +// cluster lost its connection count). +func TestMaskedL4FlowWithEmptyL7Counts(t *testing.T) { + now := t0.Add(time.Minute) + c := newTestCollector(&now) + f := tcpFlow(t0, "app", "web", "db", "pg", 40000, true, flowpb.Verdict_FORWARDED) + f.L7 = &flowpb.Layer7{} + c.Add(f) + res := c.Query(map[string]bool{"app": true}, 5*time.Minute) + if len(res.Edges) != 1 || res.Edges[0].Connections != 1 || res.Edges[0].HTTP != nil { + t.Fatalf("edges: %+v", res.Edges) + } +} + +func TestProxiedLegIsSkipped(t *testing.T) { + now := t0.Add(time.Minute) + c := newTestCollector(&now) + leg := tcpFlow(t0, "", "", "shop", "web", 40000, true, flowpb.Verdict_FORWARDED) + leg.Source = &flowpb.Endpoint{Labels: []string{"k8s:app=client", "k8s:io.kubernetes.pod.namespace=shop"}} + c.Add(leg) + // the pod answering Envoy: to the node, on an ephemeral port + answer := tcpFlow(t0, "shop", "web", "", "", 8080, false, flowpb.Verdict_FORWARDED) + answer.Destination = &flowpb.Endpoint{Labels: []string{"reserved:host"}} + answer.L4 = &flowpb.Layer4{Protocol: &flowpb.Layer4_TCP{TCP: &flowpb.TCP{SourcePort: 8080, DestinationPort: 49726}}} + c.Add(answer) + // a pod dialling the API server on the host: a real connection + api := tcpFlow(t0, "shop", "web", "", "", 40001, true, flowpb.Verdict_FORWARDED) + api.Destination = &flowpb.Endpoint{Labels: []string{"reserved:host"}} + api.L4 = &flowpb.Layer4{Protocol: &flowpb.Layer4_TCP{TCP: &flowpb.TCP{SourcePort: 40001, DestinationPort: 6443, Flags: &flowpb.TCPFlags{SYN: true}}}} + c.Add(api) + res := c.Query(map[string]bool{"shop": true}, 5*time.Minute) + if len(res.Edges) != 1 || res.Edges[0].Port != "6443/TCP" { + t.Errorf("want only the pod -> API server edge, got %+v", res.Edges) + } +} diff --git a/images/cluster-map/src/internal/traffic/http.go b/images/cluster-map/src/internal/traffic/http.go new file mode 100644 index 0000000..ad12f19 --- /dev/null +++ b/images/cluster-map/src/internal/traffic/http.go @@ -0,0 +1,259 @@ +/* +Copyright 2026 The cluster-map Authors + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package traffic + +import ( + "net/url" + "regexp" + "sort" + "strconv" + "strings" + "time" + + flowpb "github.com/cilium/cilium/api/v1/flow" +) + +// HTTP statistics come from Hubble's L7 events, which exist only where a +// CiliumNetworkPolicy sends a port through Envoy (see docs: "HTTP +// details"). Only RESPONSE events are read: they carry the method, URL, +// status and latency together. Headers are never stored (and not even +// requested, see flowFields); the query string is cut off the URL, since +// tokens travel there too. + +// latencyBoundsMs are the histogram buckets; percentiles are reported as +// the upper bound of the bucket they fall in. +var latencyBoundsMs = []float64{1, 2, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 5000, 10000} + +// maxPathsPerEdge bounds memory per edge per minute; beyond it requests +// still count, the path breakdown just stops growing. +const maxPathsPerEdge = 64 + +type hist [14]int // len(latencyBoundsMs)+1 + +func (h *hist) add(ms float64) { + i := sort.SearchFloat64s(latencyBoundsMs, ms) + h[i]++ +} + +func (h *hist) merge(o *hist) { + for i := range h { + h[i] += o[i] + } +} + +// quantile returns the upper bound (ms) of the bucket holding the q-th +// request; for the overflow bucket, the largest bound. +func (h *hist) quantile(q float64) float64 { + total := 0 + for _, n := range h { + total += n + } + if total == 0 { + return 0 + } + target, seen := int(q*float64(total)+0.999999), 0 + for i, n := range h { + seen += n + if seen >= target { + if i < len(latencyBoundsMs) { + return latencyBoundsMs[i] + } + return latencyBoundsMs[len(latencyBoundsMs)-1] + } + } + return latencyBoundsMs[len(latencyBoundsMs)-1] +} + +type pathKey struct{ method, path string } + +type pathAgg struct { + count, errors int + lat hist +} + +type httpAgg struct { + status [6]int // index = first digit of the code, 0 for anything else + lat hist + paths map[pathKey]*pathAgg + last int64 +} + +// HTTPStats is the HTTP part of a traffic edge. +type HTTPStats struct { + Requests int `json:"requests"` + Status map[string]int `json:"status"` // "2xx", "3xx", "4xx", "5xx", "other" + P50Ms float64 `json:"p50Ms"` + P95Ms float64 `json:"p95Ms"` + Paths []HTTPPath `json:"paths"` + lat hist +} + +// HTTPPath is one method and path template on an edge. +type HTTPPath struct { + Method string `json:"method"` + Path string `json:"path"` + Count int `json:"count"` + Errors int `json:"errors"` // 5xx + P95Ms float64 `json:"p95Ms"` +} + +var idSegment = regexp.MustCompile(`^([0-9]+|[0-9a-fA-F]{8,}|[0-9a-fA-F]{8}-[0-9a-fA-F-]{27,}|.{41,})$`) + +// PathTemplate reduces a URL to a low-cardinality path: no scheme, host +// or query, and segments that look like IDs (numbers, hex, UUIDs, long +// tokens) replaced by ":id". +func PathTemplate(raw string) string { + p := raw + if u, err := url.Parse(raw); err == nil { + p = u.Path + } else if i := strings.IndexAny(p, "?#"); i >= 0 { + p = p[:i] + } + if p == "" { + return "/" + } + segs := strings.Split(p, "/") + for i, s := range segs { + if s != "" && idSegment.MatchString(s) { + segs[i] = ":id" + } + } + out := strings.Join(segs, "/") + if len(out) > 100 { + out = out[:100] + "…" + } + return out +} + +// addHTTP records one L7 RESPONSE event, turned around into the request's +// direction (client -> server). Called with c.mu held. +func (c *Collector) addHTTP(b *bucket, f *flowpb.Flow, client, server Endpoint, ts time.Time) { + h := f.GetL7().GetHttp() + if h == nil || f.GetL7().GetType() != flowpb.L7FlowType_RESPONSE { + return + } + port := "?" + if t := f.GetL4().GetTCP(); t != nil { + port = strconv.FormatUint(uint64(t.GetSourcePort()), 10) + "/TCP" // the server's side of the response + } + k := edgeKey{client.ID, server.ID, port} + if b.http == nil { + b.http = map[edgeKey]*httpAgg{} + } + a := b.http[k] + if a == nil { + a = &httpAgg{paths: map[pathKey]*pathAgg{}} + b.http[k] = a + } + code := int(h.GetCode()) + class := 0 + if code >= 100 && code < 600 { + class = code / 100 + } + a.status[class]++ + ms := float64(f.GetL7().GetLatencyNs()) / 1e6 + a.lat.add(ms) + a.last = max(a.last, ts.Unix()) + pk := pathKey{h.GetMethod(), PathTemplate(h.GetUrl())} + pa := a.paths[pk] + if pa == nil { + if len(a.paths) >= maxPathsPerEdge { + return + } + pa = &pathAgg{} + a.paths[pk] = pa + } + pa.count++ + if class == 5 { + pa.errors++ + } + pa.lat.add(ms) +} + +// httpStats merges an edge's per-minute aggregates into its report. +func httpStats(aggs []*httpAgg) *HTTPStats { + var status [6]int + var lat hist + paths := map[pathKey]*pathAgg{} + for _, a := range aggs { + for i, n := range a.status { + status[i] += n + } + lat.merge(&a.lat) + for k, p := range a.paths { + m := paths[k] + if m == nil { + m = &pathAgg{} + paths[k] = m + } + m.count += p.count + m.errors += p.errors + m.lat.merge(&p.lat) + } + } + s := &HTTPStats{Status: map[string]int{}, lat: lat} + for i, n := range status { + if n == 0 { + continue + } + s.Requests += n + if i == 0 { + s.Status["other"] += n + } else { + s.Status[strconv.Itoa(i)+"xx"] = n + } + } + s.P50Ms, s.P95Ms = lat.quantile(0.5), lat.quantile(0.95) + for k, p := range paths { + s.Paths = append(s.Paths, HTTPPath{Method: k.method, Path: k.path, Count: p.count, Errors: p.errors, P95Ms: p.lat.quantile(0.95)}) + } + sort.Slice(s.Paths, func(i, j int) bool { + if s.Paths[i].Count != s.Paths[j].Count { + return s.Paths[i].Count > s.Paths[j].Count + } + return s.Paths[i].Method+s.Paths[i].Path < s.Paths[j].Method+s.Paths[j].Path + }) + if len(s.Paths) > 10 { + s.Paths = s.Paths[:10] + } + return s +} + +// mergeHTTP folds b into a, for edges merged by Restrict. +func mergeHTTP(a, b *HTTPStats) *HTTPStats { + if a == nil { + return b + } + if b == nil { + return a + } + out := &HTTPStats{Requests: a.Requests + b.Requests, Status: map[string]int{}, lat: a.lat} + out.lat.merge(&b.lat) + for k, n := range a.Status { + out.Status[k] += n + } + for k, n := range b.Status { + out.Status[k] += n + } + out.P50Ms, out.P95Ms = out.lat.quantile(0.5), out.lat.quantile(0.95) + out.Paths = append(append([]HTTPPath{}, a.Paths...), b.Paths...) + sort.Slice(out.Paths, func(i, j int) bool { return out.Paths[i].Count > out.Paths[j].Count }) + if len(out.Paths) > 10 { + out.Paths = out.Paths[:10] + } + return out +} diff --git a/images/cluster-map/src/web/static/app.css b/images/cluster-map/src/web/static/app.css index 1cf600b..acf822e 100644 --- a/images/cluster-map/src/web/static/app.css +++ b/images/cluster-map/src/web/static/app.css @@ -149,6 +149,7 @@ select, input[type="search"] { .legend .line { width: 22px; height: 0; border-top: 2px solid var(--muted); } .legend .line.traffic { border-top: 3px dashed var(--accent); } .legend .line.drop { border-top: 3px dashed var(--error); } +.legend .line.http5xx { border-top: 3px dashed var(--warn); } .legend .line.config { border-top: 1.5px solid var(--muted); } /* details panel */ @@ -183,11 +184,12 @@ select, input[type="search"] { .section-title { font-size: 0.72rem; text-transform: uppercase; letter-spacing: 0.05em; color: var(--muted); margin: 1rem 0 0.4rem; } .link-list { list-style: none; margin: 0; padding: 0; display: flex; flex-direction: column; gap: 0.2rem; } .link-list button { - width: 100%; display: flex; align-items: center; gap: 0.5rem; text-align: left; + width: 100%; display: flex; flex-wrap: wrap; align-items: center; gap: 0.2rem 0.5rem; text-align: left; background: var(--panel-2); border: 1px solid var(--line); border-radius: 6px; padding: 0.35rem 0.5rem; color: var(--text); font: inherit; cursor: pointer; } .link-list button:hover { border-color: var(--accent); } +.link-list button > span:nth-child(2) { flex: 1 1 10rem; min-width: 0; overflow-wrap: anywhere; } .link-list .meta { margin-left: auto; color: var(--muted); font-size: 0.72rem; font-variant-numeric: tabular-nums; white-space: nowrap; } .link-list .drop { color: var(--error); } .code { @@ -290,6 +292,10 @@ table.pods .warn-text { color: var(--warn); font-weight: 600; } table.vulns a { color: var(--accent); } .shield-legend { display: inline-block; width: 10px; height: 12px; background: var(--error); clip-path: polygon(50% 0, 100% 15%, 100% 50%, 50% 100%, 0 50%, 0 15%); } +/* HTTP on traffic links */ +.link-list .http-li { list-style: none; margin: -0.1rem 0 0.3rem 1.4rem; } +table.http-paths { margin-top: 0; font-size: 0.72rem; } + /* events */ .event-list { list-style: none; margin: 0; padding: 0; display: flex; flex-direction: column; gap: 0.35rem; } .event-list li { border-left: 3px solid var(--line); padding: 0.1rem 0 0.1rem 0.5rem; } diff --git a/images/cluster-map/src/web/static/app.js b/images/cluster-map/src/web/static/app.js index 4689098..a54d0e7 100644 --- a/images/cluster-map/src/web/static/app.js +++ b/images/cluster-map/src/web/static/app.js @@ -523,6 +523,7 @@ function mapStyle() { width: 'data(width)', opacity: 0.9, } }, { selector: 'edge.drop', style: { 'line-color': T.error, 'target-arrow-color': T.error } }, + { selector: 'edge.http5xx', style: { 'line-color': T.warn, 'target-arrow-color': T.warn } }, { selector: 'edge.labelled, edge:selected', style: { label: 'data(label)', 'font-size': 10, 'font-family': FONT, color: T.text, 'text-rotation': 'autorotate', 'text-background-color': T.panel, 'text-background-opacity': 0.92, 'text-background-padding': 2, 'text-background-shape': 'round-rectangle', @@ -582,15 +583,33 @@ function aggregateFlows(flows) { for (const e of flows.edges) { const key = e.source + '→' + e.target; let p = pairs.get(key); - if (!p) { p = { source: e.source, target: e.target, count: 0, fresh: 0, dropped: 0, ports: [], last: '' }; pairs.set(key, p); } + if (!p) { p = { source: e.source, target: e.target, count: 0, fresh: 0, dropped: 0, ports: [], last: '', http: null }; pairs.set(key, p); } p.count += e.connections; p.fresh += e.new; p.dropped += e.dropped; if (!p.ports.includes(e.port)) p.ports.push(e.port); if (e.last > p.last) p.last = e.last; + if (e.http) p.http = mergeHttp(p.http, e.http); } return [...pairs.values()]; } +// HTTP from several ports of one pair: counts add up, percentiles take the +// worse port (an upper bound, like the histogram buckets they come from). +function mergeHttp(a, b) { + if (!a) return { ...b, status: { ...b.status }, paths: [...(b.paths || [])] }; + a.requests += b.requests; + for (const [k, n] of Object.entries(b.status || {})) a.status[k] = (a.status[k] || 0) + n; + a.p50Ms = Math.max(a.p50Ms, b.p50Ms); + a.p95Ms = Math.max(a.p95Ms, b.p95Ms); + a.paths = [...a.paths, ...(b.paths || [])].sort((x, y) => y.count - x.count).slice(0, 10); + return a; +} +const errShare = (h) => (h.requests ? (h.status['5xx'] || 0) / h.requests : 0); +function httpSummary(h) { + const err = h.status['5xx'] || 0; + return t('http.summary', { n: h.requests, p95: fmtMs(h.p95Ms) }) + (err ? t('http.errors', { pct: Math.max(1, Math.round(errShare(h) * 100)) }) : ''); +} +const fmtMs = (ms) => (ms >= 1000 ? `≤${ms / 1000} s` : `≤${ms} ms`); function buildElements() { const els = []; @@ -642,14 +661,14 @@ function buildElements() { function trafficEdgeElement(e) { return { - group: 'edges', classes: 'traffic' + (e.dropped ? ' drop' : ''), + group: 'edges', classes: 'traffic' + (e.dropped ? ' drop' : '') + (e.http && errShare(e.http) > 0 ? ' http5xx' : ''), data: { id: 'tr:' + e.source + '→' + e.target, source: e.source, target: e.target, ...trafficEdgeData(e) }, }; } function trafficEdgeData(e) { return { width: 1.6 + Math.min(4.5, Math.log10(e.count + 1) * 1.7), - label: `${e.ports.slice(0, 3).join(', ')}${e.ports.length > 3 ? '…' : ''} · ${t('edge.conns', { n: e.count })}${e.dropped ? t('edge.drop', { n: e.dropped }) : ''}`, + label: `${e.ports.slice(0, 3).join(', ')}${e.ports.length > 3 ? '…' : ''} · ${e.http ? httpSummary(e.http) : t('edge.conns', { n: e.count })}${e.dropped ? t('edge.drop', { n: e.dropped }) : ''}`, flow: e, }; } @@ -1033,6 +1052,7 @@ function renderLegend() { el('span', {}, el('span', { class: 'line config' }), t('legend.config')), el('span', {}, el('span', { class: 'line traffic' }), t('legend.traffic')), el('span', {}, el('span', { class: 'line drop' }), t('legend.drop')), + el('span', {}, el('span', { class: 'line http5xx' }), t('legend.http5xx')), el('span', {}, t('legend.ghost')), el('span', {}, t('legend.notCollected')), ...badgeLegend(), @@ -1289,6 +1309,15 @@ function openExternalDetails(ep) { setTab('overview'); } +function httpPaths(h) { + if (!h?.paths?.length) return null; + return el('table', { class: 'pods http-paths' }, + el('thead', {}, el('tr', {}, ...[t('http.path'), t('http.requests'), 'p95', '5xx'].map((x) => el('th', {}, x)))), + el('tbody', {}, h.paths.map((p) => el('tr', {}, + el('td', {}, el('span', { class: 'muted' }, p.method + ' '), p.path), + el('td', {}, String(p.count)), el('td', {}, fmtMs(p.p95Ms)), + el('td', { class: p.errors ? 'warn-text' : '' }, String(p.errors)))))); +} function linkButton(id, meta, dropped) { const n = G.byId.get(id); const ep = state.flows?.endpoints.find((x) => x.id === id); @@ -1310,9 +1339,10 @@ function renderLinks(id) { } const tIn = [], tOut = []; for (const e of state.trafficEdges || []) { - const meta = `${e.ports.join(', ')} · ${t('edge.conns', { n: e.count })}${e.fresh ? t('edge.new', { n: e.fresh }) : ''}${e.dropped ? t('edge.drop', { n: e.dropped }) : ''}`; - if (e.target === id) tIn.push(linkButton(e.source, meta, e.dropped)); - if (e.source === id) tOut.push(linkButton(e.target, meta, e.dropped)); + const meta = `${e.ports.join(', ')} · ${t('edge.conns', { n: e.count })}${e.fresh ? t('edge.new', { n: e.fresh }) : ''}${e.dropped ? t('edge.drop', { n: e.dropped }) : ''}${e.http ? ' · ' + httpSummary(e.http) : ''}`; + const bad = e.dropped || (e.http && errShare(e.http) > 0); + if (e.target === id) { tIn.push(linkButton(e.source, meta, bad)); const tb = httpPaths(e.http); if (tb) tIn.push(el('li', { class: 'http-li' }, tb)); } + if (e.source === id) { tOut.push(linkButton(e.target, meta, bad)); const tb = httpPaths(e.http); if (tb) tOut.push(el('li', { class: 'http-li' }, tb)); } } const section = (title, items, empty) => [el('div', { class: 'section-title' }, title), items.length ? el('ul', { class: 'link-list' }, items) : el('p', { class: 'muted small' }, empty)]; diff --git a/images/cluster-map/src/web/static/i18n.js b/images/cluster-map/src/web/static/i18n.js index ff4c82f..4fff1f9 100644 --- a/images/cluster-map/src/web/static/i18n.js +++ b/images/cluster-map/src/web/static/i18n.js @@ -121,6 +121,8 @@ const I18N = { 'status.noViewerToken': 'The map shows what your own permissions allow, but the request carries no user token. Sign in again; if it repeats, the DexAuthenticator does not pass the Authorization header.', 'plural.visibleNs': { one: '{n} namespace visible', other: '{n} namespaces visible' }, 'visibility.namespaces': 'You see the namespaces where your role can list pods; peers elsewhere are shown as hidden.', + 'http.summary': 'HTTP {n} req, p95 {p95}', 'http.errors': ', {pct}% 5xx', 'http.path': 'Path', 'http.requests': 'Requests', + 'legend.http5xx': 'HTTP 5xx on the link', }, ru: { 'kind.pod': 'Контроллер', 'kind.external': 'Внешний', 'folder.pod': 'Другие контроллеры', @@ -239,6 +241,8 @@ const I18N = { 'status.noViewerToken': 'Карта показывает то, что разрешают ваши права, но в запросе нет токена пользователя. Войдите заново; если повторится — DexAuthenticator не передаёт заголовок Authorization.', 'plural.visibleNs': { one: 'виден {n} namespace', few: 'видны {n} namespace', many: 'видно {n} namespace', other: 'видны {n} namespace' }, 'visibility.namespaces': 'Видны namespace, где ваша роль может смотреть поды; соседи из остальных показаны скрытыми.', + 'http.summary': 'HTTP {n} запр., p95 {p95}', 'http.errors': ', {pct}% 5xx', 'http.path': 'Путь', 'http.requests': 'Запросов', + 'legend.http5xx': 'есть ответы HTTP 5xx', }, };