v0.11.1: repeated restarts and OOM kills stay on the map for a day
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
4c5eb288cb
commit
8ed76cd01a
@@ -1,3 +1,6 @@
|
||||
v0.11.1:
|
||||
fixes:
|
||||
- "A container restarted three times or more stays marked for 24 hours after its last restart or OOM kill, not 30 minutes: a pod OOM-killed five times in a day was no longer shown 40 minutes after the last kill. A single restart still clears after 30 minutes."
|
||||
v0.11.0:
|
||||
features:
|
||||
- "HTTP details where Cilium parses L7 (a CiliumNetworkPolicy with an http rule on the port): requests, responses by class, p50 and p95 latency and the ten busiest paths on the link, IDs in paths grouped as :id. Links with 5xx responses are drawn in the warning colour. Headers are never requested from Hubble. The docs have an example policy that turns this on for one service without restricting it."
|
||||
|
||||
+1
-1
@@ -25,7 +25,7 @@ Workloads managed by non-native controllers are included: Advanced DaemonSets of
|
||||
|
||||
A card with a problem carries a badge: yellow `!` for a warning, red `!` for an error. Namespace boxes and cards, node boxes and the tree take the colour of the worst problem inside. Sources:
|
||||
|
||||
- **Object state**: containers in `CrashLoopBackOff`, `ImagePullBackOff` and similar, recent restarts and OOM kills, pods pending or unschedulable, workloads with no or not all replicas ready for more than 2 minutes, a stuck rollout, a CronJob whose last run failed, a Service with no ready pod, an Ingress routing to a Service that does not exist, a node that is not ready or is under memory, disk or PID pressure.
|
||||
- **Object state**: containers in `CrashLoopBackOff`, `ImagePullBackOff` and similar, restarts and OOM kills of the last 30 minutes (of the last 24 hours for a container restarted three times or more), pods pending or unschedulable, workloads with no or not all replicas ready for more than 2 minutes, a stuck rollout, a CronJob whose last run failed, a Service with no ready pod, an Ingress routing to a Service that does not exist, a node that is not ready or is under memory, disk or PID pressure.
|
||||
- **Warning events** of the last 30 minutes, folded per object and reason. Scheduling, volume and back-off failures badge the card at once; any other reason needs five repeats inside the window and is listed without a badge before that, so a single failed probe during a restart does not colour the map. An event's count is cumulative since its first occurrence, so a long-lived event is scaled to the window: 40 probe failures over two days is one an hour, not a burst.
|
||||
- **Prometheus alerts** that Deckhouse mirrors into `ClusterAlert` objects. An alert is attached by its labels (`deployment`, `statefulset`, `daemonset`, `pod`, `service`, `ingress`, `node`, `namespace`); one without an object or namespace goes to the cluster list. `severity_level` 1–3 is an error, 4–6 a warning, 7–9 information.
|
||||
|
||||
|
||||
+1
-1
@@ -25,7 +25,7 @@ Workload'ы под нестандартными контроллерами то
|
||||
|
||||
Карточка с проблемой несёт значок: жёлтый `!` — предупреждение, красный `!` — ошибка. Рамки и карточки namespace, рамки нод и дерево окрашиваются по самой серьёзной проблеме внутри. Источники:
|
||||
|
||||
- **Состояние объектов**: контейнеры в `CrashLoopBackOff`, `ImagePullBackOff` и подобном, недавние перезапуски и убийства за нехватку памяти, поды в ожидании или без места для запуска, workload'ы, у которых больше 2 минут не готовы все или часть реплик, застрявшая выкатка, CronJob с неудачным последним запуском, Service без готовых подов, Ingress на несуществующий Service, нода не готова или ей не хватает памяти, диска или PID.
|
||||
- **Состояние объектов**: контейнеры в `CrashLoopBackOff`, `ImagePullBackOff` и подобном, перезапуски и убийства за нехватку памяти за последние 30 минут (за сутки, если контейнер перезапускался три раза и больше), поды в ожидании или без места для запуска, workload'ы, у которых больше 2 минут не готовы все или часть реплик, застрявшая выкатка, CronJob с неудачным последним запуском, Service без готовых подов, Ingress на несуществующий Service, нода не готова или ей не хватает памяти, диска или PID.
|
||||
- **Warning-события** за последние 30 минут, сгруппированные по объекту и причине. Сбои планирования, томов и back-off дают значок сразу; любой другой причине нужно пять повторов в пределах окна, до этого она в списке без значка — одна проваленная проба во время рестарта не красит карту. Счётчик события накапливается с первого появления, поэтому давнее событие пересчитывается на окно: 40 проваленных проб за двое суток — это раз в час, а не всплеск.
|
||||
- **Алерты Prometheus**, которые Deckhouse зеркалирует в объекты `ClusterAlert`. Алерт привязывается по меткам (`deployment`, `statefulset`, `daemonset`, `pod`, `service`, `ingress`, `node`, `namespace`); алерт без объекта и namespace попадает в список кластера. `severity_level` 1–3 — ошибка, 4–6 — предупреждение, 7–9 — информация.
|
||||
|
||||
|
||||
@@ -55,10 +55,17 @@ type Problem struct {
|
||||
var (
|
||||
now = time.Now
|
||||
restartWindow = 30 * time.Minute
|
||||
eventWindow = 30 * time.Minute
|
||||
pendingAfter = 5 * time.Minute
|
||||
notReadyAfter = 2 * time.Minute // a pod starting or restarting is not yet a problem
|
||||
maxPerCard = 20
|
||||
// A container that keeps restarting stays marked much longer: one
|
||||
// restart an hour ago is history, the fifth OOM kill in a day is a
|
||||
// memory limit that is too low. Found when a panel was OOM-killed five
|
||||
// times in 23 hours and the map showed nothing 40 minutes after the
|
||||
// last one.
|
||||
repeatedRestarts int32 = 3
|
||||
repeatedWindow = 24 * time.Hour
|
||||
eventWindow = 30 * time.Minute
|
||||
pendingAfter = 5 * time.Minute
|
||||
notReadyAfter = 2 * time.Minute // a pod starting or restarting is not yet a problem
|
||||
maxPerCard = 20
|
||||
// Warning events that mark a card even when seen once. Any other reason
|
||||
// (a failed readiness probe during a restart, say) needs eventRepeats
|
||||
// occurrences in the window; below that it is listed, not badged.
|
||||
@@ -257,7 +264,7 @@ func podProblems(s *problemSet, p corev1.Pod) {
|
||||
s.add(id, Problem{Severity: Critical, Source: "state", Reason: w.Reason, Message: w.Message, Object: obj, Count: int(cs.RestartCount)})
|
||||
continue
|
||||
}
|
||||
if t := cs.LastTerminationState.Terminated; t != nil && now().Sub(t.FinishedAt.Time) < restartWindow {
|
||||
if t := cs.LastTerminationState.Terminated; t != nil && recentRestart(t.FinishedAt.Time, cs.RestartCount) {
|
||||
reason := "Restarted"
|
||||
if t.Reason == "OOMKilled" {
|
||||
reason = "OOMKilled"
|
||||
@@ -571,3 +578,10 @@ func alertProblem(s *problemSet, a *unstructured.Unstructured, podOwner map[stri
|
||||
since, _, _ := unstructured.NestedString(a.Object, "status", "startsAt")
|
||||
s.add(AlertTargetID(labels, podOwner), Problem{Severity: AlertSeverity(level), Source: "alert", Reason: name, Message: strings.TrimSpace(summary), Since: since})
|
||||
}
|
||||
|
||||
// recentRestart: the last restart is recent, or the container restarts
|
||||
// again and again and the last one was within a day.
|
||||
func recentRestart(finished time.Time, restarts int32) bool {
|
||||
age := now().Sub(finished)
|
||||
return age < restartWindow || (restarts >= repeatedRestarts && age < repeatedWindow)
|
||||
}
|
||||
|
||||
@@ -59,7 +59,7 @@ func TestProblemsFromState(t *testing.T) {
|
||||
withNow(t)
|
||||
ready := []corev1.PodCondition{{Type: corev1.PodReady, Status: corev1.ConditionTrue}}
|
||||
g := &Graph{Nodes: []Node{
|
||||
{ID: "ns/shop"}, {ID: "wl/shop/deployment/web", Kind: "deployment", Facts: map[string]string{"health": "warn"}}, {ID: "service/shop/web"}, {ID: "service/shop/idle"},
|
||||
{ID: "ns/shop"}, {ID: "wl/shop/deployment/web", Kind: "deployment", Facts: map[string]string{"health": "warn"}}, {ID: "wl/shop/deployment/api", Kind: "deployment"}, {ID: "service/shop/web"}, {ID: "service/shop/idle"},
|
||||
{ID: "ingress/shop/web"}, {ID: "wl/shop/cronjob/backup"}, {ID: "node/node-a"},
|
||||
}}
|
||||
in := problemInputs{
|
||||
@@ -78,6 +78,13 @@ func TestProblemsFromState(t *testing.T) {
|
||||
Name: "app", LastTerminationState: corev1.ContainerState{Terminated: &corev1.ContainerStateTerminated{Reason: "OOMKilled", FinishedAt: metav1.NewTime(t0.Add(-3 * time.Hour))}},
|
||||
}}},
|
||||
},
|
||||
{ // OOM-killed five times, the last 70 minutes ago: still a problem
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "api-7c-a", Namespace: "shop", Labels: map[string]string{"app": "api"}, OwnerReferences: owned("ReplicaSet", "api-7c")},
|
||||
Status: corev1.PodStatus{Phase: corev1.PodRunning, Conditions: ready, ContainerStatuses: []corev1.ContainerStatus{{
|
||||
Name: "app", RestartCount: 5,
|
||||
LastTerminationState: corev1.ContainerState{Terminated: &corev1.ContainerStateTerminated{Reason: "OOMKilled", ExitCode: 137, FinishedAt: metav1.NewTime(t0.Add(-70 * time.Minute))}},
|
||||
}}},
|
||||
},
|
||||
{ // bare pod pending for 10 minutes: badge on the namespace
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "debug", Namespace: "shop", CreationTimestamp: metav1.NewTime(t0.Add(-10 * time.Minute))},
|
||||
Status: corev1.PodStatus{Phase: corev1.PodPending, Conditions: []corev1.PodCondition{{Type: corev1.PodScheduled, Status: corev1.ConditionFalse, Message: "0/3 nodes are available"}}},
|
||||
@@ -113,6 +120,9 @@ func TestProblemsFromState(t *testing.T) {
|
||||
if _, ok := web["OOMKilled"]; ok {
|
||||
t.Error("an OOM kill three hours ago must not mark the card")
|
||||
}
|
||||
if p := problemsOf(g, "wl/shop/deployment/api")["OOMKilled"]; p.Severity != Warning || p.Count != 5 {
|
||||
t.Errorf("repeated OOM kills 70 minutes ago: %+v", problemsOf(g, "wl/shop/deployment/api"))
|
||||
}
|
||||
if _, ok := problemsOf(g, "service/shop/web")["NoEndpoints"]; ok {
|
||||
t.Error("service with a ready pod reported without endpoints")
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user