Red Orbita
Home Linux & Systems Cybersecurity Cloud & DevOps Networks & Infrastructure SIEM & Monitoring DFIR & Threat Intel Development & Other All categories Projects About Tools

ama-metrics-ksm on CrashLoopBackOff in AKS: Diagnosis and Solution by Azure Monitor Configuration

Leer en espanol
October 1, 2025 Cloud & DevOps rokitoh
ama-metrics-ksm on CrashLoopBackOff in AKS: Diagnosis and Solution by Azure Monitor Configuration

Table of contents

When working with Azure Kubernetes Service (AKS), it is normal to depend on tools like Azur ===

ama-metrics-ksm on CrashLoopBackOff in AKS: Diagnosis and Solution by Azure Monitor Configuration

When working with Azure Kubernetes Service (AKS), it is normal to rely on tools like Azure Monitor to obtain metrics and visibility of the cluster. However, one incorrect or missing configuration Azure Monitor/Log Analytics integration may cause Pods critics, like ama-metrics-ksm, they do not start and enter a cycle of constant reboots. This article details how to identify this problem, analyze the output misleading and solve it effectively reinstalling the integration.

Identifying the problem

In our cluster, we detect that a pod of the namespace kube-system It was crashing continuously:

CODE
kubectl get pods -n kube-system

Output:

CODE
kube-system   ama-metrics-ksm-84cb9d85fb-tlbpb   0/1   CrashLoopBackOff   5 (44s ago)   3m51s

The state CrashLoopBackOff indicates that Kubernetes is trying to boot the pod, but it crashes and restarts repeatedly.

Analyzing the logs

To understand the cause of the failure, we review the logs of the pod:

CODE
kubectl logs ama-metrics-ksm-84cb9d85fb-tlbpb -n kube-system

Output:

TERRAFORM
E1008 09:46:30.798386       1 wrapper.go:40] "Failed to run kube-state-metrics" err="failed to set up annotations allowlist: resource does not exist. Available resources: leases,namespaces,..."

He pod It cannot initialize because it cannot find certain resources or configurations that it expects. In this context, this error is a common indicator that he pod ama-metrics-ksm could not obtain the configuration parameters of the workspace Azure Monitor/Log Analytics (like the Resource ID or the ingest URL), so it can't boot.

    Check Azure Monitor Settings

    The diagnostic focuses on confirming whether the Azure Monitor integration configuration on the cluster from AKS is enabled and points to a workspace valid.

    Check enabled monitoring addon

    CODE
    az aks show --name <cluster-name> --resource-group <cluster-resource-group> --query addonProfiles.monitoring
    • null → addon not enabled.
    • enabled: true and workspaceResourceId → addon enabled and showing the associated workspace.

    Review configured workspace

    CODE
    az aks show --name <cluster-name> --resource-group <cluster-resource-group> --query addonProfiles.monitoring.config.logAnalyticsWorkspaceResourceID

    You must return the Complete Resource ID of Log Analytics/Azure Monitor Workspace that AMA is using.

    • If it is empty or incorrect → AMA does not have workspace configured.

    Review Pod environment variables (Optional)

    If the configuration at the level cluster seems correct, we can inspect the Deployment:

    CODE
    kubectl get deployment ama-metrics-ksm -n kube-system -o yaml

    Find variables like AZURE_MONITOR_WORKSPACE_RESOURCE_ID.

    • If they are empty or incorrect → AMA has not received the correct configuration.

      Test connectivity from the Pod

      BASH
      kubectl exec -n kube-system <ama-metrics-ksm-pod> -- curl -v https://<workspace-id>.ods.opinsights.azure.com
      • If it fails → the Pod cannot communicate with Log Analytics.
      • If successful → AMA can successfully submit metrics.

      Evidence-Based Diagnosis

      After confirming with the previous steps, it is determined that:

      1. He pod ama-metrics-ksm depends on Azure Monitor for configuration.
      2. The logs show an initialization failure (resource does not exist), which indicates the absence of required configuration of the workspace.
      3. The configuration of the cluster (via az aks show) most likely indicates that there is not a workspace valid connected or configuration is null.
      4. He CrashLoopBackOff occurs immediately upon boot.

      Conclusion: He pod fails because Azure Monitor extension in AKS does not have a valid or complete configuration of the workspace of Log Analytics/Azure Monitor, preventing the agent from initializing correctly.

      Solution: Reinstall Azure Monitor Metrics

      The most effective solution to force a complete reconfiguration and fix of the integration is re-run the update cluster explicitly enabling metrics integration with the Resource ID correct of workspace.

      SQL
      az aks update \
  --enable-azure-monitor-metrics \
  --name <cluster-name> \
  --resource-group <cluster-resource-group> \
  --azure-monitor-workspace-resource-id /subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/microsoft.monitor/accounts/<workspace-name>

      Warning: If you already had an Azure Monitor configuration, this command will will overwrite with the Resource ID specified. Make sure you use the correct ID.

      This restores the integration at the AKS level. Kubernetes will automatically recreate he pod ama-metrics-ksm with the correct configuration

      Verify that the Pod is active

      CODE
      kubectl get pods -n kube-system
      Ama-metrics-ksm en CrashLoopBackOff en AKS: Diagnóstico y Solución por Configuración de Azure Monitor

      Validate logs

      CODE
      kubectl logs -n kube-system ama-metrics-ksm-<pod-name>

      wow!

       
       

      Comments