Get metrics for pods that no longer exist

[George_Guliman]

Hello,

Can someone recommend me a method to get metrics for CPU and memory for pods that no longer exist. The reason for which they no longer exist doesn't really matter, they could have been evicted, delete by someone, replaced by updates, etc...

I am trying to get this to work from the metrics app. I am mainly trying to search for the pods that belong to a deployment by using the kubernetes.deployment.name attribute but the search in the metrics app does not return anything.

My metricbeat configuration is as follows:

1. For daemonset

metricbeatConfig:
    metricbeat.yml: |
      metricbeat.modules:
      - module: kubernetes
        metricsets:
          - container
          - node
          - pod
          - system
          - volume
        period: 1m
        host: "${NODE_NAME}"
        hosts: ["https://${NODE_NAME}:10250"]
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        ssl.verification_mode: "none"
        # If using Red Hat OpenShift remove ssl.verification_mode entry and
        # uncomment these settings:
        #ssl.certificate_authorities:
          #- /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
        processors:
        - add_kubernetes_metadata: ~
      - module: kubernetes
        enabled: true
        metricsets:
          - event
      - module: system
        period: 1m
        metricsets:
          - cpu
          - load
          - memory
          - network
          - process
          - process_summary
        processes: ['.*']
        process.include_top_n:
          by_cpu: 5
          by_memory: 5
      - module: system
        period: 1m
        metricsets:
          - filesystem
          - fsstat
        processors:
        - drop_event.when.regexp:
            system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
      output.elasticsearch:
        username: '${ELASTICSEARCH_USERNAME}'
        password: '${ELASTICSEARCH_PASSWORD}'
        protocol: https
        hosts: ["security-master:9200"]
        ssl.certificate_authorities:
          - /usr/share/metricbeat/config/certs/elastic-certificate.pem
      setup.kibana:
        host: "dummy-url:443"
        ssl.verification_mode: 'none'
      setup.dashboards.enabled: true

2. For deployment (metric server / state metrics)

 metricbeat.yml: |
      metricbeat.modules:
      - module: kubernetes
        enabled: true
        metricsets:
          - state_node
          - state_deployment
          - state_replicaset
          - state_pod
          - state_container
        period: 1m
        hosts: ["${KUBE_STATE_METRICS_HOSTS}"]
      output.elasticsearch:
        username: '${ELASTICSEARCH_USERNAME}'
        password: '${ELASTICSEARCH_PASSWORD}'
        protocol: https
        hosts: ["security-master:9200"]
        ssl.certificate_authorities:
          - /usr/share/metricbeat/config/certs/elastic-certificate.pem
      setup.kibana:
        host: "dummy-url:443"
      #setup.dashboards.enabled: true 

Thank you in advance,