ApmServer stopped infusing kubernetes.pod.name in output stream

Elastic Observability APM
, ,
1

Kibana version:
8.17.6
Elasticsearch version:
8.17.2
APM Server version:
8.16.2
APM Agent language and version:
Java/ Nodejs. Java version:
apm-agent-nodejs/4.4.1 / ; apm-agent-nodejs/3.52.2
apm-agent-java/1.38.0 ; apm-agent-java/1.52.1 etc.

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
air gapped style, downloaded from elastic documented docker artifactory and pushed to private artifactory after necessary scans.

Fresh install or upgraded from other version?
we have been upgrading apm-server from last 3 years regularly. last one was done in last quarter
Is there anything special in your setup?
Yes. Earlier, we did not have output of ApmServer to logstash and kafka. Recently we started enriching with logstash by adding some landscape specific fields

We did not change any index patterns. We enriched and sent to the earlier existing patterns elasticsearch destinations (in logstash now).

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Of late we started observing that : fields like kubernetes.pod.name ; kubernetes.namespace are not appearing in the index. kubernetes.pod.name is not at all appearing & kubernetes.namespace sometimes for certain pods is appearing in-correctly(did not capture).

what did we debug till so far:
In logstash script meant for output of apm-server, we put a simple mutate kind of plugin as the first operation in the filters in prod and dev landscapes:

      if[service][name] == "myproduct-myservice-svc"
        {
        mutate{
          add_field => { "random.field.value" => "%{[kubernetes][pod][name]}" }
        }
      }

In production, we are able to see the random.field.value as kubernetes pod name. but in dev landscapes, we are not able to see. This means that apm server is not generating the document with kubernetes.pod.name as field.

Steps to reproduce:
We are not able to understand what is causing such behaviour and data loss got introduced.
There are 3 dev landscapes and 'n' production landscapes.

Earlier in sandbox/staging -- we noticed non availability of these fields, but of late, we are observing same in our pre-prod.

Ask to community:
Please let's know pointers for debugging what might have caused such behaviour of data loss.

this is affecting of course the custom visualisations for JVM that have been created for developers

the kibana DataView and the underlying elastic index is depicted below:

image
image1099×590 67 KB

2

Hello @elastic

Well, we migrated the hardware and changed associated OS image.

Now pod name is coming from host.name.

Question:

  1. will the field name change in above index depending on the hardware/os?
  2. if so, what are the fields that we should look?

Simultaneously, I am thinking of looking into code base for possible fields in apm-agent-attach. But, information coming from horse's mouth will always have importance. It will help to convince easily.

To make things working at our side, we will consider all such fields.

Thank you for your support.