Custom Field in apm client side configuration

[abhishek.gpt1806 (Abhishek Gupta)]

I am intending to add a field so that apart from service.name we get application.name as well.
Please suggest. We have java apm elastic agent

Kibana version: 7.17.3

Elasticsearch version: 7.17.3

APM Server version: 7.17.3

APM Agent language and version: Java

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: EKS helm chart

[marone (Marwane Chahoud)]

Hey @abhishek.gpt1806 ,

Check this documentation ECS logging in Java to add ECS fields and also custom fields, maybe I think something like:

<encoder class="co.elastic.logging.logback.EcsEncoder">
    <applicationName>my-application</applicationName>
</encoder>

you define them in your application.properties file in your java application.

[felixbarny (Felix Barnsteiner)]

You can use the global_labels config option to add arbitrary metadata to your APM events.

Hope this helps

[marone (Marwane Chahoud)]

One more thing to add for your suggestion @felixbarny, I've already wanted to use the labels as suggested, but they are not added at the root document, an example for -Delastic.apm.global_labels=project=kappa:

all key,value pairs added to -Delastic.apm.global_labels=key=value[,key=value[,...]] are stored inside labels field, which is not suitable if we want to store them at the root level (without labels field).

[felixbarny (Felix Barnsteiner)]

For APM data, there is no way to add custom fields to the root level. This is to avoid potential mapping issues.

What's your use case? Why do you need labels to be at the root level?

A workaround if you really want to do this is to use an ingest pipeline to rename labels.project to project, for example.

[marone (Marwane Chahoud)]

yes good idea to use ingest pipeline to update the field name, my use case was to create for each application it's own APM index, something like apm-%{[observer.version]}-%{[labels.project]}-%{+yyyy.MM.dd} but found that it didn't work, because I assumed that the field labels.project doesn't exist when apm data are collected, it's added later when document is indexed with Elasticsearch. Am I wrong ?

[felixbarny (Felix Barnsteiner)]

@marone

I assumed that the field labels.project doesn't exist when apm data are collected, it's added later when document is indexed with Elasticsearch

The global labels should be available at this point. Not sure why it's not working. Maybe someone from the server team could help out here. I'll add the server tag to this question to ping them.

[marone (Marwane Chahoud)]

Ok following back the issue posted by @abhishek.gpt1806 , here is a solution I found to add custom fields:

I tested that using a basic simple java app with log4j2 for logging

1. Visit this link, it shows how to configure log4j2 using xml configuration
2. Follow the steps for ECS logging using java - log4j2
3. add your custom field as follow:

<EcsLayout serviceName="java.log" serviceVersion="my-app-version" serviceNodeName="my-app-cluster-node">
  <KeyValuePair key="application.name" value="my-app"/>
</EcsLayout>

The attributes of EcsLayout are default layouts defined by default from ECS, to add more custom fields use the same principal :

<EcsLayout serviceName="java.log" serviceVersion="my-app-version" serviceNodeName="my-app-cluster-node">
   <KeyValuePair key="application.name" value="my-app"/>
   <KeyValuePair key="application.os" value="linux"/>
 </EcsLayout>

Last step is to use Filebeat to ship logs from STDOUT or file stream to Elasticsearch, the problem here is that data collected with APM won't have custom fields defined above (application.name and applicaiton.os), therefore better use, as @felixbarny suggested the label field. And then use the same for your custom fields for logging like:

<KeyValuePair key="label.application.name" value="my-app"/>
<KeyValuePair key="label.application.os" value="linux"/>

like this I believe you have all your application logs and apm data use the same structure of ECS.

The output for above configuration is:

{"@timestamp":"2022-06-07T12:46:25.673Z","log.level":"ERROR","message":"Presley left no stone unturned.", "ecs.version": "1.2.0","service.name":"java.log","service.version":"my-app-version","service.node.name":"my-app-cluster-node","event.dataset":"java.log","process.thread.name":"main","log.logger":"org.example.App","application.name":"my-app","application.os":"linux"}