Jenkins.agent fields are missing from Otel jenkins logs

Elastic Observability APM
,
1

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

Kibana version: 8.16.4

Elasticsearch version:8.16.4

APM Server version:8.16.4

Jenkins otel plugin version: 3.1495.v64dcff5b_7a_6c

Original install method (e.g. download page, yum, deb, from source, etc.) and version: docker compose

Hello, I have setup Jenkins with the OpenTelemetry plugin and elasticsearch, kibana and apm-server using docker compose. I have successfully been able to send logs from Otel plug-in to elastic and configure elastic logs in jenkins open telemetry plug in (validate elasticsearch configuration passed)

I receive most logs but am missing jenkins.agent.total, online, offline etc as seen below:

image
image1892×943 160 KB

Loggers on jenkins are:

 -   loggers: 
         -   level: "FINER" 
             name: "io.jenkins.plugins.opentelemetry.job.log.OverallLog" 
         -   level: "FINER" 
             name: "io.jenkins.plugins.opentelemetry.backend.elastic" 
         -   level: "FINER" 
             name: "io.jenkins.plugins.opentelemetry.job.log.util" 
         -   level: "FINE" 
             name: "io.jenkins.plugins.opentelemetry" 
         -   level: "FINE" 
             name: "io.jenkins.plugins.opentelemetry.init" 
         -   level: "FINE" 
             name: "io.jenkins.plugins.opentelemetry.job.log.OtelLogStorageFactory" 
         -   level: "ALL" 
             name: "io.jenkins.plugins.opentelemetry.OpenTelemetrySdkProvider" 
         -   level: "FINE" 
             name: "io.jenkins.plugins.opentelemetry.job.MonitoringPipelineListener" 
         -   level: "FINE" 
             name: "io.jenkins.plugins.opentelemetry.job.MonitoringRunListener"

There are no errors in any of the elastic stack logs or jenkins system logs and have some other jenkins fields populated:

jenkins.executor
jenkins.node
jenkins.plugins.updates
jenkins.queue

is there a setting or configuration i am missing to send jenkins.agent logs to elasticsearch?

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

2

I have found this error in apm server logs:
{"log.level":"error","@timestamp":"2025-03-20T16:00:35.672Z","message":"failed to index documents in '.ds-metrics-apm.app.jenkins-default-2025.03.14-000001' (document_parsing_exception): ","component":{"binary":"apm-server","dataset":"elastic_agent.apm_server","id":"apm-default","type":"apm"},"log":{"source":"apm-default"},"documents":7,"ecs.version":"1.6.0","log.origin":{"file.line":443,"file.name":"v2@v2.3.1/appender.go","function":"github.com/elastic/go-docappender/v2.(*Appender).flush"},"service.name":"apm-server","ecs.version":"1.6.0"}
but am unsure why it is happening, could this cause the issue?

3

have force merged indexes using curl command and above error no longer exists but still do not receive jenkins.agent logs

4

The Jenkins OpenTelemetry plugin sends the Jenkins pipelines logs from the host that generates it. It means that when you run a pipeline some of it run on the Jenkins Controller, and other parts run in the Jenkins build Agents. So your Jenkins Controller and your Jenkins Build Agents must reach the Elastic Stack in order to send all the Jenkins pipeline logs.
The demo give you a reference configuration it uses Docker compose opentelemetry-plugin/demos at main · jenkinsci/opentelemetry-plugin · GitHub