Sending jenkins metrics to elastic APM

[Neelam_Zanvar]

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version: 8.4.2

Elasticsearch version: 8.4.2

APM Server version: 8.4.2

APM Agent language and version: otel plugin in jenkins 2.9.2
Browser version: chrome Version 105.0.5195.127

Original install method (e.g. download page, yum, deb, from source, etc.) and version:yum
Fresh install or upgraded from other version? fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. NO

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I've setup otel plugin on jenkins to send metrics to apm. when it navigates to the apm page to visualize the metrics few metrics show blank saying the fields are missing in the index

on the jenkins plugin page i am getting . Even after giving correct credentials to access elasticsearch. WHile mentioning the url of elasticsearch it has asked for read on permission on logs-apm-app index. how can i do that?

in the logs i'm getting this. I've setup elasticsearch credentials in apm.yml file

Steps to reproduce:

1. install apm on elk machine and added credentials to connect to elasticsearch.
2. install otel plugin on jenkins and configure it to connect to apm
3. import jenkins dashboards to kibana

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

[aninfinite (Victor Martinez)]

Jenkins overview shows blank · Issue #535 · jenkinsci/opentelemetry-plugin · GitHub contains the same request and further comments regarding some troubleshooting.
I was not able to reproduce the error locally using the same versions (Elastic stack, Jenkins, OpenTelemetry plugin).

[Neelam_Zanvar]

I need to give read access to elastic(custom) user to logs-apm.app index template. i am missing something on the apm server side i guess.
When i click on elastic observability on jenkins and it directs me to jenkins overview dashboard getting this in the logs of apm server

on apm server side in the config file i have just added

Even on jenkins opentelemetry plugin side

Please help. i am getting all the data except agent data, queue data and git data. i followed the ticked u shared but didnt get any help

[aninfinite (Victor Martinez)]

Can you verify if the elastic user can access your Elasticsearch from the Jenkins CI Controller?

For instance, if you access to your Jenkins Controller by ssh, you could use curl:

curl -XGET -u elastic:PASSSWORD https://devops.minutuscloud.com/elastic

Please replace PASSWORD with the one you filled in the Elasticsearch credentials in Jenkins

The above might help you to know whether the user exists in Elasticsearch and it's accessible from the Jenkins controller.

[Neelam_Zanvar]

@aninfinite

curl -XGET -u elastic:PASSSWORD https://devops.minutuscloud.com/elastic

Hello, thank you for your response and time. I tried to access the elastic url(which is behind reverse proxy) as you said. getting nginx error moved permanently. so i changed elastic url to the actual url ignoring reverse proxy. i am able to access it using curl from jenkins controller. and now the error changes to

[Neelam_Zanvar]

The error from plugin is gone. but in logs still i am getting this

am i missing some config in apm server?

[aninfinite (Victor Martinez)]

Do you see any traces in Kibana for the Jobs running in Jenkins? Do you see any logs for those jobs in Kibana? What do you mean with in logs still I am getting this?

On the other hand, one way to start with and get familiar with the Jenkins OpenTelemetry plugin could be by using the demos in opentelemetry-plugin/demos at master · jenkinsci/opentelemetry-plugin · GitHub and see if the same settings work in that particular environment. Then you could change the Jenkins configuration to use your elastic stack, so you can discard if anything is misconfigured

[Neelam_Zanvar]

In logs in am getting this means I have attached a screenshot of log errors, that I am getting on APM server machine when i click on elastic observability on jenkins

[Neelam_Zanvar]

yes i can see the traces of jobs running in kibana

[Neelam_Zanvar]

Hi i am still stuck with this issue. on apm logs i am getting
failed to index event (illegal_argument_exception): can't merge a non object mapping [system.cpu.load] with an object mapping","service.name":"apm-server"