Infrastructure monitoring with OpenTelemetry in Elastic Observability via APM server

[sruthi.sattiraju (Sruthi)]

Kibana version:
8.17.1

Elasticsearch version:
8.17.1
APM Server version:
8.15.2

Hi Team,

As per the link Infrastructure monitoring with OpenTelemetry in Elastic Observability — Elastic Observability Labs, using ingest path 2, we should be able to use OTEL agent for infrastructure monitoring and ingest the metric data to Elastic search via APM server.

I tried following the steps mentioned in the given link, able to collect hostmetrics but when I send data to APM server, I dont see the host under Infrastructure-> hosts page. In Discover page I see the data, but getting assgined to apm related indices not infra.

{
"@timestamp": [
"2025-05-26T11:22:52.107Z"
],
"agent.name": [
"otlp"
],
"agent.name.text": [
"otlp"
],
"agent.version": [
"unknown"
],
"data_stream.dataset": [
"apm.app.unknown"
],
"data_stream.namespace": [
"default"
],
"data_stream.type": [
"metrics"
],
"event.ingested": [
"2025-05-26T11:22:58.000Z"
],
"host.hostname": [
"host1"
],
"host.name": [
"host1"
],
"host.name.text": [
"host1"
],
"host.os.platform": [
"linux"
],
"host.os.type": [
"linux"
],
"labels.cpu": [
"cpu2"
],
"labels.state": [
"idle"
],
"metricset.name": [
"app"
],
"metricset.name.text": [
"app"
],
"observer.hostname": [
"apmserver-dev-apm-server-f8f64c99c-9lfn9"
],
"observer.type": [
"apm-server"
],
"observer.version": [
"8.15.2"
],
"processor.event": [
"metric"
],
"service.language.name": [
"unknown"
],
"service.language.name.text": [
"unknown"
],
"service.name": [
"unknown"
],
"service.name.text": [
"unknown"
],
"service.node.name": [
"host1"
],
"service.node.name.text": [
"host1"
],
"system.cpu.time": [
67458.66
],
"_id": "v0pUDJcBG8sbMoiI4v5A",
"_index": ".ds-metrics-apm.app.unknown-default-2025.05.15-000003",
"_score": null
}

I need to get this considered at hostmetric instead of apm. I am missing something? please suggest.

[stephenb (Stephen Brown)]

Hi @sruthi.sattiraju

That article is nearly a year old an in OTEL time that is ancient so we should really look at the latest documentation

What I would recommend is go follow the guided setup at this time

Make sure the system integration is installed

Observability -> Add Data

Host -> Open Telemetry

And Follow the instruction

The will create a Sample otel.yml and

This approach will not even flow through the APM Server Directly from the OTEL Collector to Elastic Search

Next Steps you should really Read About EDOT and the More Update to Date Blogs and Documentation

I would also recommend to upgrade to 8.18

Note so this approach does not even use APM server.. and will not going forward

Here is the latest documentation

Elastic Distributions of OpenTelemetry (EDOT) Now GA: Open-Source,...

Elastic Distributions of OpenTelemetry | Elastic Documentation

https://www.elastic.co/docs/reference/opentelemetry/quickstart/self-managed/hosts_vm

Here are some limitations (Like CPU etc not collected on MAC at this time etc)

Limitations of Elastic Distributions of OpenTelemetry | Elastic Documentation

[sruthi.sattiraju (Sruthi)]

Hi @stephenb thanks for you response.

I am aware that there is EDOT which I can use to integrated otel agents with Elastic and I did use the EDOT collector and it works fine. Here I am looking to integrate Otel agent (Native not via EDOT) with APMServer ->Elastic Search. And as per the given the link it is supported. Can you please help me understand if this approach is still supported? If Yes, what is that I am missing or need to configure for the data collected by otel agent to be reflected under Infrastructure ->Hosts page.

Thanks
Sruthi

[stephenb (Stephen Brown)]

@sruthi.sattiraju

APM Server version:
8.15.2

First you need to update APM server to 8.17

Please share your otel.yaml and what OS are you on?

One thing I notice is the Authorization line is wrong when using APM server you need to include Bearer directive

exporters:
  otlphttp:
      endpoint: https://myapmserver:<correctport>
    tls:
      insecure: false
    headers:
      Authorization: Bearer <sectret_token> <<<< HAVE to put in Bearer which is NOT in the sample

Not MAC OS does not work right...

I used this with v127 Contrib and it worked

receivers:
  hostmetrics:
    collection_interval: 10s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization:
            enabled: true
          system.cpu.logical.count:
            enabled: true
      memory:
        metrics:
          system.memory.utilization:
            enabled: true
      process:
        metrics:
          process.open_file_descriptors:
            enabled: true
          process.memory.utilization:
            enabled: true
          process.disk.operations:
            enabled: true
      network:
      processes:
      load:
      disk:
      filesystem:

processors:
  resourcedetection/system:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]

exporters:
  otlphttp:
    endpoint: https://myapmserver:<correctport>
    tls:
      insecure: false
    headers:
      Authorization: Bearer asdfasdfasdf
  # debug:
  #   verbosity: normal

service:
  pipelines:
    metrics/host:
      receivers: [hostmetrics]
      processors: [resourcedetection/system]
      # exporters: [debug, otlphttp] 
      exporters: [otlphttp] 

I this metrics and the host showed up in the Infra

The data should end up in the data stream metrics-apm.app.unknown-default

[sruthi.sattiraju (Sruthi)]

Hi @stephenb thanks for the inputs. As a first step, I am trying to upgrade APMServer to 18.17.1 but its failing. I am attaching log here. Can you please review and suggest.

{"log.level":"info","@timestamp":"2025-06-03T11:09:41.977Z","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).init","file.name":"beatcmd/beat.go","file.line":146},"message":"Home path: [.] Config path: [.] Data path: [./data] Logs path: [./logs]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.981Z","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).init","file.name":"beatcmd/beat.go","file.line":153},"message":"Beat ID: 98d3b65a-552c-45b5-a994-9cb45dc4d8fb","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.981Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":577},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":".","data":"data","home":".","logs":"logs"},"type":"apm-server","uuid":"98d3b65a-552c-45b5-a994-9cb45dc4d8fb"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.981Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":585},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"cff9e464471f4e641164cb38215a18e899784b37","time":"2025-01-10T16:05:01.000Z","version":"8.17.1"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.981Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":588},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":8,"version":"go1.23.4"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.981Z","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.adjustMaxProcs.diffInfof.func2","file.name":"beatcmd/maxprocs.go","file.line":68},"message":"maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.982Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":592},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","native_architecture":"amd64","boot_time":"2025-05-20T16:27:52Z","containerized":false,"name":"apmserver-dev-apm-server-9bbbf8fb4-zr6mq","ip":["127.0.0.1/8","10.3.4.215/16","::1/128","fe80::c89b:99ff:fea9:aa66/64"],"kernel_version":"5.15.0-1088-azure","mac":["ca:9b:99:a9:aa:66"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"22.04.5 LTS (Jammy Jellyfish)","major":22,"minor":4,"patch":5,"codename":"jammy"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.982Z","log.logger":"beat","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.logSystemInfo","file.name":"beatcmd/beat.go","file.line":621},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/apm-server","exe":"/opt/dynatrace/oneagent/agent/lib64/oneagentdynamizer","name":"apm-server","pid":1,"ppid":0,"seccomp":{"mode":"disabled","no_new_privs":false},"start_time":"2025-06-03T11:09:40.170Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.983Z","log.logger":"config","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/config.(*AgentConfig).setup","file.name":"config/agentconfig.go","file.line":70},"message":"using output.elasticsearch for fetching agent config","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.983Z","log.logger":"config","log.origin":{"function":"github.com/elastic/apm-server/internal/beater/config.(*APIKeyAgentAuth).setup","file.name":"config/auth.go","file.line":75},"message":"Falling back to elasticsearch output for API Key usage","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.983Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.listen","file.name":"beater/http.go","file.line":142},"message":"Listening on: [::]:8200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.983Z","log.origin":{"function":"github.com/elastic/apm-server/internal/beatcmd.(*Beat).Run","file.name":"beatcmd/beat.go","file.line":395},"message":"apm-server started.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.985Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run","file.name":"beater/beater.go","file.line":191},"message":"MaxConcurrentDecoders set to 51 based on 80 percent of 0.5gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.985Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run","file.name":"beater/beater.go","file.line":198},"message":"Aggregation.MaxServices set to 500 based on 0.5gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.985Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run","file.name":"beater/beater.go","file.line":205},"message":"Aggregation.ServiceTransactions.MaxGroups for service aggregation set to 500 based on 0.5gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.985Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run","file.name":"beater/beater.go","file.line":212},"message":"Aggregation.Transactions.MaxGroups set to 2500 based on 0.5gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.985Z","log.logger":"beater","log.origin":{"function":"github.com/elastic/apm-server/internal/beater.(*Runner).Run","file.name":"beater/beater.go","file.line":219},"message":"Aggregation.ServiceDestinations.MaxGroups set to 7500 based on 0.5gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-06-03T11:09:41.985Z","log.logger":"kibana","log.origin":{"function":"github.com/elastic/elastic-agent-libs/kibana.NewClientWithConfigDefault","file.name":"kibana/client.go","file.line":181},"message":"Kibana url: https://kibana-dev-kb:5601","service.name":"apm-server","ecs.version":"1.6.0"}
runtime: g 68: unexpected return pc for net/http.(*Transport).roundTrip called from 0x22830
stack: frame={sp:0xc00002d6c0, fp:0xc00002d8a0} stack=[0xc00002c000,0xc00002e000)
0x000000c00002d5c0: 0x000000c000320380 0x000000c00002d620
0x000000c00002d5d0: 0x00000000004b1c3b <sync.(*Once).doSlow+0x000000000000009b> 0x0000000000000004
0x000000c00002d5e0: 0x030000c00011b658 0x00000000004b1d00 <sync.(*Once).doSlow.deferwrap1+0x0000000000000000>
0x000000c00002d5f0: 0x0000000003a01544 0x0000000003a01544
0x000000c00002d600: 0x00000000004b1cc0 <sync.(*Once).doSlow.deferwrap2+0x0000000000000000> 0x0000000003a01540
0x000000c00002d610: 0x000000c00002d5e8 0x000000c00002d600
0x000000c00002d620: 0x000000c00002d640 0x00000000007910ac <net/http.envProxyFunc+0x000000000000002c>
0x000000c00002d630: 0x0000000003a01540 0x000000000233e0d0
0x000000c00002d640: 0x000000c00002d658 0x000000000078ec58 <net/http.ProxyFromEnvironment+0x0000000000000018>
0x000000c00002d650: 0x0000000002240082 0x000000c00002d6b0
0x000000c00002d660: 0x0000000000791163 <net/http.(*Transport).connectMethodForRequest+0x0000000000000083> 0x000000c0005ca3c0
0x000000c00002d670: 0x0000000000000000 0x0000000000000000
0x000000c00002d680: 0x000000000224007b 0x0000000000000004
0x000000c00002d690: 0x000000c00039f430 0x000000000000000e
0x000000c00002d6a0: 0x0000000000000000 0x0000000000000000
0x000000c00002d6b0: 0x000000c00002d890 0x000000000078fac5 <net/http.(*Transport).roundTrip+0x0000000000000aa5>
0x000000c00002d6c0: <0x000000c000282900 0x000000c0004e2e60
0x000000c00002d6d0: 0x000000c000032660 0x0000000000000000
0x000000c00002d6e0: 0x0000000000000000 0x0000000000000000
0x000000c00002d6f0: 0x0000000000000000 0x0000000000000000
0x000000c00002d700: 0x0101000000000000 0x0000000000000004
0x000000c00002d710: 0x0000000000000000 0x0000000000000000
0x000000c00002d720: 0x0000000000000000 0x0000000000000000
0x000000c00002d730: 0x0000000002598770 0x000000c0005ca3c0
0x000000c00002d740: 0x0000000000000000 0x000000c0004e2e60
0x000000c00002d750: 0x0000000000000000 0x000000000224007b
0x000000c00002d760: 0x0000000000000000 0x0000000000000000
0x000000c00002d770: 0x000000c0005ca3c0 0x000000c0005ca3c0
0x000000c00002d780: 0x000000000078ee1a <net/http.(*Transport).alternateRoundTripper+0x00000000000000ba> 0x0000000000000010
0x000000c00002d790: 0x000000c00011b888 0x000000000224007b
0x000000c00002d7a0: 0x00000000007359f7 <net/http.timeBeforeContextDeadline+0x0000000000000057> 0x0000000000000000
0x000000c00002d7b0: 0x0000000000000000 0x000000c0004e2e10
0x000000c00002d7c0: 0xc207d3dffac20ba7 0x00000002551eb8ec
0x000000c00002d7d0: 0x00000000039dcdc0 0xc207d3e4fac25e3c
0x000000c00002d7e0: 0x00000006fd36d263 0x00000000039dcdc0
0x000000c00002d7f0: 0x000000c00015eae0 0x0000000000736007 <net/http.setRequestCancel+0x0000000000000427>
0x000000c00002d800: 0x0000000002584d20 0x000000c000282900
0x000000c00002d810: 0x00000000039dcdc0 0x00000000025987e0
0x000000c00002d820: 0x000000c0002dd180 0x0000000000000140
0x000000c00002d830: 0x0000000000790100 <net/http.(*Transport).roundTrip.func1+0x0000000000000000> 0x000000c00002d7a8
0x000000c00002d840: 0x000000c00015eae0 0x00000000007acd40 <net/http.(*Transport).onceSetNextProtoDefaults-fm+0x0000000000000000>
0x000000c00002d850: 0x000000c000282900 0x000000c00011b868
0x000000c00002d860: 0x000000c000000000 0x0000000000022af0
0x000000c00002d870: 0x000000c0004856c0 0x0000000000022a97
0x000000c00002d880: 0x0000000000490f00 <runtime.call1073741824+0x0000000000000060> 0x000000c00002d830
0x000000c00002d890: 0x000000c00002d8b0 0x0000000000022830
0x000000c00002d8a0: >0x000000c000282900 0x000000c0005ca3c0
0x000000c00002d8b0: 0x000000c00011ba90 0x00000000000220f0
0x000000c00002d8c0: 0x000000c0005ca3c0 0x0000000002584d20
0x000000c00002d8d0: 0x000000c000282900 0x0000000000000000
0x000000c00002d8e0: 0x0000000000000000 0x00000000039dcdc0
0x000000c00002d8f0: 0x0000000000485a30 <runtime.mapassign_faststr+0x00000000000007f0> 0x0000000000000150
0x000000c00002d900: 0x0000000002026080 0x000000c00044c401
0x000000c00002d910: 0x0000000000000010 0x0000000000000000
0x000000c00002d920: 0x0000000000000000 0x000000c00011b998
0x000000c00002d930: 0x0000000000483545 <runtime.mapiternext+0x0000000000000185> 0x000000c0004b9f80
0x000000c00002d940: 0xc207d3e4fac25e3c 0x00000006fd36d263
0x000000c00002d950: 0x000000c00011ba48 0x00000000004810b7 <runtime.mallocgc+0x0000000000000a37>
0x000000c00002d960: 0x00000000004810b7 <runtime.mallocgc+0x0000000000000a37> 0x0000000000000010
0x000000c00002d970: 0x000000c000500008 0x000000000212b4e0
0x000000c00002d980: 0x000000c00044e5d0 0x000000000044e5d0 <runtime.findRunnable+0x0000000000000050>
0x000000c00002d990: 0x000000000044c420 <runtime.forEachPInternal+0x0000000000000080> 0x000000c00011ba78
fatal error: unknown caller pc

runtime stack:
runtime.throw({0x22376a8?, 0x7f3ff57f9c70?})
runtime/panic.go:1067 +0x48 fp=0x7f3ff57f9b20 sp=0x7f3ff57f9af0 pc=0x4883e8
runtime.(*unwinder).next(...)
runtime/traceback.go:470
runtime.copystack(0xc0004856c0, 0x800000002?)
runtime/stack.go:944 +0x6f8 fp=0x7f3ff57f9c98 sp=0x7f3ff57f9b20 pc=0x4647f8
runtime.newstack()
runtime/stack.go:1126 +0x489 fp=0x7f3ff57f9dd8 sp=0x7f3ff57f9c98 pc=0x464fc9
runtime.morestack()
runtime/asm_amd64.s:621 +0x7a fp=0x7f3ff57f9de0 sp=0x7f3ff57f9dd8 pc=0x48f51a

goroutine 68 gp=0xc0004856c0 m=6 mp=0xc000500008 [copystack]:
runtime.mallocgc(0x10?, 0x1dd9f40?, 0x1?)
runtime/malloc.go:983 +0x100c fp=0xc00002d408 sp=0xc00002d400 pc=0x48168c
runtime.makeslice(0x0?, 0x7f40203e6f18?, 0x2572ef8?)
runtime/slice.go:116 +0x49 fp=0xc00002d430 sp=0xc00002d408 pc=0x48b029
strings.genSplit(...)
strings/strings.go:250
strings.Split({0x0, 0x0}, {0x2572ef8, 0x1})
strings/strings.go:306 +0x77 fp=0xc00002d490 sp=0xc00002d430 pc=0x546477
vendor/golang.org/x/net/http/httpproxy.(*config).init(0xc00031f880)
vendor/golang.org/x/net/http/httpproxy/proxy.go:212 +0xd7 fp=0xc00002d5a0 sp=0xc00002d490 pc=0x7315d7
vendor/golang.org/x/net/http/httpproxy.(*Config).ProxyFunc(...)
vendor/golang.org/x/net/http/httpproxy/proxy.go:122
net/http.envProxyFunc.func1()
net/http/transport.go:911 +0x79 fp=0xc00002d5d8 sp=0xc00002d5a0 pc=0x79c959
sync.(*Once).doSlow(0x3a01540, 0x233e0d0)
sync/once.go:76 +0x9b fp=0xc00002d630 sp=0xc00002d5d8 pc=0x4b1c3b
sync.(*Once).Do(...)
sync/once.go:67
net/http.envProxyFunc()
net/http/transport.go:910 +0x2c fp=0xc00002d650 sp=0xc00002d630 pc=0x7910ac
net/http.ProxyFromEnvironment(0xc0005ca3c0)
net/http/transport.go:448 +0x18 fp=0xc00002d668 sp=0xc00002d650 pc=0x78ec58
net/http.(*Transport).connectMethodForRequest(0xc000282900, 0xc0004e2e60)
net/http/transport.go:926 +0x83 fp=0xc00002d6c0 sp=0xc00002d668 pc=0x791163
net/http.(*Transport).roundTrip(0xc000282900, 0xc0005ca3c0)
net/http/transport.go:623 +0xaa5 fp=0xc00002d8a0 sp=0xc00002d6c0 pc=0x78fac5
net/http.(*Transport).RoundTrip(0xc0005ca3c0?, 0x2584d20?)
net/http/roundtrip.go:30 +0x13 fp=0xc00002d8c0 sp=0xc00002d8a0 pc=0x79e7d3
net/http.send(0xc0005ca280, {0x2584d20, 0xc000282900}, {0x0?, 0x0?, 0x39dcdc0?})
net/http/client.go:259 +0x5e4 fp=0xc00002daa0 sp=0xc00002d8c0 pc=0x735344
net/http.(*Client).send(0xc00044e510, 0xc0005ca280, {0x0?, 0x0?, 0x39dcdc0?})
net/http/client.go:180 +0x98 fp=0xc00002db18 sp=0xc00002daa0 pc=0x734bf8
net/http.(*Client).do(0xc00044e510, 0xc0005ca280)
net/http/client.go:725 +0x8bc fp=0xc00002dd08 sp=0xc00002db18 pc=0x736ebc
net/http.(*Client).Do(...)
net/http/client.go:590
go.elastic.co/apm/v2/transport.(*HTTPTransport).refreshMajorServerVersion(0xc00027d830, {0x25987e0, 0xc0002dd180})
go.elastic.co/apm/v2@v2.6.2/transport/http.go:611 +0x25a fp=0xc00002df48 sp=0xc00002dd08 pc=0xa261da
go.elastic.co/apm/v2/transport.(*HTTPTransport).MajorServerVersion(0x2598770?, {0x25987e0?, 0xc0002dd180?}, 0x0?)
go.elastic.co/apm/v2@v2.6.2/transport/http.go:600 +0x36 fp=0xc00002df70 sp=0xc00002df48 pc=0xa25f36
go.elastic.co/apm/v2.(*Tracer).maybeRefreshServerVersion(0xc00032b6c0, {0x2598770, 0xc0004e2d70}, 0x2540be400)
go.elastic.co/apm/v2@v2.6.2/tracer.go:1414 +0xd2 fp=0xc00002dfb0 sp=0xc00002df70 pc=0xa6e6b2
go.elastic.co/apm/v2.(*Tracer).loop.gowrap4()
go.elastic.co/apm/v2@v2.6.2/tracer.go:925 +0x2c fp=0xc00002dfe0 sp=0xc00002dfb0 pc=0xa6d4cc
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00002dfe8 sp=0xc00002dfe0 pc=0x491241
created by go.elastic.co/apm/v2.(*Tracer).loop in goroutine 66
go.elastic.co/apm/v2@v2.6.2/tracer.go:925 +0x465

goroutine 1 gp=0xc0000061c0 m=nil [semacquire]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:424 +0xce fp=0xc0004574c0 sp=0xc0004574a0 pc=0x48850e
runtime.goparkunlock(...)
runtime/proc.go:430
runtime.semacquire1(0xc00005c150, 0x0, 0x1, 0x0, 0x12)
runtime/sema.go:178 +0x8fd fp=0xc000457568 sp=0xc0004574c0 pc=0x45ed3d
sync.runtime_Semacquire(0x222e636?)
runtime/sema.go:71 +0x25 fp=0xc0004575a0 sp=0xc000457568 pc=0x48a305
sync.(*WaitGroup).Wait(0x0?)
sync/waitgroup.go:118 +0x48 fp=0xc0004575c8 sp=0xc0004575a0 pc=0x4b39c8

The Go Programming Language

golang.org/x/sync@v0.10.0/errgroup/errgroup.go:56 +0x25 fp=0xc0004575e8 sp=0xc0004575c8 pc=0xb3f2e5
github.com/elastic/apm-server/internal/beatcmd.(*Beat).Run(0xc00032b040, {0x2598770, 0xc0004e2640})
github.com/elastic/apm-server/internal/beatcmd/beat.go:396 +0xcf6 fp=0xc000457bb8 sp=0xc0004575e8 pc=0x1c0f396
github.com/elastic/apm-server/internal/beatcmd.genRunCmd.func1(0xc0001eab00?, {0x2224ca6?, 0x4?, 0x2224caa?})
github.com/elastic/apm-server/internal/beatcmd/run.go:47 +0xae fp=0xc000457c28 sp=0xc000457bb8 pc=0x1c1838e
github.com/spf13/cobra.(*Command).execute(0xc0003f0c08, {0xc000122330, 0x3, 0x3})
github.com/spf13/cobra@v1.8.1/command.go:985 +0xaaa fp=0xc000457db0 sp=0xc000457c28 pc=0x1aa8a0a
github.com/spf13/cobra.(*Command).ExecuteC(0xc0003f0f08)
github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff fp=0xc000457e88 sp=0xc000457db0 pc=0x1aa92df
github.com/spf13/cobra.(*Command).Execute(...)
github.com/spf13/cobra@v1.8.1/command.go:1041
main.Main()
github.com/elastic/apm-server/x-pack/apm-server/main.go:280 +0x32 fp=0xc000457f38 sp=0xc000457e88 pc=0x1c998d2
main.main()
github.com/elastic/apm-server/x-pack/apm-server/main.go:286 +0x13 fp=0xc000457f50 sp=0xc000457f38 pc=0x1c99ad3
runtime.main()
runtime/proc.go:272 +0x28b fp=0xc000457fe0 sp=0xc000457f50 pc=0x447ccb
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000457fe8 sp=0xc000457fe0 pc=0x491241

goroutine 2 gp=0xc000006700 m=nil [force gc (idle)]:
runtime.gopark(...)
runtime/proc.go:424
runtime.goparkunlock(0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:430 +0xcf fp=0xc000084fb0 sp=0xc000084f90 pc=0x44818f
runtime.forcegchelper()
runtime/proc.go:337 +0xaa fp=0xc000084fe0 sp=0xc000084fb0 pc=0x44800a
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000084fe8 sp=0xc000084fe0 pc=0x491241
created by runtime.init.7 in goroutine 1
runtime/proc.go:325 +0x1a

goroutine 3 gp=0xc000006c40 m=nil [GC sweep wait]:
runtime.gopark(...)
runtime/proc.go:424
runtime.goparkunlock(0x39ea101?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:430 +0xcf fp=0xc000085758 sp=0xc000085738 pc=0x44818f
runtime.bgsweep(0xc0000aa000)
runtime/mgcsweep.go:317 +0xcf fp=0xc0000857c8 sp=0xc000085758 pc=0x42b50f
runtime.gcenable.gowrap1()
runtime/mgc.go:204 +0x25 fp=0xc0000857e0 sp=0xc0000857c8 pc=0x41d5a5
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0000857e8 sp=0xc0000857e0 pc=0x491241
created by runtime.gcenable in goroutine 1
runtime/mgc.go:204 +0x66

goroutine 4 gp=0xc000006e00 m=nil [GC scavenge wait]:
runtime.gopark(...)
runtime/proc.go:424
runtime.goparkunlock(0x2572e60?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:430 +0xcf fp=0xc000085f80 sp=0xc000085f60 pc=0x44818f
runtime.(*scavengerState).park(0x39dcf40)
runtime/mgcscavenge.go:425 +0x45 fp=0xc000085fa8 sp=0xc000085f80 pc=0x428985
runtime.bgscavenge(0xc0000aa000)
runtime/mgcscavenge.go:658 +0x59 fp=0xc000085fc8 sp=0xc000085fa8 pc=0x428f19
runtime.gcenable.gowrap2()
runtime/mgc.go:205 +0x25 fp=0xc000085fe0 sp=0xc000085fc8 pc=0x41d545
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000085fe8 sp=0xc000085fe0 pc=0x491241
created by runtime.gcenable in goroutine 1
runtime/mgc.go:205 +0xa5

goroutine 17 gp=0xc000104380 m=nil [finalizer wait]:
runtime.gopark(0x13?, 0x0?, 0x68?, 0x1d?, 0x39dec00?)
runtime/proc.go:424 +0xce fp=0xc000084620 sp=0xc000084600 pc=0x48850e
runtime.runfinq()
runtime/mfinal.go:193 +0x107 fp=0xc0000847e0 sp=0xc000084620 pc=0x41c3e7
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0000847e8 sp=0xc0000847e0 pc=0x491241
created by runtime.createfing in goroutine 1
runtime/mfinal.go:163 +0x3d

goroutine 18 gp=0xc000334000 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x6?)
runtime/proc.go:424 +0xce fp=0xc0000806f8 sp=0xc0000806d8 pc=0x48850e
runtime.chanrecv(0xc000118690, 0x0, 0x1)
runtime/chan.go:639 +0x526 fp=0xc000080790 sp=0xc0000806f8 pc=0x40a5a6
runtime.chanrecv1(0x0?, 0x0?)
runtime/chan.go:489 +0x12 fp=0xc0000807b8 sp=0xc000080790 pc=0x40a052
runtime.unique_runtime_registerUniqueMapCleanup.func1(...)
runtime/mgc.go:1781
runtime.unique_runtime_registerUniqueMapCleanup.gowrap1()
runtime/mgc.go:1784 +0x2f fp=0xc0000807e0 sp=0xc0000807b8 pc=0x42080f
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0000807e8 sp=0xc0000807e0 pc=0x491241
created by unique.runtime_registerUniqueMapCleanup in goroutine 1
runtime/mgc.go:1779 +0x96

goroutine 6 gp=0xc000334fc0 m=nil [GC worker (idle)]:
runtime.gopark(0xc00036bb01?, 0x0?, 0xc0?, 0x4f?, 0xc000080fa8?)
runtime/proc.go:424 +0xce fp=0xc000080f38 sp=0xc000080f18 pc=0x48850e
runtime.gcBgMarkWorker(0xc0003fc0e0)
runtime/mgc.go:1412 +0xe9 fp=0xc000080fc8 sp=0xc000080f38 pc=0x41faa9
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1328 +0x25 fp=0xc000080fe0 sp=0xc000080fc8 pc=0x41f985
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000080fe8 sp=0xc000080fe0 pc=0x491241
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1328 +0x105

goroutine 33 gp=0xc000484380 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:424 +0xce fp=0xc0004a0738 sp=0xc0004a0718 pc=0x48850e
runtime.gcBgMarkWorker(0xc0003fc0e0)
runtime/mgc.go:1412 +0xe9 fp=0xc0004a07c8 sp=0xc0004a0738 pc=0x41faa9
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1328 +0x25 fp=0xc0004a07e0 sp=0xc0004a07c8 pc=0x41f985
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a07e8 sp=0xc0004a07e0 pc=0x491241
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1328 +0x105

goroutine 20 gp=0xc000335180 m=nil [GC worker (idle)]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:424 +0xce fp=0xc000081738 sp=0xc000081718 pc=0x48850e
runtime.gcBgMarkWorker(0xc0003fc0e0)
runtime/mgc.go:1412 +0xe9 fp=0xc0000817c8 sp=0xc000081738 pc=0x41faa9
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1328 +0x25 fp=0xc0000817e0 sp=0xc0000817c8 pc=0x41f985
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0000817e8 sp=0xc0000817e0 pc=0x491241
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1328 +0x105

goroutine 21 gp=0xc000335340 m=nil [GC worker (idle)]:
runtime.gopark(0x43ac2fcbc6933?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:424 +0xce fp=0xc000081f38 sp=0xc000081f18 pc=0x48850e
runtime.gcBgMarkWorker(0xc0003fc0e0)
runtime/mgc.go:1412 +0xe9 fp=0xc000081fc8 sp=0xc000081f38 pc=0x41faa9
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1328 +0x25 fp=0xc000081fe0 sp=0xc000081fc8 pc=0x41f985
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000081fe8 sp=0xc000081fe0 pc=0x491241
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1328 +0x105

goroutine 7 gp=0xc000007a40 m=nil [semacquire]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:424 +0xce fp=0xc0006cddf0 sp=0xc0006cddd0 pc=0x48850e
runtime.goparkunlock(...)
runtime/proc.go:430
runtime.semacquire1(0x39de03c, 0x0, 0x0, 0x0, 0x12)
runtime/sema.go:178 +0x8fd fp=0xc0006cde98 sp=0xc0006cddf0 pc=0x45ed3d
runtime.semacquire(...)
runtime/sema.go:129
runtime.gcMarkDone()
runtime/mgc.go:861 +0x32 fp=0xc0006cdf38 sp=0xc0006cde98 pc=0x41e1b2
runtime.gcBgMarkWorker(0xc0003fc0e0)
runtime/mgc.go:1548 +0x331 fp=0xc0006cdfc8 sp=0xc0006cdf38 pc=0x41fcf1
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1328 +0x25 fp=0xc0006cdfe0 sp=0xc0006cdfc8 pc=0x41f985
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0006cdfe8 sp=0xc0006cdfe0 pc=0x491241
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1328 +0x105

goroutine 8 gp=0xc000007c00 m=5 mp=0xc000480008 [GC mark termination]:
runtime.systemstack_switch()
runtime/asm_amd64.s:479 +0x8 fp=0xc0004a3e98 sp=0xc0004a3e88 pc=0x48f3a8
runtime.forEachP(...)
runtime/proc.go:2000
runtime.gcMarkDone()
runtime/mgc.go:885 +0x159 fp=0xc0004a3f38 sp=0xc0004a3e98 pc=0x41e2d9
runtime.gcBgMarkWorker(0xc0003fc0e0)
runtime/mgc.go:1548 +0x331 fp=0xc0004a3fc8 sp=0xc0004a3f38 pc=0x41fcf1
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1328 +0x25 fp=0xc0004a3fe0 sp=0xc0004a3fc8 pc=0x41f985
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a3fe8 sp=0xc0004a3fe0 pc=0x491241
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1328 +0x105

goroutine 9 gp=0xc000007dc0 m=nil [GC worker (idle)]:
runtime.gopark(0x43ac2fcbc6b42?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:424 +0xce fp=0xc000087738 sp=0xc000087718 pc=0x48850e
runtime.gcBgMarkWorker(0xc0003fc0e0)
runtime/mgc.go:1412 +0xe9 fp=0xc0000877c8 sp=0xc000087738 pc=0x41faa9
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1328 +0x25 fp=0xc0000877e0 sp=0xc0000877c8 pc=0x41f985
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0000877e8 sp=0xc0000877e0 pc=0x491241
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1328 +0x105

goroutine 34 gp=0xc000484540 m=nil [GC worker (idle)]:
runtime.gopark(0x43ac2fc17649e?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:424 +0xce fp=0xc0004a0f38 sp=0xc0004a0f18 pc=0x48850e
runtime.gcBgMarkWorker(0xc0003fc0e0)
runtime/mgc.go:1412 +0xe9 fp=0xc0004a0fc8 sp=0xc0004a0f38 pc=0x41faa9
runtime.gcBgMarkStartWorkers.gowrap1()
runtime/mgc.go:1328 +0x25 fp=0xc0004a0fe0 sp=0xc0004a0fc8 pc=0x41f985
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a0fe8 sp=0xc0004a0fe0 pc=0x491241
created by runtime.gcBgMarkStartWorkers in goroutine 1
runtime/mgc.go:1328 +0x105

goroutine 63 gp=0xc000335500 m=nil [select, locked to thread]:
runtime.gopark(...)
runtime/proc.go:424
runtime.selectgo(0xc0004a37a8, 0xc0004a3790, 0x0?, 0x0, 0xc0002dc930?, 0x1)
runtime/select.go:335 +0xc2b fp=0xc0004a3758 sp=0xc0004a3570 pc=0x45cbeb
runtime.ensureSigM.func1()
runtime/signal_unix.go:1077 +0x19f fp=0xc0004a37e0 sp=0xc0004a3758 pc=0x47cfdf
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0004a37e8 sp=0xc0004a37e0 pc=0x491241
created by runtime.ensureSigM in goroutine 1
runtime/signal_unix.go:1060 +0xc8

goroutine 64 gp=0xc000484a80 m=7 mp=0xc000590008 [syscall]:
runtime.notetsleepg(0x3a023a0, 0xffffffffffffffff)
runtime/lock_futex.go:246 +0x29 fp=0xc00049e7a0 sp=0xc00049e778 pc=0x410869
os/signal.signal_recv()
runtime/sigqueue.go:152 +0x29 fp=0xc00049e7c0 sp=0xc00049e7a0 pc=0x48ae49
os/signal.loop()
os/signal/signal_unix.go:23 +0x13 fp=0xc00049e7e0 sp=0xc00049e7c0 pc=0x1a789b3
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00049e7e8 sp=0xc00049e7e0 pc=0x491241
created by os/signal.Notify.func1.1 in goroutine 1
os/signal/signal.go:151 +0x1f

goroutine 23 gp=0xc000504380 m=nil [chan receive]:
runtime.gopark(0x0?, 0x0?, 0x0?, 0x0?, 0x0?)
runtime/proc.go:424 +0xce fp=0xc0001ac680 sp=0xc0001ac660 pc=0x48850e
runtime.chanrecv(0xc0002dca80, 0xc0001ac7b8, 0x1)
runtime/chan.go:639 +0x526 fp=0xc0001ac718 sp=0xc0001ac680 pc=0x40a5a6
runtime.chanrecv1(0x0?, 0x0?)
runtime/chan.go:489 +0x12 fp=0xc0001ac740 sp=0xc0001ac718 pc=0x40a052
github.com/elastic/elastic-agent-libs/service.HandleSignals.func1()
github.com/elastic/elastic-agent-libs@v0.17.5/service/service.go:50 +0x55 fp=0xc0001ac7e0 sp=0xc0001ac740 pc=0x1aba835
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0001ac7e8 sp=0xc0001ac7e0 pc=0x491241
created by github.com/elastic/elastic-agent-libs/service.HandleSignals in goroutine 1
github.com/elastic/elastic-agent-libs@v0.17.5/service/service.go:49 +0x156

goroutine 65 gp=0xc000504700 m=nil [chan receive]:
runtime.gopark(0x1ddeec0?, 0xc00049c6c8?, 0xdc?, 0x55?, 0xc00049c6c8?)
runtime/proc.go:424 +0xce fp=0xc00049c698 sp=0xc00049c678 pc=0x48850e
runtime.chanrecv(0xc000306000, 0x0, 0x1)
runtime/chan.go:639 +0x526 fp=0xc00049c730 sp=0xc00049c698 pc=0x40a5a6
runtime.chanrecv1(0xc0004e2a00?, 0xc00003e0f0?)
runtime/chan.go:489 +0x12 fp=0xc00049c758 sp=0xc00049c730 pc=0x40a052
github.com/elastic/apm-server/internal/beater.(*Runner).Run.func1()
github.com/elastic/apm-server/internal/beater/beater.go:166 +0x39 fp=0xc00049c7e0 sp=0xc00049c758 pc=0x181afd9
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00049c7e8 sp=0xc00049c7e0 pc=0x491241
created by github.com/elastic/apm-server/internal/beater.(*Runner).Run in goroutine 27
github.com/elastic/apm-server/internal/beater/beater.go:165 +0x186

goroutine 26 gp=0xc0005048c0 m=nil [select]:
runtime.gopark(...)
runtime/proc.go:424
runtime.selectgo(0xc00011aee0, 0xc00002bec0, 0x0?, 0x0, 0x0?, 0x1)
runtime/select.go:335 +0xc2b fp=0xc00011ae88 sp=0xc00011aca0 pc=0x45cbeb
github.com/elastic/apm-server/internal/beatcmd.adjustMaxProcs({0x2598770, 0xc00003e0f0}, 0x6fc23ac00, 0xc0005120f0)
github.com/elastic/apm-server/internal/beatcmd/maxprocs.go:54 +0x196 fp=0xc00011af48 sp=0xc00011ae88 pc=0x1c16036
github.com/elastic/apm-server/internal/beatcmd.(*Beat).Run.func2()
github.com/elastic/apm-server/internal/beatcmd/beat.go:348 +0x29 fp=0xc00011af78 sp=0xc00011af48 pc=0x1c0f8a9
golang.org/x/sync/errgroup.(*Group).Go.func1()
golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50 fp=0xc00011afe0 sp=0xc00011af78 pc=0xb3f450
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc00011afe8 sp=0xc00011afe0 pc=0x491241
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96

goroutine 27 gp=0xc000504a80 m=nil [chan send]:
runtime.gopark(0x43f885?, 0x233dd78?, 0x28?, 0x32?, 0x48ebcd?)
runtime/proc.go:424 +0xce fp=0xc000113198 sp=0xc000113178 pc=0x48850e
runtime.chansend(0xc000118af0, 0xc0001132f0, 0x1, 0x0?)
runtime/chan.go:270 +0x38d fp=0xc000113208 sp=0xc000113198 pc=0x4094ad
runtime.chansend1(0xc00032b6c0?, 0xc0001132f8?)
runtime/chan.go:156 +0x17 fp=0xc000113238 sp=0xc000113208 pc=0x409117
go.elastic.co/apm/v2.newTracer({{0xc00039f420, 0xa}, {0x0, 0x0}, {0x0, 0x0}, {0x2585260, 0xc00027d830}, 0x2540be400, 0x6fc23ac00, ...})
go.elastic.co/apm/v2@v2.6.2/tracer.go:539 +0x1085 fp=0xc0001133d8 sp=0xc000113238 pc=0xa665e5
go.elastic.co/apm/v2.DefaultTracer()
go.elastic.co/apm/v2@v2.6.2/tracer.go:75 +0x198 fp=0xc000113728 sp=0xc0001133d8 pc=0xa63918
github.com/elastic/beats/v7/libbeat/instrumentation.(*instrumentation).Tracer(0xc0002f6de0?)
github.com/elastic/beats/v7@v7.0.0-alpha2.0.20250108165006-190661702823/libbeat/instrumentation/instrumentation.go:53 +0x1b fp=0xc000113750 sp=0xc000113728 pc=0xbe23db
github.com/elastic/apm-server/internal/beater.(*Runner).Run(0xc00003f040, {0x2598770, 0xc00003e0f0})
github.com/elastic/apm-server/internal/beater/beater.go:269 +0x9f8 fp=0xc000113f50 sp=0xc000113750 pc=0x1817a98
github.com/elastic/apm-server/internal/beatcmd.(*Beat).Run.func4()
github.com/elastic/apm-server/internal/beatcmd/beat.go:393 +0x24 fp=0xc000113f78 sp=0xc000113f50 pc=0x1c0f7c4
golang.org/x/sync/errgroup.(*Group).Go.func1()
golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78 +0x50 fp=0xc000113fe0 sp=0xc000113f78 pc=0xb3f450
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000113fe8 sp=0xc000113fe0 pc=0x491241
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 1
golang.org/x/sync@v0.10.0/errgroup/errgroup.go:75 +0x96

goroutine 66 gp=0xc000485340 m=nil [semacquire]:
runtime.gopark(0xc000121690?, 0x41874b?, 0x80?, 0x5f?, 0x48f3f2?)
runtime/proc.go:424 +0xce fp=0xc0000275b8 sp=0xc000027598 pc=0x48850e
runtime.goparkunlock(...)
runtime/proc.go:430
runtime.semacquire1(0x39de03c, 0x0, 0x0, 0x0, 0x12)
runtime/sema.go:178 +0x8fd fp=0xc000027660 sp=0xc0000275b8 pc=0x45ed3d
runtime.semacquire(...)
runtime/sema.go:129
runtime.gcMarkDone()
runtime/mgc.go:861 +0x32 fp=0xc000027700 sp=0xc000027660 pc=0x41e1b2
runtime.gcAssistAlloc(0xc000485340)
runtime/mgcmark.go:552 +0x245 fp=0xc000027760 sp=0xc000027700 pc=0x422ac5
runtime.deductAssistCredit(...)
runtime/malloc.go:1349
runtime.mallocgc(0x100000, 0x1dd9fc0, 0x1)
runtime/malloc.go:1037 +0x128 fp=0xc000027858 sp=0xc000027760 pc=0x4807a8
runtime.makeslice(0x2224c26?, 0x4?, 0x7f4020067040?)
runtime/slice.go:116 +0x49 fp=0xc000027880 sp=0xc000027858 pc=0x48b029
go.elastic.co/apm/v2/internal/ringbuffer.New(...)
go.elastic.co/apm/v2@v2.6.2/internal/ringbuffer/buffer.go:57
go.elastic.co/apm/v2.(*Tracer).loop(0xc00032b6c0)
go.elastic.co/apm/v2@v2.6.2/tracer.go:957 +0x665 fp=0xc000027fc8 sp=0xc000027880 pc=0xa6a4e5
go.elastic.co/apm/v2.newTracer.gowrap1()
go.elastic.co/apm/v2@v2.6.2/tracer.go:538 +0x25 fp=0xc000027fe0 sp=0xc000027fc8 pc=0xa667e5
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc000027fe8 sp=0xc000027fe0 pc=0x491241
created by go.elastic.co/apm/v2.newTracer in goroutine 27
go.elastic.co/apm/v2@v2.6.2/tracer.go:538 +0x1013

goroutine 67 gp=0xc000485500 m=nil [chan receive]:
runtime.gopark(0x1500?, 0xc0004c87e0?, 0x0?, 0x0?, 0x7f40203e6108?)
runtime/proc.go:424 +0xce fp=0xc0001ade08 sp=0xc0001adde8 pc=0x48850e
runtime.chanrecv(0xc000118c40, 0xc0001adf18, 0x1)
runtime/chan.go:639 +0x526 fp=0xc0001adea0 sp=0xc0001ade08 pc=0x40a5a6
runtime.chanrecv2(0x3981f30?, 0x1f54d20?)
runtime/chan.go:494 +0x12 fp=0xc0001adec8 sp=0xc0001adea0 pc=0x40a072
go.elastic.co/apm/v2.(*Tracer).loop.func2()
go.elastic.co/apm/v2@v2.6.2/tracer.go:910 +0x1f8 fp=0xc0001adfe0 sp=0xc0001adec8 pc=0xa6d758
runtime.goexit({})
runtime/asm_amd64.s:1700 +0x1 fp=0xc0001adfe8 sp=0xc0001adfe0 pc=0x491241
created by go.elastic.co/apm/v2.(*Tracer).loop in goroutine 66
go.elastic.co/apm/v2@v2.6.2/tracer.go:907 +0x378

[stephenb (Stephen Brown)]

@sruthi.sattiraju

Hi @stephenb thanks for the inputs. As a first step, I am trying to upgrade APMServer to 18.17.1 but its failing. I am attaching log here. Can you please review and suggest.

Exactly how are you upgrading?

I am not familiar with that error

[sruthi.sattiraju (Sruthi)]

Hi @stephenb after upgrading apmserver to 8.18.0 I am able to see the otel agent data in Kibana under Infrastructure->hosts page.

Thanks for you help.

[stephenb (Stephen Brown)]

Curious what the problem with the install was... Would be nice if you let us know in case someone else runs into the same problem

Glad it's working!