No services found in kibana services tab

Elastic Observability APM
,
1

Hi everyone,

I have upgraded the ELK stack on my server from version 8.1.1 to 8.6.2. After upgrading when I went to see the services running in Kibana, no services were visible there also in traces tab no traces were collected even though all my application are running with apm agent attached.

Also I tried removing the apm integration and adding it again. While checking status for apm-server, I got it has been setup correctly .

Kindly guide me where I'm going wrong, and what can be the possible issues.

Thanks.

2

Hey @APandey, couple of questions so we can identify the cause here:

  • Did any services show up before?
  • What agents are you using? (what language, but also whether you are using Elastic APM agents or OpenTelemetry agents)
  • Do you see any errors or warnings in either the agent logs or APM Server logs?
  • Do you see any APM data in the APM data streams? The easiest way to check this is in Discover (select the APM data view), but you can also run some searches:
GET *apm*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "now-30m"
            }
          }
        },
        {
          "term": {
            "processor.event": "transaction"
          }
        }
      ]
    }
  }
}

GET *apm*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "now-30m"
            }
          }
        },
        {
          "term": {
            "metricset.name": "transaction"
          }
        }
      ]
    }
  }
}

A screenshot or copy/paste from the responses would be helpful.

4

Hi Dario,

Yes before upgrading to v8.6.2 services were there.
We are using Java APM Agent v1.36.1
There are no errors in agent logs or APM server logs, also in apm server we can see the requests with status code 202 for each services.
In Discover (APM data view) no data is present and I also tried running this query which returned empty data as response.

{
  "took": 8,
  "timed_out": false,
  "_shards": {
    "total": 89,
    "successful": 89,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

One more thing the default index that is created for storing traces '.ds-traces...' that is also missing.
I'm on windows environment and have used snapshot and restore option to restore old data.

5

Hi @APandey could you please provide the APM Server logs?

6

Hi @marclop,

Here are the logs of APM server

D:\ELK_v8.6.2\apm-server-8.6.2-windows-x86_64\apm-server-8.6.2-windows-x86_64>apm-server -c apm-server.yml -e
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.035+0530","log.origin":{"file.name":"beatcmd/beat.go","file.line":138},"message":"Home path: [D:\\ELK_v8.6.2\\apm-server-8.6.2-windows-x86_64\\apm-server-8.6.2-windows-x86_64] Config path: [D:\\ELK_v8.6.2\\apm-server-8.6.2-windows-x86_64\\apm-server-8.6.2-windows-x86_64] Data path: [D:\\ELK_v8.6.2\\apm-server-8.6.2-windows-x86_64\\apm-server-8.6.2-windows-x86_64\\data] Logs path: [D:\\ELK_v8.6.2\\apm-server-8.6.2-windows-x86_64\\apm-server-8.6.2-windows-x86_64\\logs]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.037+0530","log.origin":{"file.name":"beatcmd/beat.go","file.line":145},"message":"Beat ID: 1e158870-c7fb-4337-8f87-51cc78f78062","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.040+0530","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":573},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"D:\\ELK_v8.6.2\\apm-server-8.6.2-windows-x86_64\\apm-server-8.6.2-windows-x86_64","data":"D:\\ELK_v8.6.2\\apm-server-8.6.2-windows-x86_64\\apm-server-8.6.2-windows-x86_64\\data","home":"D:\\ELK_v8.6.2\\apm-server-8.6.2-windows-x86_64\\apm-server-8.6.2-windows-x86_64","logs":"D:\\ELK_v8.6.2\\apm-server-8.6.2-windows-x86_64\\apm-server-8.6.2-windows-x86_64\\logs"},"type":"apm-server","uuid":"1e158870-c7fb-4337-8f87-51cc78f78062"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.040+0530","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":581},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"8638b035d700e5e85e376252402b5375e4d4190b","time":"2023-02-13T13:01:54.000Z","version":"8.6.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.040+0530","log.origin":{"file.name":"beatcmd/maxprocs.go","file.line":68},"message":"maxprocs: Leaving GOMAXPROCS=4: CPU quota undefined","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.041+0530","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":584},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"windows","arch":"amd64","max_procs":4,"version":"go1.18.10"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.071+0530","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":588},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-03-30T12:25:32+05:30","name":"cldx-1103-1217","ip":["localhost/23","::1/128","127.0.0.1/8","fe80::5efe:ac8:f89b/128"],"kernel_version":"10.0.14393.5786 (rs1_release.230308-2129)","mac":["00:50:56:b7:0d:d7","00:00:00:00:00:00:00:e0"],"os":{"type":"windows","family":"windows","platform":"windows","name":"Windows Server 2016 Standard","version":"10.0","major":10,"minor":0,"patch":0,"build":"14393.5786"},"timezone":"IST","timezone_offset_sec":19800,"id":"69e6e870-a351-460a-83a8-363a6abd17cd"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.071+0530","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":617},"message":"Process info","service.name":"apm-server","system_info":{"process":{"cwd":"D:\\ELK_v8.6.2\\apm-server-8.6.2-windows-x86_64\\apm-server-8.6.2-windows-x86_64","exe":"D:\\ELK_v8.6.2\\apm-server-8.6.2-windows-x86_64\\apm-server-8.6.2-windows-x86_64\\apm-server.exe","name":"apm-server.exe","pid":1000,"ppid":12364,"start_time":"2023-04-03T15:54:31.952+0530"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.081+0530","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":142},"message":"Listening on: localhost:8200","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.082+0530","log.origin":{"file.name":"beatcmd/beat.go","file.line":391},"message":"apm-server started.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.088+0530","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":195},"message":"no cgroups detected, falling back to total system memory","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.088+0530","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":214},"message":"MaxConcurrentDecoders set to 1023 based on 8.0gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-04-03T15:54:32.131+0530","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.134+0530","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":40},"message":"blocking ingestion until all preconditions are satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.134+0530","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":701},"message":"modelindexer.EventBufferSize set to 8191 based on 8.0gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.135+0530","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":715},"message":"modelindexer.MaxRequests set to 21 based on 8.0gb of memory","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.138+0530","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":104},"message":"creating transaction metrics aggregation with config: {Interval:1m0s MaxTransactionGroups:10000 HDRHistogramSignificantFigures:2}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.143+0530","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":119},"message":"creating service destinations aggregation with config: {Interval:1m0s MaxGroups:10000}","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.144+0530","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.145+0530","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.148+0530","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.162+0530","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.164+0530","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.165+0530","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.167+0530","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/traces added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.169+0530","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/metrics added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.170+0530","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/logs added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.170+0530","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":201},"message":"Starting apm-server [8638b035d700e5e85e376252402b5375e4d4190b built 2023-02-13 13:01:54 +0000 +0000]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.168+0530","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":68},"message":"no longer blocking ingestion as all precondition checks are now satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.171+0530","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":94},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-04-03T15:54:32.173+0530","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":104},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
7

@APandey thanks but those logs aren't useful since they span 200 milliseconds and include the server initialization.

If your APM Server is running as a daemon, it should be logging to a .ndjson file. If you could package a few of those .ndjson files, and upload them so we can have a look and make sure there aren't any obvious errors that would be useful.

8

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.