Not able to see APM data in Kibana

Elastic Observability APM
1

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

Kibana version: 6.6.1 (ElasticCloud)

Elasticsearch version: 6.6.1 (ElasticCloud)

APM Server version: 6.6.1 (ElasticCloud)

APM Agent language and version: Golang agent.

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version?

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
From my application log:

 {"level":"debug","time":"2019-07-10T12:35:23+10:00","message":"sent request with 3 transactions, 43 spans, 0 errors, 1 metricset"}

From monitoring infra dashboard in Kibana:

04%20pm
Screen Shot 2019-07-10 at 12.39.04 pm.png1546×622 76.3 KB

But I can't see anything in APM tab:
No services were found

2

The fact that events are getting sent is a good start... and it seems from the monitoring dashboard they're being happily processed by the server.

I guess the data is in Elasticsearch and it's something to do with the query being made.

  • Is there a steady flow of events? What time range are you using in the APM UI?
  • Can you see any APM docs for the "apm-*" index pattern in Discover using the same time range?
3

The APM UI time range is 4 hour. I can see the events when I start/stop my local application. This local application is running with http in my localhost dev.

I can't see any data.
When I go to Setting > Index management, there is exactly 100 indexes starts by apm-* . I am not sure is there is any upper limit of number of apm-* indexes. Thus there is no further data got indexed.

4

Sorry, I don't understand this bit. Can you please rephrase?

You can have more than 100 indices, though that does seem like an unusually "round" number. The indices should have the date in them. What's the most recent apm- index name?

5

I can see the events when I start/stop my local application.

I see the fluctuation of graph is matching to the time i start to run my application. So I am sure apm server receive metrics data.

What's the most recent apm- index name?
The most recent apm-* name is some date in April:

  • apm-6.6.0-metric-2019.04.14
  • apm-6.6.0-transaction-2019.04.08
  • apm-6.6.0-metric-2019.02.15
    ....
6

OK, that's interesting. What does the "Output Failed Events Rate" graph look like? If there are no current indices, that would suggest indexing into Elasticsearch is failing.

The APM server will accept events and send them asynchronously to Elasticsearch, so it's possible to have "Ok" or "Accepted" responses like in your above graph, but for the documents to then fail to be written to Elasticsearch.

7

Thanks for the hint. When I look at "Output Failed Events Rate" graph(12 hours window), it has failed and dropped rate like this:

25%20pm
Screen Shot 2019-07-10 at 4.58.25 pm.png1488×606 72.3 KB

8

If you're unable to identify the cause of that, I'd suggest opening an issue at https://cloud.elastic.co/help.

9

Thanks @axw. I finally find out the root cause.
The new index doesn't automatically created because there is a setting in Elastic Cloud auto_create_index = false.
After I change it to true, new indexes of apm-* can be created now.

1 Like
10

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.