This is to troubleshoot this issue in GitHub - You can find more information here: https://github.com/elastic/apm/issues/211
Kibana version: 7.5.2
Elasticsearch version: 7.5.2
APM Server version: 7.5.2
APM Agent language and version: elastic-apm-agent-1.12.0.jar
Browser version: Google Chrome Version 80.0.3987.87 (Official Build) (64-bit)
Original install method (e.g. download page, yum, deb, from source, etc.) and version: yum install apm-server [From 7.x repository]
Fresh install or upgraded from other version?: Upgraded APM from 7.5.0 to 7.5.2
Is there anything special in your setup?
- I use Logstash to send events to from APM;
- I have a load balancer in front of APM;
- I did not change any configuration. At 7.5.0 I was seeing events in the APM app in Kibana, after the upgrade to 7.5.2 I wasn't seeing anything.
- One thing I tried is to run setup from APM server to create the index templates to try to resolve the issue - it didn't work.
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
Steps to reproduce:
- yum install -y apm-server
- systemctl restart apm-server
Errors in browser console (if relevant):
I do not see errors, but those three API calls take multiple seconds to finish:
Provide logs and/or server output (if relevant):
- The agent log shows everything being fine (nothing changed here anyway):
2020-02-10 10:53:35.579 [apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: { "build_date": "2020-01-15T10:22:43Z", "build_sha": "0683f8094f42fecedd54cee09c0991febd8ce59a", "version": "7.5.2"}
2020-02-10 10:53:35.668 [apm-remote-config-poller] INFO co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Received new configuration from APM Server: {}
- In the Elasticsearch logs, I see the following events being repeated over and over (with a bunch of active gc.log):
[2020-02-12T21:32:28,003][DEBUG][o.e.a.s.TransportSearchAction] [master-node-1] [apm-7.5.2][0], node[FYBwRUMpTpm2yZs6FPNnXg], [P], s[STARTED], a[id=MaHC49h9RamInCFnwUuyEA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[apm-*, apm-*, apm-*], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":0,"query":{"bool":{"filter":[{"terms":{"processor.event":["transaction","error","metric"],"boost":1.0}},{"range":{"@timestamp":{"from":1581471148098,"to":1581557548098,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"range":{"observer.version_major":{"from":7,"to":null,"include_lower":true,"include_upper":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"environments":{"terms":{"field":"service.environment","missing":"ENVIRONMENT_NOT_DEFINED","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}]
org.elasticsearch.transport.RemoteTransportException: [master-node-1][172.31.20.22:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [host.hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
org.elasticsearch.transport.RemoteTransportException: [master-node-1][172.31.20.22:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [service.name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
org.elasticsearch.transport.RemoteTransportException: [master-node-1][172.31.20.22:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [service.environment] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
- GET apm-*/_mapping/field/service.name yields this in the dev tools (truncated for clarity):
"apm-7.5.2" : {
"mappings" : {
"service.name" : {
"full_name" : "service.name",
"mapping" : {
"name" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
},
"apm-7.5.0" : {
"mappings" : {
"service.name" : {
"full_name" : "service.name",
"mapping" : {
"name" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}
}
},