APM transactions not showing in Kibana after upgrade from 7.5.0 to 7.5.2

This is to troubleshoot this issue in GitHub - You can find more information here: https://github.com/elastic/apm/issues/211

Kibana version: 7.5.2

Elasticsearch version: 7.5.2

APM Server version: 7.5.2

APM Agent language and version: elastic-apm-agent-1.12.0.jar

Browser version: Google Chrome Version 80.0.3987.87 (Official Build) (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: yum install apm-server [From 7.x repository]

Fresh install or upgraded from other version?: Upgraded APM from 7.5.0 to 7.5.2

Is there anything special in your setup?

  • I use Logstash to send events to from APM;
  • I have a load balancer in front of APM;
  • I did not change any configuration. At 7.5.0 I was seeing events in the APM app in Kibana, after the upgrade to 7.5.2 I wasn't seeing anything.
  • One thing I tried is to run setup from APM server to create the index templates to try to resolve the issue - it didn't work.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:

  1. yum install -y apm-server
  2. systemctl restart apm-server

Errors in browser console (if relevant):
I do not see errors, but those three API calls take multiple seconds to finish:

https://kibana.example.com/api/apm/ui_filters/environments?start=2020-02-12T17%3A47%3A44.903Z&end=2020-02-13T17%3A47%3A44.904Z

https://kibana.example.com/api/apm/ui_filters/local_filters/services?uiFilters={}&start=2020-02-12T17%3A47%3A44.903Z&end=2020-02-13T17%3A47%3A44.904Z&filterNames=["host"%2C"agentName"]

https://kibana.example.com/api/apm/services?start=2020-02-12T17%3A47%3A44.903Z&end=2020-02-13T17%3A47%3A44.904Z&uiFilters={}

Provide logs and/or server output (if relevant):

  • The agent log shows everything being fine (nothing changed here anyway):
2020-02-10 10:53:35.579 [apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: {  "build_date": "2020-01-15T10:22:43Z",  "build_sha": "0683f8094f42fecedd54cee09c0991febd8ce59a",  "version": "7.5.2"}
2020-02-10 10:53:35.668 [apm-remote-config-poller] INFO co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Received new configuration from APM Server: {}
  • In the Elasticsearch logs, I see the following events being repeated over and over (with a bunch of active gc.log):
[2020-02-12T21:32:28,003][DEBUG][o.e.a.s.TransportSearchAction] [master-node-1] [apm-7.5.2][0], node[FYBwRUMpTpm2yZs6FPNnXg], [P], s[STARTED], a[id=MaHC49h9RamInCFnwUuyEA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[apm-*, apm-*, apm-*], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":0,"query":{"bool":{"filter":[{"terms":{"processor.event":["transaction","error","metric"],"boost":1.0}},{"range":{"@timestamp":{"from":1581471148098,"to":1581557548098,"include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"range":{"observer.version_major":{"from":7,"to":null,"include_lower":true,"include_upper":true,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"environments":{"terms":{"field":"service.environment","missing":"ENVIRONMENT_NOT_DEFINED","size":10,"min_doc_count":1,"shard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]}}}}}]
org.elasticsearch.transport.RemoteTransportException: [master-node-1][172.31.20.22:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [host.hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
org.elasticsearch.transport.RemoteTransportException: [master-node-1][172.31.20.22:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [service.name] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
org.elasticsearch.transport.RemoteTransportException: [master-node-1][172.31.20.22:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [service.environment] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
  • GET apm-*/_mapping/field/service.name yields this in the dev tools (truncated for clarity):
  "apm-7.5.2" : {
    "mappings" : {
      "service.name" : {
        "full_name" : "service.name",
        "mapping" : {
          "name" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          }
        }
      }
    }
  },

  "apm-7.5.0" : {
    "mappings" : {
      "service.name" : {
        "full_name" : "service.name",
        "mapping" : {
          "name" : {
            "type" : "keyword",
            "ignore_above" : 1024
          }
        }
      }
    }
  },

Hi there,

Thanks for the info, the 7.5.2 mapping definitely looks wrong - service.name type should not be "text".

Can you stop the apm-server, run apm-server setup -e, and then start it again and query the mapping once more? That should fix it for you.

Hope it helps,

Juan

Hello Juan,

Thank you for the prompt response. I don't understand clearly how I fixed the issue. Before I saw your update, I was able to fix the issue myself, and those are the steps I followed:

  1. Stop APM & Logstash from sending APM events to Elasticsearch;
  2. Deleted the "apm-7.5.2" index, index template & index pattern;
  3. From the "APM" app in Kibana, under "Setup Instructions" in clicked on "Load Kibana objects";
  4. I ran the following setup command:
apm-server setup -e -E output.logstash.enabled=false -E output.elasticsearch.hosts=['elastic.example.com:9200'] -E setup.kibana.host=kibana.example.com:5601
  1. Started Logstash & APM back up;
  2. My APM metrics were now showing in the Kibana Dashboard.

That leads me to more questions...

  1. At first, when I tried to fix the issue, I was trying to run command:
apm-server setup --index-management -E apm-server.ilm.enabled=true -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["elastic.example.com:9200"]' -c /etc/apm-server/apm-server.yml

... how does that command differ from the other one?

  1. Why do I get this issue in the first place? The error I reported about was after an upgrade from 7.5.0 to 7.5.2, but I just recently installed a production environment from scratch which installed 7.6.0 directly and I had the same index issue. It's causing a bit of grief when we are trying to automate things and end up realizing that we have to do manual steps after every upgrade.

Thank you for your assistance.
Joey

Hi,

If you are using Logstash, you always need to run the setup command on upgrades and fresh installs, as Logstash doesn't know what template APM needs.
You would need to automate that.

As for the setup subcomand you paste, I am not sure what is wrong ... for what I can see, it should work.
For next time, I recommend always to pass -e so you can see if there are any errors.

I hope this helps!

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.