No field found for [context.service.name] in mapping with types []

[StianOvrevage]

After upgrading ElasticSearch, APM-server and APM-agent-nodejs, Kibana fails to retrieve docs from ES with error "Courier Fetch: 5 of 36 shards failed.".

Using devtools I see this reply from _msearch:

{"responses":[{"took":334,"timed_out":false,"_shards":{"total":37,"successful":32,"skipped":0,"failed":5,"failures":[{"shard":0,"index":"apm-6.1.0-2018.02.09","node":"Tyn8Z0joSUGchl-STaXGaA","reason":{"type":"script_exception","reason":"runtime error","script_stack":["org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:81)","org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)","doc['context.service.name'].value"," ^---- HERE"],"script":"doc['context.service.name'].value","lang":"painless","caused_by":{"type":"illegal_argument_exception","reason":"No field found for [context.service.name] in mapping with types []"}}}]},"hits":{"total":9390,"max_score":null,"hits":[{"_index":"apm-6.2.1-2018.02.09","_type":"doc","_id":"TKtRfGEBrafOyGmJT6eh","_version":1,"_score":null,"_source":{"listening":"0.0.0.0:8200","@timestamp":"2018-02-09T20:45:57.582Z","beat":{"hostname":"cust-k30-apm-server-d8d448f74-m8jm2","name":"cust-k30-apm-server-d8d448f74-m8jm2","version":"6.2.1"}},"fields":{"view errors":[null],"error id icon":[null],"@timestamp":["2018-02-09T20:45:57.582Z"]},"sort":[1518209157582]}]},"aggregations":{"2":{"buckets":[{"key_as_string":"2018-02-08T16:00:00.000-05:00","key":1518123600000,"doc_count":909},{"key_as_string":"2018-02-08T16:30:00.000-05:00","key":1518125400000,"doc_count":1267},{"key_as_string":"2018-02-08T17:00:00.000-05:00","key":1518127200000,"doc_count":1097},{"key_as_string":"2018-02-08T17:30:00.000-05:00","key":1518129000000,"doc_count":1123},{"key_as_string":"2018-02-08T18:00:00.000-05:00","key":1518130800000,"doc_count":1098},{"key_as_string":"2018-02-08T18:30:00.000-05:00","key":1518132600000,"doc_count":1094},{"key_as_string":"2018-02-09T14:00:00.000-05:00","key":1518202800000,"doc_count":42},{"key_as_string":"2018-02-09T14:30:00.000-05:00","key":1518204600000,"doc_count":1090},{"key_as_string":"2018-02-09T15:00:00.000-05:00","key":1518206400000,"doc_count":1096},{"key_as_string":"2018-02-09T15:30:00.000-05:00","key":1518208200000,"doc_count":574}]}},"status":200}]}

[sqren (Søren Louv Jansen)]

Hi Stian,

Thanks for your interest in APM!

There were some breaking changes in apm-server between 6.1 and 6.2. It seems your indexes were created with 6.1, which is not compatible with Kibana 6.2.

Since 6.1 was a beta release for us, we don't provide a migration strategy. The easiest solution is to delete your apm* indexes (note: you will lose all apm-related data) - this will cause them to be re-created with the new index mappings, and everything should work from there.

I hope this resolves the issue. Let me know if you have other questions.