No services found after upgrading APM

ggio · August 26, 2021, 10:15am

Kibana version:
7.14
Elasticsearch version:
7.14
APM Server version:
7.14
Fresh install or upgraded from other version?
Upgraded from 7.7.0

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
After upgrading APM server to version 7.14, I'm not able to see any service in Kibana APM UI.
Kibana reports "No services found".
From Elasticsearch log I see these messages:

[2021-08-26T12:10:26,196][DEBUG][o.e.a.a.i.c.AutoCreateAction] [elk1] Auto-creating index apm-7.14.0-span-2021.08.26
[2021-08-26T12:10:26,202][INFO ][o.e.c.m.MetadataCreateIndexService] [elk1] failed on parsing mappings on index creation [apm-7.14.0-span-2021.08.26]
org.elasticsearch.index.mapper.MapperParsingException: Failed to parse mapping [_doc]: unknown parameter [path] on mapper [image] of type [keyword]
        at org.elasticsearch.index.mapper.MapperService.parseMappings(MapperService.java:437) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.MapperService.mergeAndApplyMappings(MapperService.java:353) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.MapperService.merge(MapperService.java:335) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.updateIndexMappingsAndBuildSortOrder(MetadataCreateIndexService.java:1046) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.lambda$applyCreateIndexWithTemporaryService$3(MetadataCreateIndexService.java:409) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.indices.IndicesService.withTempIndexService(IndicesService.java:631) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexWithTemporaryService(MetadataCreateIndexService.java:407) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequestWithV1Templates(MetadataCreateIndexService.java:493) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:370) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:377) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.action.admin.indices.create.AutoCreateAction$TransportAction$1.execute(AutoCreateAction.java:175) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.ClusterStateUpdateTask.execute(ClusterStateUpdateTask.java:48) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:691) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.MasterService.calculateTaskOutputs(MasterService.java:313) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.MasterService.runTasks(MasterService.java:208) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.MasterService.access$000(MasterService.java:62) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.MasterService$Batcher.run(MasterService.java:140) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.TaskBatcher.runIfNotProcessed(TaskBatcher.java:139) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.cluster.service.TaskBatcher$BatchedTask.run(TaskBatcher.java:177) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:241) [elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:204) [elasticsearch-7.14.0.jar:7.14.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:831) [?:?]
Caused by: org.elasticsearch.index.mapper.MapperParsingException: unknown parameter [path] on mapper [image] of type [keyword]
        at org.elasticsearch.index.mapper.FieldMapper$Builder.parse(FieldMapper.java:1165) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.FieldMapper$TypeParser.parse(FieldMapper.java:1246) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.FieldMapper$TypeParser.parse(FieldMapper.java:1221) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:325) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:242) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:212) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:325) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:242) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parse(ObjectMapper.java:212) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseProperties(ObjectMapper.java:325) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.ObjectMapper$TypeParser.parseObjectOrDocumentTypeProperties(ObjectMapper.java:242) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.RootObjectMapper$TypeParser.parse(RootObjectMapper.java:153) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.MappingParser.parse(MappingParser.java:110) ~[elasticsearch-7.14.0.jar:7.14.0]
        at org.elasticsearch.index.mapper.MapperService.parseMappings(MapperService.java:435) ~[elasticsearch-7.14.0.jar:7.14.0]
        ... 24 more

It seems there is some sort of conflict on mappings, how can this issue can be fixed?

leprovokateur · August 26, 2021, 1:57pm

Hi,

did you execute

apm-server setup

after the update?

Best regards,
Robert

ggio · August 26, 2021, 2:08pm

yes, but nothing seems to be changed, the output of

apm-server setup

returns this output:

Automatically disabled ILM as custom index settings configured.
Index setup finished.
Loaded Ingest pipelines

axw · August 30, 2021, 6:33am

Can you please provide us a copy of the template apm-7.14.0? You can fetch this from Elasticsearch with GET /_template/apm-7.14.0 in Kibana dev tools.

ggio · September 3, 2021, 8:29am

I found the reason why I was getting the error.

For some reason, apm-7.7.0 index template was being applied on apm-7.14 indexes

I update manually the index pattern and APM immediately start to work properly

system · September 24, 2021, 4:30am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.