Now I can solved this problem
index name “apm-7.15.1-onboarding-2021.12.02” and “apm-7.15.1-profile-000001” not have mapping field “service.name”. I check index setting for index name “apm-7.15.1-onboarding-2021.12.02” and “apm-7.15.1-profile-000001” on Kibana Dev Tools menu with
GET apm-7.15.1-onboarding-2021.12.02/_settings
GET apm-7.15.1-profile-000001/_settings
I found index name “apm-7.15.1-onboarding-2021.12.02” and “apm-7.15.1-profile-000001” not use index template name apm-7.15.1 (this template default by Elastic). Usually index setting "index.mapping.total_fields.limit" default = 1000 but index name “apm-7.15.1-onboarding-2021.12.02” and “apm-7.15.1-profile-000001” have total field over 1000 fields.
so another index template if you not setting
"index.mapping.total_fields.limit" your index template use default = 1000. if using index template: apm-7.15.1, it is set "index.mapping.total_fields.limit": 2000
After understanding this problem I solved it by this step
- Stop apm server
- Delete index apm-* on Dev Tools menu with
DELETE apm-*
- Unset index_template for index name: apm-*
(index name apm-* must use default index_template name: apm-7.15.1)
- Start apm server
- Check index setting use template: apm-* on Kibana Dev Tools menu with
GET apm-7.15.1-onboarding-2021.12.02/_settings
GET apm-7.15.1-profile-000001/_settings
after checked index name “apm-7.15.1-onboarding-2021.12.02” and “apm-7.15.1-profile-000001” have index mapping “index.mapping.total_fields.limit”: 2000
- Check mapping field all index name apm-* have field service.name
GET apm-*/_mapping/field/service.name
{
"apm-7.15.1-onboarding-2021.12.02" : {
"mappings" : {
"service.name" : {
"full_name" : "service.name",
"mapping" : {
"name" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}
}
},
"apm-7.15.1-profile-000001" : {
"mappings" : {
"service.name" : {
"full_name" : "service.name",
"mapping" : {
"name" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}
}
},
"apm-7.15.1-span-000001" : {
"mappings" : {
"service.name" : {
"full_name" : "service.name",
"mapping" : {
"name" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}
}
},
"apm-7.15.1-metric-000001" : {
"mappings" : {
"service.name" : {
"full_name" : "service.name",
"mapping" : {
"name" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}
}
},
"apm-7.15.1-error-000001" : {
"mappings" : {
"service.name" : {
"full_name" : "service.name",
"mapping" : {
"name" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}
}
},
"apm-7.15.1-transaction-000001" : {
"mappings" : {
"service.name" : {
"full_name" : "service.name",
"mapping" : {
"name" : {
"type" : "keyword",
"ignore_above" : 1024
}
}
}
}
}
}