Hi, from time to time but re-occuring we loosing data in metricbeat-* index. We noticed this happen at 2am when index snapshot is taken. The lost data causing dashboards to fail.
Err:
Fielddata is disabled on text fields by default. Set fielddata=true on [monitor.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
Version: 7.3.0
Build: 25250
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [monitor.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"heartbeat-7.3.0","node":"zCzsicBKRUKhzVN7qu5MTg","reason":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [monitor.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [monitor.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.","caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [monitor.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}},"status":400}
at https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/bundles/commons.bundle.js:3:3686152
at Function.Promise.try (https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/bundles/commons.bundle.js:3:1907221)
at https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/bundles/commons.bundle.js:3:1906582
at Array.map (<anonymous>)
at Function.Promise.map (https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/bundles/commons.bundle.js:3:1906537)
at callResponseHandlers (https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/bundles/commons.bundle.js:3:3685164)
at https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/bundles/commons.bundle.js:3:3667932
at processQueue (https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/built_assets/dlls/vendors.bundle.dll.js:499:199747)
at https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/built_assets/dlls/vendors.bundle.dll.js:499:200710
at Scope.$digest (https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/built_assets/dlls/vendors.bundle.dll.js:499:210518)
To add to the issue is that it seems that during index snapshottting that occurs daily at 1AM the index is deleted and new index with short name is created. This causing data from merticbeat not being indexed.
