Metricbeat index gets deleted loosing data

Hi, from time to time but re-occuring we loosing data in metricbeat-* index. We noticed this happen at 2am when index snapshot is taken. The lost data causing dashboards to fail.

Err:
Fielddata is disabled on text fields by default. Set fielddata=true on [monitor.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.

Version: 7.3.0
Build: 25250
Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [monitor.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"heartbeat-7.3.0","node":"zCzsicBKRUKhzVN7qu5MTg","reason":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [monitor.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}],"caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [monitor.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.","caused_by":{"type":"illegal_argument_exception","reason":"Fielddata is disabled on text fields by default. Set fielddata=true on [monitor.id] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."}}},"status":400}
    at https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/bundles/commons.bundle.js:3:3686152
    at Function.Promise.try (https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/bundles/commons.bundle.js:3:1907221)
    at https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/bundles/commons.bundle.js:3:1906582
    at Array.map (<anonymous>)
    at Function.Promise.map (https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/bundles/commons.bundle.js:3:1906537)
    at callResponseHandlers (https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/bundles/commons.bundle.js:3:3685164)
    at https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/bundles/commons.bundle.js:3:3667932
    at processQueue (https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/built_assets/dlls/vendors.bundle.dll.js:499:199747)
    at https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/built_assets/dlls/vendors.bundle.dll.js:499:200710
    at Scope.$digest (https://56f36f75cf6d28343bdda2cabdc16b6c.eu-west-1.aws.found.io/built_assets/dlls/vendors.bundle.dll.js:499:210518)

To add to the issue is that it seems that during index snapshottting that occurs daily at 1AM the index is deleted and new index with short name is created. This causing data from merticbeat not being indexed.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.