Heartbeat mappings missing

Elastic Stack Beats
1

Hello,
I'm using ELK 7.17.0 with Heartbeat 7.17 on Linux.
Before starting Heartbeat, I ran the setup command to load index template for Heartbeat and tested the configuration successfully. After starting the Heartbeat and viewing Uptime Monitors tab in Kibana, I get below error:
image

I've tried to stop Heartbeat, delete Heartbeat index and template, run setup and start again, but that didn't help at all.
When using Discover, I can view the data incoming every 5s as per my configuration though.
Also, I've noticed below error in Elastic log file:

[2022-03-09T14:02:29,789][DEBUG][o.e.a.s.TransportSearchAction] [ifcm_elastic_1] [FcZ8e2yvSA2H786OAkCwTA][heartbeat-7.17.0][0]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[heartbeat-7.17.0], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_n
o_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=nul
l, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, enableFieldsEmulation=false, source={"size":0,"query":{"bool":{"must
":[{"range":{"@timestamp":{"from":"now-1d/d","to":"now","include_lower":true,"include_upper":false,"boost":1.0}}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"unique_monitors":{"cardinality":{"field":"monitor.id"}},"unique_locations":{"cardinality":{"field":"observe
r.geo.name","missing":"N/A"}},"monitor_name":{"string_stats":{"field":"monitor.name","show_distribution":false}},"observer_loc_name":{"string_stats":{"field":"observer.geo.name","show_distribution":false}},"monitors":{"terms":{"field":"monitor.id","size":1000,"min_doc_count":1,"sh
ard_min_doc_count":0,"show_term_doc_count_error":false,"order":[{"_count":"desc"},{"_key":"asc"}]},"aggregations":{"docs":{"top_hits":{"from":0,"size":1,"version":false,"seq_no_primary_term":false,"explain":false,"_source":{"includes":["monitor.timespan"],"excludes":[]}}}}}}}}] la
stShard [true]
org.elasticsearch.transport.RemoteTransportException: [ifcm_elastic_1][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: java.lang.IllegalArgumentException: Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [monitor.id] in order to load field data by uninverting the inverted index. Note that this can use significant memory.
        at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:875) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:112) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.index.query.SearchExecutionContext.getForField(SearchExecutionContext.java:297) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.aggregations.support.AggregationContext$ProductionAggregationContext.buildFieldData(AggregationContext.java:403) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.aggregations.support.AggregationContext.buildFieldContext(AggregationContext.java:98) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.internalResolve(ValuesSourceConfig.java:138) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.resolve(ValuesSourceConfig.java:59) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.resolveConfig(ValuesSourceAggregationBuilder.java:454) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:416) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder$LeafOnly.doBuild(ValuesSourceAggregationBuilder.java:113) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:129) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:394) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.SearchService.parseSource(SearchService.java:1239) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:982) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:618) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:483) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionRunnable.lambda$supply$0(ActionRunnable.java:47) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:62) ~[elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.17.0.jar:7.17.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.17.0.jar:7.17.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
        at java.lang.Thread.run(Thread.java:833) [?:?]

Note that I've tested my configuration on different environment on ELK 8.0 and Heartbeat 8.0 and it worked without issues, even without ever using the setup command.

2

Hello @alytkowski ,

I think monitor.id is getting indexed as text field, it should be a keyword, can you verify that and see the mappings aren't correct for monitor.id

Regards

3

This is how the mappings look like for heartbeat-7.17.0 index:

      "monitor" : {
          "properties" : {
            "check_group" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "duration" : {
              "properties" : {
                "us" : {
                  "type" : "long"
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ip" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "status" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "timespan" : {
              "properties" : {
                "gte" : {
                  "type" : "date"
                },
                "lt" : {
                  "type" : "date"
                }
              }
            },
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },

And that's how it looks in the heartbeat-7.17.0 template:

             "monitor" : {
                "properties" : {
                  "duration" : {
                    "properties" : {
                      "us" : {
                        "type" : "long"
                      }
                    }
                  },
                  "ip" : {
                    "type" : "ip"
                  },
                  "name" : {
                    "ignore_above" : 1024,
                    "type" : "keyword",
                    "fields" : {
                      "text" : {
                        "norms" : false,
                        "analyzer" : "simple",
                        "type" : "text"
                      }
                    }
                  },
                  "check_group" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "timespan" : {
                    "type" : "date_range"
                  },
                  "id" : {
                    "ignore_above" : 1024,
                    "type" : "keyword",
                    "fields" : {
                      "text" : {
                        "norms" : false,
                        "analyzer" : "simple",
                        "type" : "text"
                      }
                    }
                  },
                  "type" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "status" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  }
                }
              },
4

So it seems like index template mapping is not getting applied to the index mappings.
I think steps outlined in the troubleshooting guide for mapping should have solved this.

Basically stop all heartbeats, perform DELETE heartbeat-* and then restart heartbeat.

@jkambic @Andrew_Cholakian1 any further tips here?

5

I managed to fix the issue. However, the Deleting of heartbeat indexes wouldn't help, as I did this before. The issue was that Heartbeat always created "heartbeat-7.17.0-*" pattern inside index template, but the index created by heartbeat was: heartbeat-7.17.0.
So the template wasn't actually applied! To fix, I've disabled the auto template creation like this:
setup.template.overwrite: false
setup.template.enabled: false

Then I deleted the heartbeat index and altered index template pattern to "heartbeat-7.17.0*".
After starting heartbeat, the template was now correctly applied and everything works now!

From my perspective this is a clear bug from Heartbeat side.
Cheers.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.