# Uptime app 7.9.1 - "Internal server error"

**URL:** <https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384>\
**Category:** Synthetics\
**Created:** [September 11, 2020, 8:58pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384 "2020-09-11T20:58:56Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![ManuelF](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@ManuelF](https://discuss.elastic.co/u/ManuelF)\
**Post date:** [September 11, 2020, 8:58pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/1 "2020-09-11T20:58:57Z")

</div>

Hi,

**Note:** I am running ELK 7.9.1. All beats were updated to match same version

Recently Uptime app stopped showing data. It used to work fine and I was monitoring a few machines. Now it is showing nothing else but this error message:

 ![Uptime issue](https://us1.discourse-cdn.com/elastic/original/3X/4/a/4a6d4dcf2715e6b18fa1628c3c5787958574436f.png)

I manually created index `heartbeat-*` to track data shipping and see all fields. Thanks to this I can confirm the data its being delivered to Elasticsearch:

 ![Discover](https://us1.discourse-cdn.com/elastic/original/3X/5/a/5ab4d0de97b8bff1a77945b2341d2d797633acee.png)

If you see just a few of data is because I have been researching and trying different things. In few words:

- Stopped all beats
- Removed all indexes, templates and policies (until Uptime showed that no data was available, and recommended to install Heartbeat)
- Reinstalled all beats (matching ELK version 7.9.1)
- Ran heartbeat setup from just one machine and confirmed creation of all related indexes, templates and policies. Found no errors in the beat logs.
- Started heartbeat service in just one machine. Found no errors in the beat logs
- Confirmed data is arriving to Elasticsearch as I can see it in Discovery but Uptime started to show the same error.

Any ideas?

Thank you

---

<div class="post-metadata">

**Author:** ![jkambic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkambic/32/53306_2.png) [@jkambic](https://discuss.elastic.co/u/jkambic)\
**Post date:** [September 14, 2020, 3:29pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/2 "2020-09-14T15:29:08Z")

</div>

Hi @ManuelF, sorry you're running into this. There are a few other things we'd like to see to help zero in on the cause of your issue:

- If you could check your Kibana server log and see if any errors are being logged, sharing those would help us.
- Please share the mapping that you're using, as some of the field data types may be significant.

---

<div class="post-metadata">

**Author:** ![ManuelF](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@ManuelF](https://discuss.elastic.co/u/ManuelF)\
**Post date:** [September 14, 2020, 4:27pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/3 "2020-09-14T16:27:25Z")

</div>

Hi @jkambic and thank you for dedicating your valuable time to help me.

Regarding you request, I have some fresh error logs from Kibana mentioning `"No mapping found"`, that I think may be related to the issue.

Can you please tell me how can I get the mapping you're asking for?

Thank you

## Log output

```
{"type":"log","@timestamp":"2020-09-14T16:04:00Z","tags":["error","http"],"pid":28700,"message":"{ Error: [script_exception] runtime error, with { script_stack={ 0=\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)\" & 1=\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)\" & 2=\"loc = doc[\\\"observer.geo.name\\\"].size() == 0 ? \\\"\\\" : doc[\\\"observer.geo.name\\\"][0];\\n\\n // One concern here is memory since we could build pretty gigantic maps. I've opted to\\n // stick to a simple <String,String> map to reduce memory overhead. This means we do\\n // a little string parsing to treat these strings as records that stay lexicographically\\n // sortable (which is important later).\\n // We encode the ID and location as $id.len:$id$loc\\n String \" & 3=\" ^---- HERE\" } & script=\"\\n def loc = doc[\\\"observer.geo.name\\\"].size() == 0 ? \\\"\\\" : doc[\\\"observer.geo.name\\\"][0];\\n\\n // One concern here is memory since we could build pretty gigantic maps. I've opted to\\n // stick to a simple <String,String> map to reduce memory overhead. This means we do\\n // a little string parsing to treat these strings as records that stay lexicographically\\n // sortable (which is important later).\\n // We encode the ID and location as $id.len:$id$loc\\n String id = doc[\\\"monitor.id\\\"][0];\\n String idLenDelim = Integer.toHexString(id.length()) + \\\":\\\" + id;\\n String idLoc = loc == null ? idLenDelim : idLenDelim + loc;\\n\\n String status = doc[\\\"summary.down\\\"][0] > 0 ? \\\"d\\\" : \\\"u\\\";\\n String timeAndStatus = doc[\\\"@timestamp\\\"][0].toInstant().toEpochMilli().toString() + status;\\n state.locStatus[idLoc] = timeAndStatus;\\n state.totalDocs++;\\n \" & lang=\"painless\" & position={ offset=25 & start=15 & end=515 } }\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/lodash/lodash.js:4949:19)\n at IncomingMessage.emit (events.js:203:15)\n at endReadableNT (_stream_readable.js:1145:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)\n status: 400,\n displayName: 'BadRequest',\n message:\n '[script_exception] runtime error, with { script_stack={ 0=\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)\" & 1=\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)\" & 2=\"loc = doc[\\\\\"observer.geo.name\\\\\"].size() == 0 ? \\\\\"\\\\\" : doc[\\\\\"observer.geo.name\\\\\"][0];\\\\n\\\\n // One concern here is memory since we could build pretty gigantic maps. I\\'ve opted to\\\\n // stick to a simple <String,String> map to reduce memory overhead. This means we do\\\\n // a little string parsing to treat these strings as records that stay lexicographically\\\\n // sortable (which is important later).\\\\n // We encode the ID and location as $id.len:$id$loc\\\\n String \" & 3=\" ^---- HERE\" } & script=\"\\\\n def loc = doc[\\\\\"observer.geo.name\\\\\"].size() == 0 ? \\\\\"\\\\\" : doc[\\\\\"observer.geo.name\\\\\"][0];\\\\n\\\\n // One concern here is memory since we could build pretty gigantic maps. I\\'ve opted to\\\\n // stick to a simple <String,String> map to reduce memory overhead. This means we do\\\\n // a little string parsing to treat these strings as records that stay lexicographically\\\\n // sortable (which is important later).\\\\n // We encode the ID and location as $id.len:$id$loc\\\\n String id = doc[\\\\\"monitor.id\\\\\"][0];\\\\n String idLenDelim = Integer.toHexString(id.length()) + \\\\\":\\\\\" + id;\\\\n String idLoc = loc == null ? idLenDelim : idLenDelim + loc;\\\\n\\\\n String status = doc[\\\\\"summary.down\\\\\"][0] > 0 ? \\\\\"d\\\\\" : \\\\\"u\\\\\";\\\\n String timeAndStatus = doc[\\\\\"@timestamp\\\\\"][0].toInstant().toEpochMilli().toString() + status;\\\\n state.locStatus[idLoc] = timeAndStatus;\\\\n state.totalDocs++;\\\\n \" & lang=\"painless\" & position={ offset=25 & start=15 & end=515 } }',\n path: '/heartbeat-7*/_search',\n query: {},\n body:\n { error:\n { root_cause: [Array],\n type: 'search_phase_execution_exception',\n reason: 'all shards failed',\n phase: 'query',\n grouped: true,\n failed_shards: [Array] },\n status: 400 },\n statusCode: 400,\n response:\n '{\"error\":{\"root_cause\":[{\"type\":\"script_exception\",\"reason\":\"runtime error\",\"script_stack\":[\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)\",\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)\",\"loc = doc[\\\\\"observer.geo.name\\\\\"].size() == 0 ? \\\\\"\\\\\" : doc[\\\\\"observer.geo.name\\\\\"][0];\\\\n\\\\n // One concern here is memory since we could build pretty gigantic maps. I\\'ve opted to\\\\n // stick to a simple <String,String> map to reduce memory overhead. This means we do\\\\n // a little string parsing to treat these strings as records that stay lexicographically\\\\n // sortable (which is important later).\\\\n // We encode the ID and location as $id.len:$id$loc\\\\n String \",\" ^---- HERE\"],\"script\":\"\\\\n def loc = doc[\\\\\"observer.geo.name\\\\\"].size() == 0 ? \\\\\"\\\\\" : doc[\\\\\"observer.geo.name\\\\\"][0];\\\\n\\\\n // One concern here is memory since we could build pretty gigantic maps. I\\'ve opted to\\\\n // stick to a simple <String,String> map to reduce memory overhead. This means we do\\\\n // a little string parsing to treat these strings as records that stay lexicographically\\\\n // sortable (which is important later).\\\\n // We encode the ID and location as $id.len:$id$loc\\\\n String id = doc[\\\\\"monitor.id\\\\\"][0];\\\\n String idLenDelim = Integer.toHexString(id.length()) + \\\\\":\\\\\" + id;\\\\n String idLoc = loc == null ? idLenDelim : idLenDelim + loc;\\\\n\\\\n String status = doc[\\\\\"summary.down\\\\\"][0] > 0 ? \\\\\"d\\\\\" : \\\\\"u\\\\\";\\\\n String timeAndStatus = doc[\\\\\"@timestamp\\\\\"][0].toInstant().toEpochMilli().toString() + status;\\\\n state.locStatus[idLoc] = timeAndStatus;\\\\n state.totalDocs++;\\\\n \",\"lang\":\"painless\",\"position\":{\"offset\":25,\"start\":15,\"end\":515}}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"heartbeat-7.9.1\",\"node\":\"lzHHjlpBRC-kkvGCdMHQyQ\",\"reason\":{\"type\":\"script_exception\",\"reason\":\"runtime error\",\"script_stack\":[\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)\",\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)\",\"loc = doc[\\\\\"observer.geo.name\\\\\"].size() == 0 ? \\\\\"\\\\\" : doc[\\\\\"observer.geo.name\\\\\"][0];\\\\n\\\\n // One concern here is memory since we could build pretty gigantic maps. I\\'ve opted to\\\\n // stick to a simple <String,String> map to reduce memory overhead. This means we do\\\\n // a little string parsing to treat these strings as records that stay lexicographically\\\\n // sortable (which is important later).\\\\n // We encode the ID and location as $id.len:$id$loc\\\\n String \",\" ^---- HERE\"],\"script\":\"\\\\n def loc = doc[\\\\\"observer.geo.name\\\\\"].size() == 0 ? \\\\\"\\\\\" : doc[\\\\\"observer.geo.name\\\\\"][0];\\\\n\\\\n // One concern here is memory since we could build pretty gigantic maps. I\\'ve opted to\\\\n // stick to a simple <String,String> map to reduce memory overhead. This means we do\\\\n // a little string parsing to treat these strings as records that stay lexicographically\\\\n // sortable (which is important later).\\\\n // We encode the ID and location as $id.len:$id$loc\\\\n String id = doc[\\\\\"monitor.id\\\\\"][0];\\\\n String idLenDelim = Integer.toHexString(id.length()) + \\\\\":\\\\\" + id;\\\\n String idLoc = loc == null ? idLenDelim : idLenDelim + loc;\\\\n\\\\n String status = doc[\\\\\"summary.down\\\\\"][0] > 0 ? \\\\\"d\\\\\" : \\\\\"u\\\\\";\\\\n String timeAndStatus = doc[\\\\\"@timestamp\\\\\"][0].toInstant().toEpochMilli().toString() + status;\\\\n state.locStatus[idLoc] = timeAndStatus;\\\\n state.totalDocs++;\\\\n \",\"lang\":\"painless\",\"position\":{\"offset\":25,\"start\":15,\"end\":515},\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"No field found for [observer.geo.name] in mapping with types []\"}}}]},\"status\":400}',\n toString: [Function],\n toJSON: [Function] }"}
{"type":"error","@timestamp":"2020-09-14T16:04:00Z","tags":[],"pid":28700,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:170:34)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?dateRangeStart=2020-09-14T15%3A48%3A59.997Z&dateRangeEnd=2020-09-14T16%3A03%3A59.997Z","query":{"dateRangeStart":"2020-09-14T15:48:59.997Z","dateRangeEnd":"2020-09-14T16:03:59.997Z"},"pathname":"/api/uptime/snapshot/count","path":"/api/uptime/snapshot/count?dateRangeStart=2020-09-14T15%3A48%3A59.997Z&dateRangeEnd=2020-09-14T16%3A03%3A59.997Z","href":"/api/uptime/snapshot/count?dateRangeStart=2020-09-14T15%3A48%3A59.997Z&dateRangeEnd=2020-09-14T16%3A03%3A59.997Z"},"message":"Internal Server Error"}
```

---

<div class="post-metadata">

**Author:** ![ManuelF](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@ManuelF](https://discuss.elastic.co/u/ManuelF)\
**Post date:** [September 14, 2020, 4:28pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/4 "2020-09-14T16:28:12Z")

</div>

```
{"type":"log","@timestamp":"2020-09-14T16:07:06Z","tags":["error","plugins","securitySolution","plugins","securitySolution"],"pid":28700,"message":"[-] nextSearchAfter threw an error [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"urkM35QpQCSWIat5lO6S1A\" & index=\"auditbeat-7.9.1\" } (and) [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"XETuA6BrTOyOdlwZbHXi1g\" & index=\"logs-index_pattern_placeholder\" } (and) [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"ItAOsatYTwy8DJJomN5n-A\" & index=\"packetbeat-7.9.1\" } (and) [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"9vAd9OqlTBW3RAhgypzxcw\" & index=\"winlogbeat-7.9.1\" } :: {\"path\":\"/apm-*-transaction*%2Cauditbeat-*%2Cfilebeat-*%2Clogs-*%2Cpacketbeat-*%2Cwinlogbeat-*/_search\",\"query\":{\"allow_no_indices\":true,\"size\":100,\"ignore_unavailable\":true},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"filter\\\":[{\\\"bool\\\":{\\\"must\\\":[],\\\"filter\\\":[{\\\"bool\\\":{\\\"filter\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"match\\\":{\\\"event.kind\\\":\\\"alert\\\"}}],\\\"minimum_should_match\\\":1}},{\\\"bool\\\":{\\\"must_not\\\":{\\\"bool\\\":{\\\"should\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"match\\\":{\\\"event.module\\\":\\\"endgame\\\"}}],\\\"minimum_should_match\\\":1}},{\\\"bool\\\":{\\\"should\\\":[{\\\"match\\\":{\\\"event.module\\\":\\\"endpoint\\\"}}],\\\"minimum_should_match\\\":1}}],\\\"minimum_should_match\\\":1}}}}]}}],\\\"should\\\":[],\\\"must_not\\\":[]}},{\\\"bool\\\":{\\\"filter\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"range\\\":{\\\"event.ingested\\\":{\\\"gte\\\":\\\"2020-09-14T16:01:06.935Z\\\"}}}],\\\"minimum_should_match\\\":1}},{\\\"bool\\\":{\\\"should\\\":[{\\\"range\\\":{\\\"event.ingested\\\":{\\\"lte\\\":\\\"2020-09-14T16:07:06.935Z\\\"}}}],\\\"minimum_should_match\\\":1}}]}},{\\\"match_all\\\":{}}]}},\\\"sort\\\":[{\\\"event.ingested\\\":{\\\"order\\\":\\\"asc\\\"}}]}\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"urkM35QpQCSWIat5lO6S1A\\\",\\\"index\\\":\\\"auditbeat-7.9.1\\\"},{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"XETuA6BrTOyOdlwZbHXi1g\\\",\\\"index\\\":\\\"logs-index_pattern_placeholder\\\"},{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"ItAOsatYTwy8DJJomN5n-A\\\",\\\"index\\\":\\\"packetbeat-7.9.1\\\"},{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"9vAd9OqlTBW3RAhgypzxcw\\\",\\\"index\\\":\\\"winlogbeat-7.9.1\\\"}],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"can_match\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[{\\\"shard\\\":0,\\\"index\\\":\\\"auditbeat-7.9.1\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"urkM35QpQCSWIat5lO6S1A\\\",\\\"index\\\":\\\"auditbeat-7.9.1\\\"}},{\\\"shard\\\":0,\\\"index\\\":\\\"logs-index_pattern_placeholder\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"XETuA6BrTOyOdlwZbHXi1g\\\",\\\"index\\\":\\\"logs-index_pattern_placeholder\\\"}},{\\\"shard\\\":0,\\\"index\\\":\\\"packetbeat-7.9.1\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"ItAOsatYTwy8DJJomN5n-A\\\",\\\"index\\\":\\\"packetbeat-7.9.1\\\"}},{\\\"shard\\\":0,\\\"index\\\":\\\"winlogbeat-7.9.1\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"9vAd9OqlTBW3RAhgypzxcw\\\",\\\"index\\\":\\\"winlogbeat-7.9.1\\\"}}]},\\\"status\\\":400}\"}"}
{"type":"log","@timestamp":"2020-09-14T16:07:06Z","tags":["error","plugins","securitySolution","plugins","securitySolution"],"pid":28700,"message":"[-] search_after and bulk threw an error [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"urkM35QpQCSWIat5lO6S1A\" & index=\"auditbeat-7.9.1\" } (and) [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"XETuA6BrTOyOdlwZbHXi1g\" & index=\"logs-index_pattern_placeholder\" } (and) [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"ItAOsatYTwy8DJJomN5n-A\" & index=\"packetbeat-7.9.1\" } (and) [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"9vAd9OqlTBW3RAhgypzxcw\" & index=\"winlogbeat-7.9.1\" } :: {\"path\":\"/apm-*-transaction*%2Cauditbeat-*%2Cfilebeat-*%2Clogs-*%2Cpacketbeat-*%2Cwinlogbeat-*/_search\",\"query\":{\"allow_no_indices\":true,\"size\":100,\"ignore_unavailable\":true},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"filter\\\":[{\\\"bool\\\":{\\\"must\\\":[],\\\"filter\\\":[{\\\"bool\\\":{\\\"filter\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"match\\\":{\\\"event.kind\\\":\\\"alert\\\"}}],\\\"minimum_should_match\\\":1}},{\\\"bool\\\":{\\\"must_not\\\":{\\\"bool\\\":{\\\"should\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"match\\\":{\\\"event.module\\\":\\\"endgame\\\"}}],\\\"minimum_should_match\\\":1}},{\\\"bool\\\":{\\\"should\\\":[{\\\"match\\\":{\\\"event.module\\\":\\\"endpoint\\\"}}],\\\"minimum_should_match\\\":1}}],\\\"minimum_should_match\\\":1}}}}]}}],\\\"should\\\":[],\\\"must_not\\\":[]}},{\\\"bool\\\":{\\\"filter\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"range\\\":{\\\"event.ingested\\\":{\\\"gte\\\":\\\"2020-09-14T16:01:06.935Z\\\"}}}],\\\"minimum_should_match\\\":1}},{\\\"bool\\\":{\\\"should\\\":[{\\\"range\\\":{\\\"event.ingested\\\":{\\\"lte\\\":\\\"2020-09-14T16:07:06.935Z\\\"}}}],\\\"minimum_should_match\\\":1}}]}},{\\\"match_all\\\":{}}]}},\\\"sort\\\":[{\\\"event.ingested\\\":{\\\"order\\\":\\\"asc\\\"}}]}\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"urkM35QpQCSWIat5lO6S1A\\\",\\\"index\\\":\\\"auditbeat-7.9.1\\\"},{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"XETuA6BrTOyOdlwZbHXi1g\\\",\\\"index\\\":\\\"logs-index_pattern_placeholder\\\"},{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"ItAOsatYTwy8DJJomN5n-A\\\",\\\"index\\\":\\\"packetbeat-7.9.1\\\"},{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"9vAd9OqlTBW3RAhgypzxcw\\\",\\\"index\\\":\\\"winlogbeat-7.9.1\\\"}],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"can_match\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[{\\\"shard\\\":0,\\\"index\\\":\\\"auditbeat-7.9.1\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"urkM35QpQCSWIat5lO6S1A\\\",\\\"index\\\":\\\"auditbeat-7.9.1\\\"}},{\\\"shard\\\":0,\\\"index\\\":\\\"logs-index_pattern_placeholder\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"XETuA6BrTOyOdlwZbHXi1g\\\",\\\"index\\\":\\\"logs-index_pattern_placeholder\\\"}},{\\\"shard\\\":0,\\\"index\\\":\\\"packetbeat-7.9.1\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"ItAOsatYTwy8DJJomN5n-A\\\",\\\"index\\\":\\\"packetbeat-7.9.1\\\"}},{\\\"shard\\\":0,\\\"index\\\":\\\"winlogbeat-7.9.1\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"9vAd9OqlTBW3RAhgypzxcw\\\",\\\"index\\\":\\\"winlogbeat-7.9.1\\\"}}]},\\\"status\\\":400}\"} name: \"External Alerts\" id: \"07e9eaec-9a67-41e5-9fd7-3919801783c9\" rule id: \"eb079c62-4481-4d6e-9643-3ca499df7aaa\" signals index: \".siem-signals-default\""}
{"type":"log","@timestamp":"2020-09-14T16:07:06Z","tags":["error","plugins","securitySolution","plugins","securitySolution"],"pid":28700,"message":"Bulk Indexing of signals failed. Check logs for further details. name: \"External Alerts\" id: \"07e9eaec-9a67-41e5-9fd7-3919801783c9\" rule id: \"eb079c62-4481-4d6e-9643-3ca499df7aaa\" signals index: \".siem-signals-default\""}
```

---

<div class="post-metadata">

**Author:** ![jkambic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkambic/32/53306_2.png) [@jkambic](https://discuss.elastic.co/u/jkambic)\
**Post date:** [September 14, 2020, 10:21pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/5 "2020-09-14T22:21:42Z")

</div>

Hi @ManuelF, thank you for sharing that output.

**tldr:** Try stopping Heartbeat, deleting your indices, and re-starting Heartbeat. Detailed instructions below:

At this point, I'd recommend trying this to troubleshoot:

1. Run `GET {INDEX_NAME}/_mapping/field/observer.geo.name`, you will probably see an empty object.
2. Stop Heartbeat(s).
3. Delete Heartbeat's indices.  
a. Run `_cat/indices` to verify all Heartbeat indices are gone.  
b. Verify the Uptime solution gives you an empty prompt saying the expected index is not found.
4. Start Heartbeat.  
a. Run `_cat/indices` until you see `heartbeat*` (or whichever name you've chosen).  
b. Run `GET {INDEX_NAME}/_mapping/field/observer.geo.name`, and verify that the mapping contains this field, and that its type is `keyword`.
5. Refresh your Uptime page and verify the list and overview count appear as intended.

> Removed all indexes, templates and policies

One common way that the solution ends up in this state is if the `heartbeat*` index is deleted while the agent is still running. I was able to reproduce the issue by running `DELETE heartbeat*` while Heartbeat was still running. As the agent continues to ship data, Elasticsearch creates a new index with inferred mappings. Then when the Uptime solution tries to run its queries that leverage certain fields, the inferred field types are incompatible with the types the query expects (or as in this case the field is absent from the mapping altogether).

---

<div class="post-metadata">

**Author:** ![ManuelF](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@ManuelF](https://discuss.elastic.co/u/ManuelF)\
**Post date:** [September 15, 2020, 2:51pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/6 "2020-09-15T14:51:26Z")

</div>

Hi @jkambic,

- Ran: `GET heartbeat-7.9.1/_mapping/field/observer.geo.name`

- Response:

- Stopped Heartbeat

- Removed index heartbeat-7.9.1

- Removed Legacy index template heartbeat-7.9.1

- Ran `GET /_cat/indices`. Found the other beats indexes, but nothing mentioning `heartbeat`

- Confirmed Uptime can't find any index to read from  

- Ran `.\heartbeat.exe setup` to create index and template (no errors)

- Started `heartbeat` service (no errors)

- Ran: `GET /_cat/indices`

- Response:

`green open heartbeat-7.9.1 iF4pc8ySSza5XtM_z40yzg 1 0 19 0 119.8kb 119.8kb`

- Confirmed index and index template added for `heartbeat-7.9.1`
- Ran: `GET heartbeat-7.9.1/_mapping/field/observer.geo.name`
- Response:  
{  
"heartbeat-7.9.1" : {  
"mappings" : { }  
}  
}

From this point the issue has been reproduced. `Uptime` is showing again the `"Internal server error"`. Is this Heartbeat version not mapping correctly when running the setup?

---

<div class="post-metadata">

**Author:** ![ManuelF](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@ManuelF](https://discuss.elastic.co/u/ManuelF)\
**Post date:** [September 15, 2020, 3:17pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/7 "2020-09-15T15:17:35Z")

</div>

## Update

It looks like the filed we need it's not being mapped as expected, so I ran:  
`GET heartbeat-7.9.1/_mapping/` to see all fields available.  
Response:

```
{
  "heartbeat-7.9.1" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "agent" : {
          "properties" : {
            "ephemeral_id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "ecs" : {
          "properties" : {
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "event" : {
          "properties" : {
            "dataset" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "icmp" : {
          "properties" : {
            "requests" : {
              "type" : "long"
            },
            "rtt" : {
              "properties" : {
                "us" : {
                  "type" : "long"
                }
              }
            }
          }
        },
        "monitor" : {
          "properties" : {
            "check_group" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "duration" : {
              "properties" : {
                "us" : {
                  "type" : "long"
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ip" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "status" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "timespan" : {
              "properties" : {
                "gte" : {
                  "type" : "date"
                },
                "lt" : {
                  "type" : "date"
                }
              }
            },
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "observer" : {
          "properties" : {
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ip" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "mac" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "summary" : {
          "properties" : {
            "down" : {
              "type" : "long"
            },
            "up" : {
              "type" : "long"
            }
          }
        },
        "url" : {
          "properties" : {
            "domain" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "full" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "scheme" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        }
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![ManuelF](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@ManuelF](https://discuss.elastic.co/u/ManuelF)\
**Post date:** [September 15, 2020, 4:58pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/8 "2020-09-15T16:58:08Z")

</div>

Now I can see several `500 Internal Server Error` in my browser dev tool. What should I do next?

---

<div class="post-metadata">

**Author:** ![ManuelF](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@ManuelF](https://discuss.elastic.co/u/ManuelF)\
**Post date:** [September 17, 2020, 2:11pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/9 "2020-09-17T14:11:43Z")

</div>

Hi @jkambic,

Anything new I can try to fix this issue? I'm starting to think this might be a bug in ELK 7.9.1 or at least for the apps Observer and Security and/or the beats (v7.9.1).

I am experiencing similar issue with app Security:

> [@(ELK 7.9.1) Security - Hosts and Security - Network missing data](https://discuss.elastic.co/t/elk-7-9-1-security-hosts-and-security-network-missing-data/248576/13):
>
> Hi @Frank_Hassanabad, This morning I decided to install ELK 7.9.1 and all beats from scratch. This is a clean installation. I just setup Auditbeat and Heartbeat to test Uptime and Security. Uptime is displaying the error "Internal Server Error" and Security is displaying the error "Data Fetch Failure", and when click on "See the full error(s)" you can see the know error that we were talking about from the start. Not sure where the root issue is,…

If there is anything else I can try, please help me.

Thank you

---

<div class="post-metadata">

**Author:** ![Andrew\_Cholakian1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_cholakian1/32/3612_2.png) [@Andrew\_Cholakian1](https://discuss.elastic.co/u/Andrew_Cholakian1)\
**Post date:** [September 17, 2020, 5:29pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/10 "2020-09-17T17:29:26Z")

</div>

@Manuelf something is definitely off, but it's hard to say. I just tested a fresh download of heartbeat 7.9.1 and the mapping worked flawless, as did the app.

How are you running elasticsearch? Are just running a regular version of it on your laptop? My guess is there's something different about how you're running your stack, with something messing up your install somehow.

---

<div class="post-metadata">

**Author:** ![ManuelF](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@ManuelF](https://discuss.elastic.co/u/ManuelF)\
**Post date:** [September 17, 2020, 6:37pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/11 "2020-09-17T18:37:34Z")

</div>

I am using SELKS. I thought that running a previous version I would avoid this issue, but it did not happen as expected. I don't think the problem is the ELK version anymore. This morning I installed SELKS 6, which comes with ELK v7.7.0 and did not configure anything else except Heartbeat and Auditbeat. As soon as Heartbeat started sending data, Observer started displaying the error related to the wrong field.

It seems that the problem is in a wrong mapping, but what can be causing it and how to fix it?

---

<div class="post-metadata">

**Author:** ![ManuelF](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@ManuelF](https://discuss.elastic.co/u/ManuelF)\
**Post date:** [September 17, 2020, 7:30pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/12 "2020-09-17T19:30:32Z")

</div>

I tend to think this could be related to wrong use of ILM. I include in the config file of each beat the following lines:

```
setup.ilm.overwrite: true
setup.ilm.check_exists: false

```

What do you think? Does this make any sense to you?

---

<div class="post-metadata">

**Author:** ![ManuelF](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@ManuelF](https://discuss.elastic.co/u/ManuelF)\
**Post date:** [September 17, 2020, 10:17pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/13 "2020-09-17T22:17:26Z")

</div>

Hi @Andrew_Cholakian1 and @jkambic,

I think I may have found the root cause. I installed ELK from scratch (one more time). Since I concluded the version is not the issue, I reinstalled ELK 7.9.1 + Auditbeat and Heartbeat 7.9.1.

What I did different this time was NOT include the following lines in the beats config files:

```auto
setup.ilm.overwrite: true
setup.ilm.check_exists: false

```

Then I followed the regular steps:

- Install service
- Edit config file
- Ran `setup`
- Started beat service

This time both Observer and Security are working as expected and without errors. Mappings and template for both beats have also the expected values.

My issue is resolved now.

Thank you very much for all your help and for sharing your valuable time 👍

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 11, 2020, 10:17pm UTC](https://discuss.elastic.co/t/uptime-app-7-9-1-internal-server-error/248384/14 "2020-10-11T22:17:31Z")

</div>

This topic was automatically closed 24 days after the last reply. New replies are no longer allowed.