Uptime app 7.9.1 - "Internal server error"

[ManuelF (Manuel)]

Hi,

Note: I am running ELK 7.9.1. All beats were updated to match same version

Recently Uptime app stopped showing data. It used to work fine and I was monitoring a few machines. Now it is showing nothing else but this error message:

I manually created index heartbeat-* to track data shipping and see all fields. Thanks to this I can confirm the data its being delivered to Elasticsearch:

If you see just a few of data is because I have been researching and trying different things. In few words:

• Stopped all beats
• Removed all indexes, templates and policies (until Uptime showed that no data was available, and recommended to install Heartbeat)
• Reinstalled all beats (matching ELK version 7.9.1)
• Ran heartbeat setup from just one machine and confirmed creation of all related indexes, templates and policies. Found no errors in the beat logs.
• Started heartbeat service in just one machine. Found no errors in the beat logs
• Confirmed data is arriving to Elasticsearch as I can see it in Discovery but Uptime started to show the same error.

Any ideas?

Thank you

[jkambic (Justin Kambic)]

Hi @ManuelF, sorry you're running into this. There are a few other things we'd like to see to help zero in on the cause of your issue:

• If you could check your Kibana server log and see if any errors are being logged, sharing those would help us.
• Please share the mapping that you're using, as some of the field data types may be significant.

[ManuelF (Manuel)]

Hi @jkambic and thank you for dedicating your valuable time to help me.

Regarding you request, I have some fresh error logs from Kibana mentioning "No mapping found", that I think may be related to the issue.

Can you please tell me how can I get the mapping you're asking for?

Thank you

Log output

{"type":"log","@timestamp":"2020-09-14T16:04:00Z","tags":["error","http"],"pid":28700,"message":"{ Error: [script_exception] runtime error, with { script_stack={ 0=\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)\" & 1=\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)\" & 2=\"loc = doc[\\\"observer.geo.name\\\"].size() == 0 ? \\\"\\\" : doc[\\\"observer.geo.name\\\"][0];\\n\\n          // One concern here is memory since we could build pretty gigantic maps. I've opted to\\n          // stick to a simple <String,String> map to reduce memory overhead. This means we do\\n          // a little string parsing to treat these strings as records that stay lexicographically\\n          // sortable (which is important later).\\n          // We encode the ID and location as $id.len:$id$loc\\n          String \" & 3=\"          ^---- HERE\" } & script=\"\\n          def loc = doc[\\\"observer.geo.name\\\"].size() == 0 ? \\\"\\\" : doc[\\\"observer.geo.name\\\"][0];\\n\\n          // One concern here is memory since we could build pretty gigantic maps. I've opted to\\n          // stick to a simple <String,String> map to reduce memory overhead. This means we do\\n          // a little string parsing to treat these strings as records that stay lexicographically\\n          // sortable (which is important later).\\n          // We encode the ID and location as $id.len:$id$loc\\n          String id = doc[\\\"monitor.id\\\"][0];\\n          String idLenDelim = Integer.toHexString(id.length()) + \\\":\\\" + id;\\n          String idLoc = loc == null ? idLenDelim : idLenDelim + loc;\\n\\n          String status = doc[\\\"summary.down\\\"][0] > 0 ? \\\"d\\\" : \\\"u\\\";\\n          String timeAndStatus = doc[\\\"@timestamp\\\"][0].toInstant().toEpochMilli().toString() + status;\\n          state.locStatus[idLoc] = timeAndStatus;\\n          state.totalDocs++;\\n        \" & lang=\"painless\" & position={ offset=25 & start=15 & end=515 } }\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/lodash/lodash.js:4949:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)\n  status: 400,\n  displayName: 'BadRequest',\n  message:\n   '[script_exception] runtime error, with { script_stack={ 0=\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)\" & 1=\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)\" & 2=\"loc = doc[\\\\\"observer.geo.name\\\\\"].size() == 0 ? \\\\\"\\\\\" : doc[\\\\\"observer.geo.name\\\\\"][0];\\\\n\\\\n          // One concern here is memory since we could build pretty gigantic maps. I\\'ve opted to\\\\n          // stick to a simple <String,String> map to reduce memory overhead. This means we do\\\\n          // a little string parsing to treat these strings as records that stay lexicographically\\\\n          // sortable (which is important later).\\\\n          // We encode the ID and location as $id.len:$id$loc\\\\n          String \" & 3=\"          ^---- HERE\" } & script=\"\\\\n          def loc = doc[\\\\\"observer.geo.name\\\\\"].size() == 0 ? \\\\\"\\\\\" : doc[\\\\\"observer.geo.name\\\\\"][0];\\\\n\\\\n          // One concern here is memory since we could build pretty gigantic maps. I\\'ve opted to\\\\n          // stick to a simple <String,String> map to reduce memory overhead. This means we do\\\\n          // a little string parsing to treat these strings as records that stay lexicographically\\\\n          // sortable (which is important later).\\\\n          // We encode the ID and location as $id.len:$id$loc\\\\n          String id = doc[\\\\\"monitor.id\\\\\"][0];\\\\n          String idLenDelim = Integer.toHexString(id.length()) + \\\\\":\\\\\" + id;\\\\n          String idLoc = loc == null ? idLenDelim : idLenDelim + loc;\\\\n\\\\n          String status = doc[\\\\\"summary.down\\\\\"][0] > 0 ? \\\\\"d\\\\\" : \\\\\"u\\\\\";\\\\n          String timeAndStatus = doc[\\\\\"@timestamp\\\\\"][0].toInstant().toEpochMilli().toString() + status;\\\\n          state.locStatus[idLoc] = timeAndStatus;\\\\n          state.totalDocs++;\\\\n        \" & lang=\"painless\" & position={ offset=25 & start=15 & end=515 } }',\n  path: '/heartbeat-7*/_search',\n  query: {},\n  body:\n   { error:\n      { root_cause: [Array],\n        type: 'search_phase_execution_exception',\n        reason: 'all shards failed',\n        phase: 'query',\n        grouped: true,\n        failed_shards: [Array] },\n     status: 400 },\n  statusCode: 400,\n  response:\n   '{\"error\":{\"root_cause\":[{\"type\":\"script_exception\",\"reason\":\"runtime error\",\"script_stack\":[\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)\",\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)\",\"loc = doc[\\\\\"observer.geo.name\\\\\"].size() == 0 ? \\\\\"\\\\\" : doc[\\\\\"observer.geo.name\\\\\"][0];\\\\n\\\\n          // One concern here is memory since we could build pretty gigantic maps. I\\'ve opted to\\\\n          // stick to a simple <String,String> map to reduce memory overhead. This means we do\\\\n          // a little string parsing to treat these strings as records that stay lexicographically\\\\n          // sortable (which is important later).\\\\n          // We encode the ID and location as $id.len:$id$loc\\\\n          String \",\"          ^---- HERE\"],\"script\":\"\\\\n          def loc = doc[\\\\\"observer.geo.name\\\\\"].size() == 0 ? \\\\\"\\\\\" : doc[\\\\\"observer.geo.name\\\\\"][0];\\\\n\\\\n          // One concern here is memory since we could build pretty gigantic maps. I\\'ve opted to\\\\n          // stick to a simple <String,String> map to reduce memory overhead. This means we do\\\\n          // a little string parsing to treat these strings as records that stay lexicographically\\\\n          // sortable (which is important later).\\\\n          // We encode the ID and location as $id.len:$id$loc\\\\n          String id = doc[\\\\\"monitor.id\\\\\"][0];\\\\n          String idLenDelim = Integer.toHexString(id.length()) + \\\\\":\\\\\" + id;\\\\n          String idLoc = loc == null ? idLenDelim : idLenDelim + loc;\\\\n\\\\n          String status = doc[\\\\\"summary.down\\\\\"][0] > 0 ? \\\\\"d\\\\\" : \\\\\"u\\\\\";\\\\n          String timeAndStatus = doc[\\\\\"@timestamp\\\\\"][0].toInstant().toEpochMilli().toString() + status;\\\\n          state.locStatus[idLoc] = timeAndStatus;\\\\n          state.totalDocs++;\\\\n        \",\"lang\":\"painless\",\"position\":{\"offset\":25,\"start\":15,\"end\":515}}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"heartbeat-7.9.1\",\"node\":\"lzHHjlpBRC-kkvGCdMHQyQ\",\"reason\":{\"type\":\"script_exception\",\"reason\":\"runtime error\",\"script_stack\":[\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:90)\",\"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:41)\",\"loc = doc[\\\\\"observer.geo.name\\\\\"].size() == 0 ? \\\\\"\\\\\" : doc[\\\\\"observer.geo.name\\\\\"][0];\\\\n\\\\n          // One concern here is memory since we could build pretty gigantic maps. I\\'ve opted to\\\\n          // stick to a simple <String,String> map to reduce memory overhead. This means we do\\\\n          // a little string parsing to treat these strings as records that stay lexicographically\\\\n          // sortable (which is important later).\\\\n          // We encode the ID and location as $id.len:$id$loc\\\\n          String \",\"          ^---- HERE\"],\"script\":\"\\\\n          def loc = doc[\\\\\"observer.geo.name\\\\\"].size() == 0 ? \\\\\"\\\\\" : doc[\\\\\"observer.geo.name\\\\\"][0];\\\\n\\\\n          // One concern here is memory since we could build pretty gigantic maps. I\\'ve opted to\\\\n          // stick to a simple <String,String> map to reduce memory overhead. This means we do\\\\n          // a little string parsing to treat these strings as records that stay lexicographically\\\\n          // sortable (which is important later).\\\\n          // We encode the ID and location as $id.len:$id$loc\\\\n          String id = doc[\\\\\"monitor.id\\\\\"][0];\\\\n          String idLenDelim = Integer.toHexString(id.length()) + \\\\\":\\\\\" + id;\\\\n          String idLoc = loc == null ? idLenDelim : idLenDelim + loc;\\\\n\\\\n          String status = doc[\\\\\"summary.down\\\\\"][0] > 0 ? \\\\\"d\\\\\" : \\\\\"u\\\\\";\\\\n          String timeAndStatus = doc[\\\\\"@timestamp\\\\\"][0].toInstant().toEpochMilli().toString() + status;\\\\n          state.locStatus[idLoc] = timeAndStatus;\\\\n          state.totalDocs++;\\\\n        \",\"lang\":\"painless\",\"position\":{\"offset\":25,\"start\":15,\"end\":515},\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"No field found for [observer.geo.name] in mapping with types []\"}}}]},\"status\":400}',\n  toString: [Function],\n  toJSON: [Function] }"}
{"type":"error","@timestamp":"2020-09-14T16:04:00Z","tags":[],"pid":28700,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:170:34)\n    at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?dateRangeStart=2020-09-14T15%3A48%3A59.997Z&dateRangeEnd=2020-09-14T16%3A03%3A59.997Z","query":{"dateRangeStart":"2020-09-14T15:48:59.997Z","dateRangeEnd":"2020-09-14T16:03:59.997Z"},"pathname":"/api/uptime/snapshot/count","path":"/api/uptime/snapshot/count?dateRangeStart=2020-09-14T15%3A48%3A59.997Z&dateRangeEnd=2020-09-14T16%3A03%3A59.997Z","href":"/api/uptime/snapshot/count?dateRangeStart=2020-09-14T15%3A48%3A59.997Z&dateRangeEnd=2020-09-14T16%3A03%3A59.997Z"},"message":"Internal Server Error"}

[ManuelF (Manuel)]

{"type":"log","@timestamp":"2020-09-14T16:07:06Z","tags":["error","plugins","securitySolution","plugins","securitySolution"],"pid":28700,"message":"[-] nextSearchAfter threw an error [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"urkM35QpQCSWIat5lO6S1A\" & index=\"auditbeat-7.9.1\" } (and) [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"XETuA6BrTOyOdlwZbHXi1g\" & index=\"logs-index_pattern_placeholder\" } (and) [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"ItAOsatYTwy8DJJomN5n-A\" & index=\"packetbeat-7.9.1\" } (and) [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"9vAd9OqlTBW3RAhgypzxcw\" & index=\"winlogbeat-7.9.1\" } :: {\"path\":\"/apm-*-transaction*%2Cauditbeat-*%2Cfilebeat-*%2Clogs-*%2Cpacketbeat-*%2Cwinlogbeat-*/_search\",\"query\":{\"allow_no_indices\":true,\"size\":100,\"ignore_unavailable\":true},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"filter\\\":[{\\\"bool\\\":{\\\"must\\\":[],\\\"filter\\\":[{\\\"bool\\\":{\\\"filter\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"match\\\":{\\\"event.kind\\\":\\\"alert\\\"}}],\\\"minimum_should_match\\\":1}},{\\\"bool\\\":{\\\"must_not\\\":{\\\"bool\\\":{\\\"should\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"match\\\":{\\\"event.module\\\":\\\"endgame\\\"}}],\\\"minimum_should_match\\\":1}},{\\\"bool\\\":{\\\"should\\\":[{\\\"match\\\":{\\\"event.module\\\":\\\"endpoint\\\"}}],\\\"minimum_should_match\\\":1}}],\\\"minimum_should_match\\\":1}}}}]}}],\\\"should\\\":[],\\\"must_not\\\":[]}},{\\\"bool\\\":{\\\"filter\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"range\\\":{\\\"event.ingested\\\":{\\\"gte\\\":\\\"2020-09-14T16:01:06.935Z\\\"}}}],\\\"minimum_should_match\\\":1}},{\\\"bool\\\":{\\\"should\\\":[{\\\"range\\\":{\\\"event.ingested\\\":{\\\"lte\\\":\\\"2020-09-14T16:07:06.935Z\\\"}}}],\\\"minimum_should_match\\\":1}}]}},{\\\"match_all\\\":{}}]}},\\\"sort\\\":[{\\\"event.ingested\\\":{\\\"order\\\":\\\"asc\\\"}}]}\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"urkM35QpQCSWIat5lO6S1A\\\",\\\"index\\\":\\\"auditbeat-7.9.1\\\"},{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"XETuA6BrTOyOdlwZbHXi1g\\\",\\\"index\\\":\\\"logs-index_pattern_placeholder\\\"},{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"ItAOsatYTwy8DJJomN5n-A\\\",\\\"index\\\":\\\"packetbeat-7.9.1\\\"},{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"9vAd9OqlTBW3RAhgypzxcw\\\",\\\"index\\\":\\\"winlogbeat-7.9.1\\\"}],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"can_match\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[{\\\"shard\\\":0,\\\"index\\\":\\\"auditbeat-7.9.1\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"urkM35QpQCSWIat5lO6S1A\\\",\\\"index\\\":\\\"auditbeat-7.9.1\\\"}},{\\\"shard\\\":0,\\\"index\\\":\\\"logs-index_pattern_placeholder\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"XETuA6BrTOyOdlwZbHXi1g\\\",\\\"index\\\":\\\"logs-index_pattern_placeholder\\\"}},{\\\"shard\\\":0,\\\"index\\\":\\\"packetbeat-7.9.1\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"ItAOsatYTwy8DJJomN5n-A\\\",\\\"index\\\":\\\"packetbeat-7.9.1\\\"}},{\\\"shard\\\":0,\\\"index\\\":\\\"winlogbeat-7.9.1\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"9vAd9OqlTBW3RAhgypzxcw\\\",\\\"index\\\":\\\"winlogbeat-7.9.1\\\"}}]},\\\"status\\\":400}\"}"}
{"type":"log","@timestamp":"2020-09-14T16:07:06Z","tags":["error","plugins","securitySolution","plugins","securitySolution"],"pid":28700,"message":"[-] search_after and bulk threw an error [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"urkM35QpQCSWIat5lO6S1A\" & index=\"auditbeat-7.9.1\" } (and) [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"XETuA6BrTOyOdlwZbHXi1g\" & index=\"logs-index_pattern_placeholder\" } (and) [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"ItAOsatYTwy8DJJomN5n-A\" & index=\"packetbeat-7.9.1\" } (and) [query_shard_exception] No mapping found for [event.ingested] in order to sort on, with { index_uuid=\"9vAd9OqlTBW3RAhgypzxcw\" & index=\"winlogbeat-7.9.1\" } :: {\"path\":\"/apm-*-transaction*%2Cauditbeat-*%2Cfilebeat-*%2Clogs-*%2Cpacketbeat-*%2Cwinlogbeat-*/_search\",\"query\":{\"allow_no_indices\":true,\"size\":100,\"ignore_unavailable\":true},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"filter\\\":[{\\\"bool\\\":{\\\"must\\\":[],\\\"filter\\\":[{\\\"bool\\\":{\\\"filter\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"match\\\":{\\\"event.kind\\\":\\\"alert\\\"}}],\\\"minimum_should_match\\\":1}},{\\\"bool\\\":{\\\"must_not\\\":{\\\"bool\\\":{\\\"should\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"match\\\":{\\\"event.module\\\":\\\"endgame\\\"}}],\\\"minimum_should_match\\\":1}},{\\\"bool\\\":{\\\"should\\\":[{\\\"match\\\":{\\\"event.module\\\":\\\"endpoint\\\"}}],\\\"minimum_should_match\\\":1}}],\\\"minimum_should_match\\\":1}}}}]}}],\\\"should\\\":[],\\\"must_not\\\":[]}},{\\\"bool\\\":{\\\"filter\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"range\\\":{\\\"event.ingested\\\":{\\\"gte\\\":\\\"2020-09-14T16:01:06.935Z\\\"}}}],\\\"minimum_should_match\\\":1}},{\\\"bool\\\":{\\\"should\\\":[{\\\"range\\\":{\\\"event.ingested\\\":{\\\"lte\\\":\\\"2020-09-14T16:07:06.935Z\\\"}}}],\\\"minimum_should_match\\\":1}}]}},{\\\"match_all\\\":{}}]}},\\\"sort\\\":[{\\\"event.ingested\\\":{\\\"order\\\":\\\"asc\\\"}}]}\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"urkM35QpQCSWIat5lO6S1A\\\",\\\"index\\\":\\\"auditbeat-7.9.1\\\"},{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"XETuA6BrTOyOdlwZbHXi1g\\\",\\\"index\\\":\\\"logs-index_pattern_placeholder\\\"},{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"ItAOsatYTwy8DJJomN5n-A\\\",\\\"index\\\":\\\"packetbeat-7.9.1\\\"},{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"9vAd9OqlTBW3RAhgypzxcw\\\",\\\"index\\\":\\\"winlogbeat-7.9.1\\\"}],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"can_match\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[{\\\"shard\\\":0,\\\"index\\\":\\\"auditbeat-7.9.1\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"urkM35QpQCSWIat5lO6S1A\\\",\\\"index\\\":\\\"auditbeat-7.9.1\\\"}},{\\\"shard\\\":0,\\\"index\\\":\\\"logs-index_pattern_placeholder\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"XETuA6BrTOyOdlwZbHXi1g\\\",\\\"index\\\":\\\"logs-index_pattern_placeholder\\\"}},{\\\"shard\\\":0,\\\"index\\\":\\\"packetbeat-7.9.1\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"ItAOsatYTwy8DJJomN5n-A\\\",\\\"index\\\":\\\"packetbeat-7.9.1\\\"}},{\\\"shard\\\":0,\\\"index\\\":\\\"winlogbeat-7.9.1\\\",\\\"node\\\":\\\"lzHHjlpBRC-kkvGCdMHQyQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"query_shard_exception\\\",\\\"reason\\\":\\\"No mapping found for [event.ingested] in order to sort on\\\",\\\"index_uuid\\\":\\\"9vAd9OqlTBW3RAhgypzxcw\\\",\\\"index\\\":\\\"winlogbeat-7.9.1\\\"}}]},\\\"status\\\":400}\"} name: \"External Alerts\" id: \"07e9eaec-9a67-41e5-9fd7-3919801783c9\" rule id: \"eb079c62-4481-4d6e-9643-3ca499df7aaa\" signals index: \".siem-signals-default\""}
{"type":"log","@timestamp":"2020-09-14T16:07:06Z","tags":["error","plugins","securitySolution","plugins","securitySolution"],"pid":28700,"message":"Bulk Indexing of signals failed. Check logs for further details. name: \"External Alerts\" id: \"07e9eaec-9a67-41e5-9fd7-3919801783c9\" rule id: \"eb079c62-4481-4d6e-9643-3ca499df7aaa\" signals index: \".siem-signals-default\""}

[jkambic (Justin Kambic)]

Hi @ManuelF, thank you for sharing that output.

tldr: Try stopping Heartbeat, deleting your indices, and re-starting Heartbeat. Detailed instructions below:

At this point, I'd recommend trying this to troubleshoot:

1. Run GET {INDEX_NAME}/_mapping/field/observer.geo.name, you will probably see an empty object.
2. Stop Heartbeat(s).
3. Delete Heartbeat's indices.
   a. Run _cat/indices to verify all Heartbeat indices are gone.
   b. Verify the Uptime solution gives you an empty prompt saying the expected index is not found.
4. Start Heartbeat.
   a. Run _cat/indices until you see heartbeat* (or whichever name you've chosen).
   b. Run GET {INDEX_NAME}/_mapping/field/observer.geo.name, and verify that the mapping contains this field, and that its type is keyword.
5. Refresh your Uptime page and verify the list and overview count appear as intended.

Removed all indexes, templates and policies

One common way that the solution ends up in this state is if the heartbeat* index is deleted while the agent is still running. I was able to reproduce the issue by running DELETE heartbeat* while Heartbeat was still running. As the agent continues to ship data, Elasticsearch creates a new index with inferred mappings. Then when the Uptime solution tries to run its queries that leverage certain fields, the inferred field types are incompatible with the types the query expects (or as in this case the field is absent from the mapping altogether).

[ManuelF (Manuel)]

Hi @jkambic,

• Ran: GET heartbeat-7.9.1/_mapping/field/observer.geo.name

• Response:

  {
  "error" : {
  "root_cause" : [
  {
  "type" : "index_not_found_exception",
  "reason" : "no such index [heartbeat-7.9.1]",
  "resource.type" : "index_or_alias",
  "resource.id" : "heartbeat-7.9.1",
  "index_uuid" : "na",
  "index" : "heartbeat-7.9.1"
  }
  ],
  "type" : "index_not_found_exception",
  "reason" : "no such index [heartbeat-7.9.1]",
  "resource.type" : "index_or_alias",
  "resource.id" : "heartbeat-7.9.1",
  "index_uuid" : "na",
  "index" : "heartbeat-7.9.1"
  },
  "status" : 404
  }

• Stopped Heartbeat

• Removed index heartbeat-7.9.1

• Removed Legacy index template heartbeat-7.9.1

• Ran GET /_cat/indices. Found the other beats indexes, but nothing mentioning heartbeat

• Confirmed Uptime can't find any index to read from
  

  

• Ran .\heartbeat.exe setup to create index and template (no errors)

• Started heartbeat service (no errors)

• Ran: GET /_cat/indices

• Response:

green open heartbeat-7.9.1 iF4pc8ySSza5XtM_z40yzg 1 0 19 0 119.8kb 119.8kb

• Confirmed index and index template added for heartbeat-7.9.1
• Ran: GET heartbeat-7.9.1/_mapping/field/observer.geo.name
• Response:
  {
  "heartbeat-7.9.1" : {
  "mappings" : { }
  }
  }

From this point the issue has been reproduced. Uptime is showing again the "Internal server error". Is this Heartbeat version not mapping correctly when running the setup?

[ManuelF (Manuel)]

Update

It looks like the filed we need it's not being mapped as expected, so I ran:
GET heartbeat-7.9.1/_mapping/ to see all fields available.
Response:

{
  "heartbeat-7.9.1" : {
    "mappings" : {
      "properties" : {
        "@timestamp" : {
          "type" : "date"
        },
        "agent" : {
          "properties" : {
            "ephemeral_id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "ecs" : {
          "properties" : {
            "version" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "event" : {
          "properties" : {
            "dataset" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "icmp" : {
          "properties" : {
            "requests" : {
              "type" : "long"
            },
            "rtt" : {
              "properties" : {
                "us" : {
                  "type" : "long"
                }
              }
            }
          }
        },
        "monitor" : {
          "properties" : {
            "check_group" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "duration" : {
              "properties" : {
                "us" : {
                  "type" : "long"
                }
              }
            },
            "id" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ip" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "status" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "timespan" : {
              "properties" : {
                "gte" : {
                  "type" : "date"
                },
                "lt" : {
                  "type" : "date"
                }
              }
            },
            "type" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "observer" : {
          "properties" : {
            "hostname" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "ip" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "mac" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },
        "summary" : {
          "properties" : {
            "down" : {
              "type" : "long"
            },
            "up" : {
              "type" : "long"
            }
          }
        },
        "url" : {
          "properties" : {
            "domain" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "full" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "scheme" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        }
      }
    }
  }
}

[ManuelF (Manuel)]

Now I can see several 500 Internal Server Error in my browser dev tool. What should I do next?

[ManuelF (Manuel)]

Hi @jkambic,

Anything new I can try to fix this issue? I'm starting to think this might be a bug in ELK 7.9.1 or at least for the apps Observer and Security and/or the beats (v7.9.1).

I am experiencing similar issue with app Security:

(ELK 7.9.1) Security - Hosts and Security - Network missing data

Hi @Frank_Hassanabad, This morning I decided to install ELK 7.9.1 and all beats from scratch. This is a clean installation. I just setup Auditbeat and Heartbeat to test Uptime and Security. Uptime is displaying the error "Internal Server Error" and Security is displaying the error "Data Fetch Failure", and when click on "See the full error(s)" you can see the know error that we were talking about from the start. [visualization error 2] [visualization error] Not sure where the root issue is,…

If there is anything else I can try, please help me.

Thank you

[Andrew_Cholakian1 (Andrew Cholakian)]

@Manuelf something is definitely off, but it's hard to say. I just tested a fresh download of heartbeat 7.9.1 and the mapping worked flawless, as did the app.

How are you running elasticsearch? Are just running a regular version of it on your laptop? My guess is there's something different about how you're running your stack, with something messing up your install somehow.

[ManuelF (Manuel)]

I am using SELKS. I thought that running a previous version I would avoid this issue, but it did not happen as expected. I don't think the problem is the ELK version anymore. This morning I installed SELKS 6, which comes with ELK v7.7.0 and did not configure anything else except Heartbeat and Auditbeat. As soon as Heartbeat started sending data, Observer started displaying the error related to the wrong field.

It seems that the problem is in a wrong mapping, but what can be causing it and how to fix it?

[ManuelF (Manuel)]

I tend to think this could be related to wrong use of ILM. I include in the config file of each beat the following lines:

setup.ilm.overwrite: true
setup.ilm.check_exists: false

What do you think? Does this make any sense to you?

[ManuelF (Manuel)]

Hi @Andrew_Cholakian1 and @jkambic,

I think I may have found the root cause. I installed ELK from scratch (one more time). Since I concluded the version is not the issue, I reinstalled ELK 7.9.1 + Auditbeat and Heartbeat 7.9.1.

What I did different this time was NOT include the following lines in the beats config files:

setup.ilm.overwrite: true
setup.ilm.check_exists: false

Then I followed the regular steps:

• Install service
• Edit config file
• Ran setup
• Started beat service

This time both Observer and Security are working as expected and without errors. Mappings and template for both beats have also the expected values.

My issue is resolved now.

Thank you very much for all your help and for sharing your valuable time