# Shards failed warning on Network dashboard in SIEM app **URL:** \ **Category:** SIEM\ **Created:** [February 19, 2020, 7:14am UTC](https://discuss.elastic.co/t/shards-failed-warning-on-network-dashboard-in-siem-app/219908 "2020-02-19T07:14:13Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![wconnell](https://avatars.discourse-cdn.com/v4/letter/w/f4b2a3/32.png) [@wconnell](https://discuss.elastic.co/u/wconnell)\ **Post date:** [February 19, 2020, 7:14am UTC](https://discuss.elastic.co/t/shards-failed-warning-on-network-dashboard-in-siem-app/219908/1 "2020-02-19T07:14:13Z")
Hi there, I'm getting a shards failed warning in the Network dashboard of the SIEM app (v7.6). I'm unable to diagnose as when I click the "Show details" button, nothing happens. I used the inspect tool in Chrome and it cites an error each time I click ("Uncaught error: Overlays was not set"). ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/4/f43326074397b2b33ba57f0fdb371c9b629f6031.png) Any ideas how I can get past this and see what the error is? I suspected it was a field name conflict in my Kibana index pattern, but when I refreshed it in the config, it didn't cite any conflicts. And again, I can't see the 2 offending shards because that button does nothing when I click on it. Thanks for any help! ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [February 19, 2020, 10:47am UTC](https://discuss.elastic.co/t/shards-failed-warning-on-network-dashboard-in-siem-app/219908/2 "2020-02-19T10:47:17Z")
If you have access to the Elasticsearch logs, it's probably easiest to find the error by looking there. If not, maybe try to use the "Index Management" and/or "Stack Management" pages under Kibana Settings to see if any of your indices reports errors. We'll be looking into why that "Show details" button doesn't work. I know it's annoying, sorry about that. ---
**Author:** ![wconnell](https://avatars.discourse-cdn.com/v4/letter/w/f4b2a3/32.png) [@wconnell](https://discuss.elastic.co/u/wconnell)\ **Post date:** [February 19, 2020, 7:04pm UTC](https://discuss.elastic.co/t/shards-failed-warning-on-network-dashboard-in-siem-app/219908/3 "2020-02-19T19:04:18Z")
It's a hosted cluster (via GCP marketplace), so I only have access to a subset of the Elasticsearch log it seems. No kibana logs. I would need an SRE to install filebeat on the nodes as I don't have direct access to the nodes myself. I do have monitoring to another cluster set up but again, a push button to enable logging to the monitoring cluster would be nice. Didn't see anything in the Index Management pages under Kibana. I refreshed the index pattern and hoped to see a conflict but there was none. I suspect it may have something to do with `source.geo.country_iso_code` as I'm not seeing anything in that table despite having logs where that field is populated. Again, if one of my indices were storing it as something other than a keyword, I would have expected to see a conflict when refreshing the index pattern, right? ---
**Author:** ![wconnell](https://avatars.discourse-cdn.com/v4/letter/w/f4b2a3/32.png) [@wconnell](https://discuss.elastic.co/u/wconnell)\ **Post date:** [February 19, 2020, 11:56pm UTC](https://discuss.elastic.co/t/shards-failed-warning-on-network-dashboard-in-siem-app/219908/4 "2020-02-19T23:56:24Z")
I found an issue that may be related to this. About a week ago in response to [this issue](https://discuss.elastic.co/t/siem-app-doesnt-use-timezone-setting/216906/11), I changed my template mapping for timestamps to use `epoch_second`. Logs were previously stored like this: `"@timestamp": "2020-02-03 23:59:00"` And now they're stored like this: `"@timestamp": 1582070340` I just realized that since making that change, none of those logs are being returned in the queries being made by the visualizations. The reason is because they filter for timestamps using `epoch_millis`, like this: ```auto { "aggregations": { "top_countries_count": { "cardinality": { "field": "source.geo.country_iso_code" } }, "source": { "terms": { "field": "source.geo.country_iso_code", "size": 10, "order": { "bytes_out": "desc" } }, "aggs": { "bytes_in": { "sum": { "field": "destination.bytes" } }, "bytes_out": { "sum": { "field": "source.bytes" } }, "flows": { "cardinality": { "field": "network.community_id" } }, "source_ips": { "cardinality": { "field": "source.ip" } }, "destination_ips": { "cardinality": { "field": "destination.ip" } } } } }, "query": { "bool": { "filter": [ { "bool": { "must": [], "filter": [], "should": [], "must_not": [] } }, { "range": { "@timestamp": { "gte": 1582070330, "lte": 11582070350 } } } ] } } } ``` Again, notice this part: ```auto "range": { "@timestamp": { "gte": 1582070330, "lte": 11582070350 } } ``` That returns nothing. When I switch it to instead use `epoch_seconds`, it returns my logs. I'm still not sure if that's related to this issue - you think I should open another ticket for that? ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [February 20, 2020, 12:11pm UTC](https://discuss.elastic.co/t/shards-failed-warning-on-network-dashboard-in-siem-app/219908/5 "2020-02-20T12:11:19Z")
Can you post your mappings for the `@timestamp` field, please? And maybe a sample document as returned by the Elasticsearch API (or copied from discover)? I would expect that as long as the field has a date type, the query would still work well. I had another thought about the Network page: is it possible to click the "Inspect" buttons from the top of the widgets? If that works, it might allow us to identify which one of the queries trips over. ---
**Author:** ![wconnell](https://avatars.discourse-cdn.com/v4/letter/w/f4b2a3/32.png) [@wconnell](https://discuss.elastic.co/u/wconnell)\ **Post date:** [February 20, 2020, 6:35pm UTC](https://discuss.elastic.co/t/shards-failed-warning-on-network-dashboard-in-siem-app/219908/6 "2020-02-20T18:35:15Z")
Mappings for `@timestamp` are as follows. I recently switched to using epoch\_second: ```auto "@timestamp": { "format": "yyyy-MM-dd HH:mm:ss||epoch_second", "index": true, "ignore_malformed": false, "store": false, "type": "date", "doc_values": true }, ``` Here's an example document: ```auto { "_index" : "some-index-2020.02.16-000014", "_type" : "_doc", "_id" : "AR7AVnABzoWS80HG3IVd", "_score" : 0.0, "_source" : { "destination" : { "geo" : { "continent_name" : "North America", "region_iso_code" : "US-WA", "city_name" : "Seattle", "country_iso_code" : "US", "country_name" : "United States", "region_name" : "Washington", "location" : { "lon" : -102.2432, "lat" : 27.24 } }, "as" : { "number" : 12345, "organization" : { "name" : "Google, Inc." } }, "port" : 443, "bytes" : 1234, "ip" : "1.2.3.4" }, "source" : { "bytes" : 1234, "ip" : "192.168.1.1" }, "firewall" : { "logs" : { "rule" : "interwebz", "url" : { "type" : "troubleshooting" } } }, "frequency" : 4, "tags" : [ "help", "please" ], "network" : { "application" : "ssl", "bytes" : 1234, "transport" : "tcp" }, "observer" : { "hostname" : "something.com" }, "@timestamp" : 1581983940, "event.module" : "firewall", "related" : { "ip" : [ "1.2.3.4", "192.168.1.1" ] }, "event" : { "dataset" : "firewall.logs", "outcome" : "permit" } } } ``` Again, since storing the logs in `epoch_second` format, none of the visualizations in the SIEM app show my logs. I do see these logs in Discover, Dashboards, and Visualizations. When I hit inspect and run the query against the console, it returns nothing (even though I definitely have logs for that time period). I get it to return my logs when I include `"format": "epoch_millis"` in the queries: ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/b/1b1da95e52a6a0fd68b6394db0c2c3b61b27c6c0.png) Note that the Inspect tool on all the visualizations does not include this format field: ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/f/df8e4648e64749ffb0cdea3347bfb91e642c886e.png) And for comparison, the Discover page Inspect includes the format with the range (albeit a different format): ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/a/3a5185564503eed928e39c63f1d62fa491663c74.png) Not sure if this is related to my issue or a completely separate issue altogether. ---
**Author:** ![wconnell](https://avatars.discourse-cdn.com/v4/letter/w/f4b2a3/32.png) [@wconnell](https://discuss.elastic.co/u/wconnell)\ **Post date:** [February 25, 2020, 10:05pm UTC](https://discuss.elastic.co/t/shards-failed-warning-on-network-dashboard-in-siem-app/219908/7 "2020-02-25T22:05:20Z")
Pinging this thread, thanks! Please let me know if you have other ideas for troubleshooting. ---
**Author:** ![Magnus\_Kessler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnus_kessler/32/42001_2.png) [@Magnus\_Kessler](https://discuss.elastic.co/u/Magnus_Kessler)\ **Post date:** [March 2, 2020, 1:37pm UTC](https://discuss.elastic.co/t/shards-failed-warning-on-network-dashboard-in-siem-app/219908/8 "2020-03-02T13:37:17Z")
The format of a `date` data type field determines which formats are understood during indexing, but also which date formats can be used for querying. If the field is set to `epoch_seconds`, only the numerical form is allowed. The [default](https://www.elastic.co/guide/en/elasticsearch/reference/current/date.html) is to allow for ISO8601 formatted strings as well as `epoch_millis`. Kibana assumes one of these and fails with `epoch_second`. If the time format absolutely has to be `epoch_second`, I'd recommend setting the [format string](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html) to something like `"strict_date_optional_time||epoch_second"`. Alternatively (and preferred IMO), the values could be adjusted (seconds multiplied by `1000`) to be in `epoch_millis`. Mixing `epoch_second` with `epoch_millis` is not possible. ---
**Author:** ![Frank\_Hassanabad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/frank_hassanabad/32/49255_2.png) [@Frank\_Hassanabad](https://discuss.elastic.co/u/Frank_Hassanabad)\ **Post date:** [March 3, 2020, 12:04am UTC](https://discuss.elastic.co/t/shards-failed-warning-on-network-dashboard-in-siem-app/219908/9 "2020-03-03T00:04:40Z")
I would recommend for now wconnell that you switch over to using: ```auto strict_date_optional_time||epoch_millis ``` and re-mapping and re-indexing. This is the default for Elastic Search when it does its dynamic date mappings: [https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html#date-detection](https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-field-mapping.html#date-detection) When you insert data, you can insert your data as `epoch_millis` or as an ISO8601 standard which would be something like: ```auto 2020-03-02T23:55:17.303Z ``` I would suggest inserting your data into your index as ISO8601 with zulu as that is the path tested the most by us as it is the default Elastic Search choice for date times. However, inserting your data as `epoch_seconds` will be really bad for the SIEM application to interpret things as it is making an assumption that what you have is Epoch in milliseconds and not in regular seconds when it sees numbers. Magnus\_Kessler is correct in the assumption that are expecting milliseconds and not seconds when it comes to `Epoch`. Magnus\_Kessler, you are correct that there is a difference between how the SIEM application is processing date times vs how Discover is processing date times with regards to both input and output. We are tracking both of these in these two tickets at the moment and put notes on both of them for this: > > > As highlighted in this discuss topic, when @timestamp values are in a format that do not contain timezone designators, they will... > > > Describe the bug: > In the SIEM application if you have a particular type of custom date time mapping shown below you will... We are hoping to make things in the future more mapping agnostic and friendly with date timestamps but for now I would re-index and re-map using: ```auto strict_date_optional_time||epoch_millis ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 31, 2020, 12:04am UTC](https://discuss.elastic.co/t/shards-failed-warning-on-network-dashboard-in-siem-app/219908/10 "2020-03-31T00:04:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.