# SIEM doesn't show any Winlogbeat events, despite ES receiving them **URL:** \ **Category:** SIEM\ **Created:** [March 18, 2020, 12:21am UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008 "2020-03-18T00:21:35Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Aura](https://avatars.discourse-cdn.com/v4/letter/a/c4cdca/32.png) [@Aura](https://discuss.elastic.co/u/Aura)\ **Post date:** [March 18, 2020, 12:21am UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/1 "2020-03-18T00:21:35Z")
Hi, New ELK stack user here. I just installed v7.6.1 on a Ubuntu Server 18.04 that I have running on an old Lenovo I had lying around. Took me a couple of hours to install, set up, and get working. One issue I can't seem to fix, nor explain, is that despite receiving Winlogbeat events, these aren't shown/reflected in the SIEM app, but other events (Filebeat and Packetbeat) are. I'm using Logstash in the middle, so Filebeat, Packetbeat and Winlogbeat are sending events to Logstash, which outputs them to Elasticsearch. My index patterns are all defined on the beat name: filebeat-\* packetbeat-\* winlogbeat-\* Has anyone encountered this issue before and if so, how did you solve it? ![](https://us1.discourse-cdn.com/elastic/original/3X/a/5/a56af166d7f63b87d71ad3b108f6cfcf04644625.png) Thank you! ---
**Author:** ![Aura](https://avatars.discourse-cdn.com/v4/letter/a/c4cdca/32.png) [@Aura](https://discuss.elastic.co/u/Aura)\ **Post date:** [March 21, 2020, 7:19pm UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/2 "2020-03-21T19:19:49Z")
No one has any idea as to why this is happening? I'll take any pointers or feedback you can give me. ---
**Author:** ![Mike\_Paquette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_paquette/32/119011_2.png) [@Mike\_Paquette](https://discuss.elastic.co/u/Mike_Paquette)\ **Post date:** [March 22, 2020, 12:28pm UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/3 "2020-03-22T12:28:36Z")
Hi, thanks for checking out Elastic SIEM. There are several reasons that your winlogbeat events might not populate all the tables in the SIEM app. Often these can be related to how winlogbeat was set up, or whether winlogbeat "modules" have been configured (e.g., [https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-module-security.html](https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-module-security.html)). Questions: You show that your SIEM overview page host events widget is not recognizing your winlogbeat events - can you see your winlogbeat events being populated in any other SIEM views, such as the Hosts views? Can you see your winlogbeat events in Kibana Discover? If so, could you attach a JSON format representation of an event here? (Be sure to anonymize any private fields values before sharing)? ---
**Author:** ![Aura](https://avatars.discourse-cdn.com/v4/letter/a/c4cdca/32.png) [@Aura](https://discuss.elastic.co/u/Aura)\ **Post date:** [March 26, 2020, 1:16pm UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/4 "2020-03-26T13:16:46Z")
Sorry for the delay, it seems like this time it is I who forgot to check my notifications. The Winlogbeat events aren't populating any other SIEM views, no. I can see my Winlogbeat events in Kibana Discover, yes. And sure, here it is: ```auto { "_index": "winlogbeat-7.6.1-2020.03.26", "_type": "_doc", "_id": "fSXqFHEBaYYLgoekEj_l", "_version": 1, "_score": null, "_source": { "ecs": { "version": "1.4.0" }, "tags": [ "beats_input_codec_plain_applied" ], "beat": { "ip": "192.168.1.10" }, "agent": { "hostname": "MY-COMPUTER", "id": "e6f9c935-a991-4f62-89d2-495187cb36f1", "version": "7.6.1", "type": "winlogbeat", "ephemeral_id": "a73af4a2-625b-4a54-850c-d8acc209acaa" }, "user": { "domain": "Window Manager", "id": "S-1-5-90-0-3", "name": "DWM-3" }, "@timestamp": "2020-03-26T03:36:54.989Z", "message": "An account was logged off.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-90-0-3\n\tAccount Name:\t\tDWM-3\n\tAccount Domain:\t\tWindow Manager\n\tLogon ID:\t\t0x956C7C0\n\nLogon Type:\t\t\t2\n\nThis event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.", "log": { "level": "information" }, "@version": "1", "host": { "os": { "kernel": "10.0.18362.720 (WinBuild.160101.0800)", "name": "Windows 10 Pro", "platform": "windows", "family": "windows", "version": "10.0", "build": "18363.720" }, "hostname": "MY-COMPUTER", "id": "477ecab9-ba80-47a6-92e2-8b0109f9059f", "architecture": "x86_64", "name": "MY-COMPUTER" }, "winlog": { "logon": { "id": "0x956c7c0", "type": "Interactive" }, "event_data": { "TargetDomainName": "Window Manager", "TargetUserSid": "S-1-5-90-0-3", "TargetUserName": "DWM-3", "TargetLogonId": "0x956c7c0", "LogonType": "2" }, "task": "Logoff", "opcode": "Info", "provider_name": "Microsoft-Windows-Security-Auditing", "api": "wineventlog", "record_id": 498879, "computer_name": "MY-COMPUTER", "channel": "Security", "event_id": 4634, "keywords": [ "Audit Success" ], "provider_guid": "{54849625-5478-4994-a5ba-3e3b0328c30d}", "process": { "thread": { "id": 8040 }, "pid": 864 } }, "event": { "kind": "event", "module": "security", "code": 4634, "provider": "Microsoft-Windows-Security-Auditing", "action": "logged-out", "created": "2020-03-26T03:36:56.013Z" } }, "fields": { "@timestamp": [ "2020-03-26T03:36:54.989Z" ], "event.created": [ "2020-03-26T03:36:56.013Z" ] }, "sort": [ 1585193814989 ] } ``` ---
**Author:** ![Aura](https://avatars.discourse-cdn.com/v4/letter/a/c4cdca/32.png) [@Aura](https://discuss.elastic.co/u/Aura)\ **Post date:** [April 1, 2020, 10:31pm UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/5 "2020-04-01T22:31:44Z")
Any update on this? Thank you! ---
**Author:** ![Aura](https://avatars.discourse-cdn.com/v4/letter/a/c4cdca/32.png) [@Aura](https://discuss.elastic.co/u/Aura)\ **Post date:** [April 8, 2020, 2:40am UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/6 "2020-04-08T02:40:21Z")
Another shameless bump. ---
**Author:** ![Frank\_Hassanabad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/frank_hassanabad/32/49255_2.png) [@Frank\_Hassanabad](https://discuss.elastic.co/u/Frank_Hassanabad)\ **Post date:** [April 8, 2020, 5:13pm UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/7 "2020-04-08T17:13:10Z")
I just tested your document like so, in a browse window I got a new date time stamp like so: ```auto new Date().toISOString() "2020-04-08T17:04:07.198Z" ``` Added it as your `@timestamp` so that the last 24 hours time range would work. Then did: ```auto POST winlogbeat-7.6.0/_doc/test-id ... your document with the timestamp replaced ... ``` Note I am using 7.6.0 though. Then I looked at the hosts table and I see it showing up. ![Screen Shot 2020-04-08 at 11.09.06 AM](https://us1.discourse-cdn.com/elastic/original/3X/8/b/8b2a88e8c0d545a5e0a9e3b57226ace3cec8a423.png) There are inspect buttons next to the displays if you want to play with the query inside of developer tools to see maybe why you're not seeing data in a particular display? ![Screen Shot 2020-04-08 at 11.11.37 AM](https://us1.discourse-cdn.com/elastic/original/3X/f/3/f39f5d0191a8be9ca6f8af985b96c8991bd6aba0.png) ---
**Author:** ![Aura](https://avatars.discourse-cdn.com/v4/letter/a/c4cdca/32.png) [@Aura](https://discuss.elastic.co/u/Aura)\ **Post date:** [April 9, 2020, 3:20am UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/8 "2020-04-09T03:20:18Z")
Thank you for the reply, it seems that we might have a clue in the Response that I'm getting. I have 2 Winlogbeat indexes showing up, however, I'm getting errors for both. The Filebeat index works just fine though: ``` { "took": 12, "timed_out": false, "_shards": { "total": 151, "successful": 149, "skipped": 147, "failed": 2, "failures": [ { "shard": 0, "index": "winlogbeat-7.6.1-2020.04.08", "node": "lLto6QkkSSeIYqe672AYRA", "reason": { "type": "illegal_argument_exception", "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [host.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory." } }, { "shard": 0, "index": "winlogbeat-7.6.1-2020.04.09", "node": "lLto6QkkSSeIYqe672AYRA", "reason": { "type": "illegal_argument_exception", "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [host.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory." } } ] }, "hits": { "max_score": null, "hits": [] }, "aggregations": { "host_data": { "doc_count_error_upper_bound": 0, "sum_other_doc_count": 0, "buckets": [ { "key": "my_ubuntu_server", "doc_count": 4251, "lastSeen": { "value": 1586402083839, "value_as_string": "2020-04-09T03:14:43.839Z" }, "os": { "hits": { "total": { "value": 4251, "relation": "eq" }, "max_score": null, "hits": [ { "_index": "filebeat-7.6.1-2020.04.09", "_type": "_doc", "_id": "nSfuXHEBaYYLgoekyN1W", "_score": null, "_source": { "host": { "os": { "kernel": "4.15.0-88-generic", "codename": "bionic", "name": "Ubuntu", "family": "debian", "version": "18.04.4 LTS (Bionic Beaver)", "platform": "ubuntu" } } }, "sort": [ 1586402083839 ] } ] } } } ] }, "host_count": { "value": 1 } } } ``` It seems like these errors are centered around the host.name field, which would explain why Winlogbeat hosts aren't showing up in the Hosts section of the SIEM. ---
**Author:** ![Mike\_Paquette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_paquette/32/119011_2.png) [@Mike\_Paquette](https://discuss.elastic.co/u/Mike_Paquette)\ **Post date:** [April 9, 2020, 11:34am UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/9 "2020-04-09T11:34:35Z")
Hi @aura, as you've discovered, it seems that some of your fields are not being mapped to the correct Elasticsearch [datatypes](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/mapping-types.html). Elastic Common Schema (ECS) specifies that fields such as `host.name` must be of the [keyword](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/keyword.html) datatype. Indeed, as you've also discovered, Elastic SIEM relies specifically on `host.name` to be present as a condition for populating many visualizations with host data. I'm not sure how familiar you are with Elasticsearch [mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html) and [index templates](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/indices-templates.html), as they can be challenging. The ECS GitHub repository provides an example template [here](https://github.com/elastic/ecs/blob/1.5/generated/elasticsearch/7/template.json). FYI, we've been pulling together some additional notes about ECS fields required to use Elastic SIEM. We hope these might be helpful as you add additional data sources to your SIEM. #### Notes for Elastic SIEM users The Elastic SIEM app queries, filters, aggregates, and/or displays a growing list of ECS fields. For an optimal experience with Elastic SIEM, and to reduce time spent on updating your ingestion pipelines in the future, please follow the [Converting your data to ECS](https://github.com/MikePaquette/ecs-1/blob/update-converting-doc/docs/converting.asciidoc#ecs-conv) instructions, populating as many ECS fields as practical, paying particular attention to: - Ensure that your network events populate ECS `source` and `destination` fields. Network events may not be displayed in the SIEM app network views if `source.ip` and `destination.ip` fields are not populated. - If you have network protocol events, be sure to populate the `http` , `dns` , and `tls` field sets. - Ensure that your host events populate ECS `host` fields. Host events will not be displayed in the SIEM app host views if the `host.name` field is not populated. Recall that `host.name` should be populated with the name of the host where the event happened, not the name of a collector or observer. - If your host events contain details about process activity, ensure that your fields are mapped to the corresponding ECS `process` fields. - Be sure that for all your events, you populate the ECS categorization fields: `event.kind` , `event.category` , `event.type` , and `event.outcome` . SIEM authentication and process widgets depend on these fields being populated. Also, try to populate the ECS `event.action` field with relevant details for your event. ---
**Author:** ![Frank\_Hassanabad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/frank_hassanabad/32/49255_2.png) [@Frank\_Hassanabad](https://discuss.elastic.co/u/Frank_Hassanabad)\ **Post date:** [April 10, 2020, 1:55pm UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/10 "2020-04-10T13:55:07Z")
@Aura, ahhh, I think I might see what's going on. I bet when you first setup winlog beat you accidentally forgot to push your templates which control the mapping? [https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html](https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html) You can check your mapping in your dev tools: ![Screen Shot 2020-04-10 at 7.47.05 AM](https://us1.discourse-cdn.com/elastic/original/3X/4/9/491d7d95cfca5bbd7928e2db0c98c46eafde18fc.png) like so....To see the entire mapping of your winlog beats: ```auto GET winlogbeat-7.6.0/_mapping ``` ![Screen Shot 2020-04-10 at 7.48.30 AM](https://us1.discourse-cdn.com/elastic/original/3X/6/d/6d8c61147c4ea14c45ba43ae4d555ce2841ecc94.png) If you want to just concentrate on the field in question which sounds like `host.name` it would be like so: ```auto GET winlogbeat-7.6.0/_mapping/field/host.name ``` ![Screen Shot 2020-04-10 at 7.50.25 AM](https://us1.discourse-cdn.com/elastic/original/3X/b/a/baf8d506836cc7596b3e7d269c8a452efb093aa3.png) You can see in my screen shot that `host.name` for my winlog beats is a keyword field. If for some reason it showed up as text field I would expect the error you have above showing up which indicates that you probably just need to overwrite your winlogbeat per that link above to fix things and then maybe do a reindex of existing data to change it from text to keyword fields. Which would make sense. Without an initial template/mapping loaded Elastic Search is going to auto-create your mapping and take guesses about things being text fields and keyword fields and then when our application tries to do an aggregation or sort you will see those errors showing up. ---
**Author:** ![Aura](https://avatars.discourse-cdn.com/v4/letter/a/c4cdca/32.png) [@Aura](https://discuss.elastic.co/u/Aura)\ **Post date:** [April 10, 2020, 8:08pm UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/11 "2020-04-10T20:08:53Z")
Hi Frank, You're right, I wasn't able to import the templates from Winlogbeat as it needed a direct connection to ES, and I'm sending the beats to Logstash directly. So I'm guessing that following the instructions in this documentation should fix the issue (under "Load the template manually")? [https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html](https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html) Also, I ran your query, but on winlogbeat-7.6.1-2020.04.10 (I don't have a single "winlogbeat-7.6.1" index, a new one gets created everyday with the date...): ``` { "winlogbeat-7.6.1-2020.04.10" : { "mappings" : { "host.name" : { "full_name" : "host.name", "mapping" : { "name" : { "type" : "text", "fields" : { "keyword" : { "type" : "keyword", "ignore_above" : 256 } } } } } } } } ``` If I push the templates to ES, will I need to delete and re-create the indexes to refresh everything properly? ---
**Author:** ![Frank\_Hassanabad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/frank_hassanabad/32/49255_2.png) [@Frank\_Hassanabad](https://discuss.elastic.co/u/Frank_Hassanabad)\ **Post date:** [April 10, 2020, 8:38pm UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/12 "2020-04-10T20:38:13Z")
Nice! Sounds like we got to the bottom of this. > If I push the templates to ES, will I need to delete and re-create the indexes to refresh everything properly? Yeah I would. I would probably do a reindex but I usually work with small test sets so be careful with my advice you know 😉 : > **[Reindex API | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html)** Also for the mappings can use the instructions you found above or you can also export the mapping and then copy it over to your system and load it from dev tools or curl or any other way you want to if that helps: > **[Load the Elasticsearch index template | Winlogbeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html#load-template-manually-alternate)** Usually people use either an alias to point to the current beat or they setup ILM rules for what it's worth ... Alias's make it easier to do re-indexing and keep old data around and conduct testing before switching the alias to the new index. ILM has alias support and other benefits such as hot/warm architecture and aging of data sets which is why we use it with most things out of the box these days with beats and data indexes: > **[Aliases API | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html)** > **[Tutorial: Automate rollover with ILM | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html)** During normal beats setup it does ILM rules. Here is my ILM explain for example: ```auto GET winlogbeat-7.6.0-2020.03.03-000001/_ilm/explain "indices" : { "winlogbeat-7.6.0-2020.03.03-000001" : { "index" : "winlogbeat-7.6.0-2020.03.03-000001", "managed" : true, "policy" : "winlogbeat", "lifecycle_date_millis" : 1585859758678, "age" : "7.99d", "phase" : "hot", "phase_time_millis" : 1583267301213, "action" : "complete", "action_time_millis" : 1585859760532, "step" : "complete", "step_time_millis" : 1585859760532, "phase_execution" : { "policy" : "winlogbeat", "phase_definition" : { "min_age" : "0ms", "actions" : { "rollover" : { "max_size" : "50gb", "max_age" : "30d" } } }, "version" : 1, "modified_date_in_millis" : 1583266497938 } } } } ``` But since you have a custom system setup its up to you how you want to setup your hot/warm architecture and alias's, etc... As long as you get those mappings in everything will be right again. Good luck ☘ ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 8, 2020, 8:38pm UTC](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/13 "2020-05-08T20:38:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.