SIEM doesn't show any Winlogbeat events, despite ES receiving them

[Aura]

Hi,

New ELK stack user here. I just installed v7.6.1 on a Ubuntu Server 18.04 that I have running on an old Lenovo I had lying around. Took me a couple of hours to install, set up, and get working. One issue I can't seem to fix, nor explain, is that despite receiving Winlogbeat events, these aren't shown/reflected in the SIEM app, but other events (Filebeat and Packetbeat) are.

I'm using Logstash in the middle, so Filebeat, Packetbeat and Winlogbeat are sending events to Logstash, which outputs them to Elasticsearch.

My index patterns are all defined on the beat name:

filebeat-*
packetbeat-*
winlogbeat-*

Has anyone encountered this issue before and if so, how did you solve it?

Thank you!

[Aura]

No one has any idea as to why this is happening? I'll take any pointers or feedback you can give me.

[Mike_Paquette]

Hi, thanks for checking out Elastic SIEM. There are several reasons that your winlogbeat events might not populate all the tables in the SIEM app. Often these can be related to how winlogbeat was set up, or whether winlogbeat "modules" have been configured (e.g., https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-module-security.html).

Questions:
You show that your SIEM overview page host events widget is not recognizing your winlogbeat events - can you see your winlogbeat events being populated in any other SIEM views, such as the Hosts views?

Can you see your winlogbeat events in Kibana Discover? If so, could you attach a JSON format representation of an event here? (Be sure to anonymize any private fields values before sharing)?

[Aura]

Sorry for the delay, it seems like this time it is I who forgot to check my notifications.

The Winlogbeat events aren't populating any other SIEM views, no.

I can see my Winlogbeat events in Kibana Discover, yes. And sure, here it is:

{
  "_index": "winlogbeat-7.6.1-2020.03.26",
  "_type": "_doc",
  "_id": "fSXqFHEBaYYLgoekEj_l",
  "_version": 1,
  "_score": null,
  "_source": {
    "ecs": {
      "version": "1.4.0"
    },
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "beat": {
      "ip": "192.168.1.10"
    },
    "agent": {
      "hostname": "MY-COMPUTER",
      "id": "e6f9c935-a991-4f62-89d2-495187cb36f1",
      "version": "7.6.1",
      "type": "winlogbeat",
      "ephemeral_id": "a73af4a2-625b-4a54-850c-d8acc209acaa"
    },
    "user": {
      "domain": "Window Manager",
      "id": "S-1-5-90-0-3",
      "name": "DWM-3"
    },
    "@timestamp": "2020-03-26T03:36:54.989Z",
    "message": "An account was logged off.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-90-0-3\n\tAccount Name:\t\tDWM-3\n\tAccount Domain:\t\tWindow Manager\n\tLogon ID:\t\t0x956C7C0\n\nLogon Type:\t\t\t2\n\nThis event is generated when a logon session is destroyed. It may be positively correlated with a logon event using the Logon ID value. Logon IDs are only unique between reboots on the same computer.",
    "log": {
      "level": "information"
    },
    "@version": "1",
    "host": {
      "os": {
        "kernel": "10.0.18362.720 (WinBuild.160101.0800)",
        "name": "Windows 10 Pro",
        "platform": "windows",
        "family": "windows",
        "version": "10.0",
        "build": "18363.720"
      },
      "hostname": "MY-COMPUTER",
      "id": "477ecab9-ba80-47a6-92e2-8b0109f9059f",
      "architecture": "x86_64",
      "name": "MY-COMPUTER"
    },
    "winlog": {
      "logon": {
        "id": "0x956c7c0",
        "type": "Interactive"
      },
      "event_data": {
        "TargetDomainName": "Window Manager",
        "TargetUserSid": "S-1-5-90-0-3",
        "TargetUserName": "DWM-3",
        "TargetLogonId": "0x956c7c0",
        "LogonType": "2"
      },
      "task": "Logoff",
      "opcode": "Info",
      "provider_name": "Microsoft-Windows-Security-Auditing",
      "api": "wineventlog",
      "record_id": 498879,
      "computer_name": "MY-COMPUTER",
      "channel": "Security",
      "event_id": 4634,
      "keywords": [
        "Audit Success"
      ],
      "provider_guid": "{54849625-5478-4994-a5ba-3e3b0328c30d}",
      "process": {
        "thread": {
          "id": 8040
        },
        "pid": 864
      }
    },
    "event": {
      "kind": "event",
      "module": "security",
      "code": 4634,
      "provider": "Microsoft-Windows-Security-Auditing",
      "action": "logged-out",
      "created": "2020-03-26T03:36:56.013Z"
    }
  },
  "fields": {
    "@timestamp": [
      "2020-03-26T03:36:54.989Z"
    ],
    "event.created": [
      "2020-03-26T03:36:56.013Z"
    ]
  },
  "sort": [
    1585193814989
  ]
}

[Aura]

Any update on this? Thank you!

[Aura]

Another shameless bump.

[Frank_Hassanabad]

I just tested your document like so, in a browse window I got a new date time stamp like so:

new Date().toISOString()
"2020-04-08T17:04:07.198Z"

Added it as your @timestamp so that the last 24 hours time range would work.

Then did:

POST winlogbeat-7.6.0/_doc/test-id
... your document with the timestamp replaced ...

Note I am using 7.6.0 though. Then I looked at the hosts table and I see it showing up.

There are inspect buttons next to the displays if you want to play with the query inside of developer tools to see maybe why you're not seeing data in a particular display?

[Aura]

Thank you for the reply, it seems that we might have a clue in the Response that I'm getting. I have 2 Winlogbeat indexes showing up, however, I'm getting errors for both. The Filebeat index works just fine though:

{
  "took": 12,
  "timed_out": false,
  "_shards": {
    "total": 151,
    "successful": 149,
    "skipped": 147,
    "failed": 2,
    "failures": [
      {
        "shard": 0,
        "index": "winlogbeat-7.6.1-2020.04.08",
        "node": "lLto6QkkSSeIYqe672AYRA",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [host.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
        }
      },
      {
        "shard": 0,
        "index": "winlogbeat-7.6.1-2020.04.09",
        "node": "lLto6QkkSSeIYqe672AYRA",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [host.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
        }
      }
    ]
  },
  "hits": {
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "host_data": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": [
        {
          "key": "my_ubuntu_server",
          "doc_count": 4251,
          "lastSeen": {
            "value": 1586402083839,
            "value_as_string": "2020-04-09T03:14:43.839Z"
          },
          "os": {
            "hits": {
              "total": {
                "value": 4251,
                "relation": "eq"
              },
              "max_score": null,
              "hits": [
                {
                  "_index": "filebeat-7.6.1-2020.04.09",
                  "_type": "_doc",
                  "_id": "nSfuXHEBaYYLgoekyN1W",
                  "_score": null,
                  "_source": {
                    "host": {
                      "os": {
                        "kernel": "4.15.0-88-generic",
                        "codename": "bionic",
                        "name": "Ubuntu",
                        "family": "debian",
                        "version": "18.04.4 LTS (Bionic Beaver)",
                        "platform": "ubuntu"
                      }
                    }
                  },
                  "sort": [
                    1586402083839
                  ]
                }
              ]
            }
          }
        }
      ]
    },
    "host_count": {
      "value": 1
    }
  }
}

It seems like these errors are centered around the host.name field, which would explain why Winlogbeat hosts aren't showing up in the Hosts section of the SIEM.

[Mike_Paquette]

Hi @aura, as you've discovered, it seems that some of your fields are not being mapped to the correct Elasticsearch datatypes. Elastic Common Schema (ECS) specifies that fields such as host.name must be of the keyword datatype. Indeed, as you've also discovered, Elastic SIEM relies specifically on host.name to be present as a condition for populating many visualizations with host data.

I'm not sure how familiar you are with Elasticsearch mapping and index templates, as they can be challenging. The ECS GitHub repository provides an example template here.

FYI, we've been pulling together some additional notes about ECS fields required to use Elastic SIEM. We hope these might be helpful as you add additional data sources to your SIEM.

Notes for Elastic SIEM users

The Elastic SIEM app queries, filters, aggregates, and/or displays a growing list of ECS fields. For an optimal experience with Elastic SIEM, and to reduce time spent on updating your ingestion pipelines in the future, please follow the Converting your data to ECS instructions, populating as many ECS fields as practical, paying particular attention to:

• Ensure that your network events populate ECS source and destination fields. Network events may not be displayed in the SIEM app network views if source.ip and destination.ip fields are not populated.
• If you have network protocol events, be sure to populate the http , dns , and tls field sets.
• Ensure that your host events populate ECS host fields. Host events will not be displayed in the SIEM app host views if the host.name field is not populated. Recall that host.name should be populated with the name of the host where the event happened, not the name of a collector or observer.
• If your host events contain details about process activity, ensure that your fields are mapped to the corresponding ECS process fields.
• Be sure that for all your events, you populate the ECS categorization fields: event.kind , event.category , event.type , and event.outcome . SIEM authentication and process widgets depend on these fields being populated. Also, try to populate the ECS event.action field with relevant details for your event.

[Frank_Hassanabad]

@Aura, ahhh, I think I might see what's going on. I bet when you first setup winlog beat you accidentally forgot to push your templates which control the mapping?
https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html

You can check your mapping in your dev tools:

like so....To see the entire mapping of your winlog beats:

GET winlogbeat-7.6.0/_mapping

If you want to just concentrate on the field in question which sounds like host.name it would be like so:

GET winlogbeat-7.6.0/_mapping/field/host.name

You can see in my screen shot that host.name for my winlog beats is a keyword field. If for some reason it showed up as text field I would expect the error you have above showing up which indicates that you probably just need to overwrite your winlogbeat per that link above to fix things and then maybe do a reindex of existing data to change it from text to keyword fields.

Which would make sense. Without an initial template/mapping loaded Elastic Search is going to auto-create your mapping and take guesses about things being text fields and keyword fields and then when our application tries to do an aggregation or sort you will see those errors showing up.

[Aura]

Hi Frank,

You're right, I wasn't able to import the templates from Winlogbeat as it needed a direct connection to ES, and I'm sending the beats to Logstash directly. So I'm guessing that following the instructions in this documentation should fix the issue (under "Load the template manually")?

https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html

Also, I ran your query, but on winlogbeat-7.6.1-2020.04.10 (I don't have a single "winlogbeat-7.6.1" index, a new one gets created everyday with the date...):

{
  "winlogbeat-7.6.1-2020.04.10" : {
    "mappings" : {
      "host.name" : {
        "full_name" : "host.name",
        "mapping" : {
          "name" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          }
        }
      }
    }
  }
}

If I push the templates to ES, will I need to delete and re-create the indexes to refresh everything properly?

[Frank_Hassanabad]

Nice! Sounds like we got to the bottom of this.

If I push the templates to ES, will I need to delete and re-create the indexes to refresh everything properly?

Yeah I would. I would probably do a reindex but I usually work with small test sets so be careful with my advice you know :

Reindex API | Elasticsearch Guide [8.11] | Elastic

Also for the mappings can use the instructions you found above or you can also export the mapping and then copy it over to your system and load it from dev tools or curl or any other way you want to if that helps:

Load the Elasticsearch index template | Winlogbeat Reference [8.11] | Elastic

Usually people use either an alias to point to the current beat or they setup ILM rules for what it's worth ... Alias's make it easier to do re-indexing and keep old data around and conduct testing before switching the alias to the new index. ILM has alias support and other benefits such as hot/warm architecture and aging of data sets which is why we use it with most things out of the box these days with beats and data indexes:

Aliases API | Elasticsearch Guide [8.11] | Elastic

Tutorial: Automate rollover with ILM | Elasticsearch Guide [8.11] | Elastic

During normal beats setup it does ILM rules. Here is my ILM explain for example:

GET winlogbeat-7.6.0-2020.03.03-000001/_ilm/explain


  "indices" : {
    "winlogbeat-7.6.0-2020.03.03-000001" : {
      "index" : "winlogbeat-7.6.0-2020.03.03-000001",
      "managed" : true,
      "policy" : "winlogbeat",
      "lifecycle_date_millis" : 1585859758678,
      "age" : "7.99d",
      "phase" : "hot",
      "phase_time_millis" : 1583267301213,
      "action" : "complete",
      "action_time_millis" : 1585859760532,
      "step" : "complete",
      "step_time_millis" : 1585859760532,
      "phase_execution" : {
        "policy" : "winlogbeat",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1583266497938
      }
    }
  }
}

But since you have a custom system setup its up to you how you want to setup your hot/warm architecture and alias's, etc...

As long as you get those mappings in everything will be right again.

Good luck