ELK 5.5 : Winlogbeats no Messages displayed form ForwardedEvents

Hi,

I cannot see any messages of ForwardedEvents (WinlogBeats 5.5) in the WebUI.

Setup:
Many Remote Servers <-> Windows Log Subscription <-> One Windows Server Log Collector (Winlogbeats) <-> ELK Stak Server

Messages form local Server Events on the collector are visible.

How does the Best Practice Config look like for that case? e.g. Winlogbeat config, Logstash Input config, Elasticsearch output Config

My Logstash Input config

    input {
      beats {
        port => 5044
        ssl => true
        ssl_certificate => ".."
        ssl_key => ".."
      } 
}

My Elastic output config

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

My winlogbeat config

#======================= Winlogbeat specific options ==========================
winlogbeat.event_logs:
  - name: Application
  - name: Security
  - name: System
  - name: ForwardedEvents
    forwarded: true
    #include_xml: true

#================================ General =====================================


#================================ Outputs =====================================


#-------------------------- Elasticsearch output ------------------------------


#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["xxx.yyy.zzz.aaa:5044"]
  bulk_max_size: 1024

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["..."]

  # Certificate for SSL client authentication
  #ssl.certificate: ".."

  # Client Certificate Key
  #ssl.key: ".."

#================================ Logging =====================================

#logging.selectors: ["*"]

Kind Regards

The configuration looks fine. Events from the other event logs are present in Elasticsearch?

Can you share the log file from Winlogbeat?

How many events are there in the ForwardedEvents log?

PS C:\Users\vagrant> Get-WinEvent -ListLog ForwardedEvents                                                                                                                                         
                                                                                                                                                                                                   
LogMode   MaximumSizeInBytes RecordCount LogName                                                                                                                                                   
-------   ------------------ ----------- -------                                                                                                                                                   
Circular            20971520             ForwardedEvents

How many events are in Elasticsearch? What's the result of this query using the Dev Console in Kibana.

GET winlogbeat-*/_search
{
  "size": 0,
  "aggs": {
    "eventlogs": {
      "terms": {
        "field": "log_name",
        "size": 10,
        "order": {
          "_count": "desc"
        }
      }
    }
  }
}

Hi Andrew,

thank you for your Reply.

Can you share the log file from Winlogbeat? This is an excerpt of the Winlogbeat log, the entries keep repeating.

2017-07-17T15:29:28+02:00 INFO Metrics logging every 30s
2017-07-17T15:29:28+02:00 INFO Max Retries set to: 3
2017-07-17T15:29:28+02:00 INFO Activated logstash as output plugin.
2017-07-17T15:29:28+02:00 INFO Publisher name: <Name of LogCollector Server>
2017-07-17T15:29:28+02:00 INFO Flush Interval set to: 1s
2017-07-17T15:29:28+02:00 INFO Max Bulk Size set to: 1024
2017-07-17T15:29:28+02:00 INFO State will be read from and persisted to C:\ProgramData\winlogbeat\.winlogbeat.yml
2017-07-17T15:29:28+02:00 INFO winlogbeat start running.
2017-07-17T15:29:28+02:00 INFO EventLog[System] Successfully published 1 events
2017-07-17T15:29:29+02:00 INFO EventLog[System] Successfully published 1 events
2017-07-17T15:29:34+02:00 INFO EventLog[ForwardedEvents] Successfully published 84 events
2017-07-17T15:29:35+02:00 INFO EventLog[ForwardedEvents] Successfully published 20 events
2017-07-17T15:29:38+02:00 INFO EventLog[ForwardedEvents] Successfully published 92 events
2017-07-17T15:29:39+02:00 INFO EventLog[ForwardedEvents] Successfully published 21 events
2017-07-17T15:29:50+02:00 INFO EventLog[ForwardedEvents] Successfully published 100 events
2017-07-17T15:29:50+02:00 INFO EventLog[ForwardedEvents] Successfully published 5 events
2017-07-17T15:29:51+02:00 INFO EventLog[ForwardedEvents] Successfully published 1 events
2017-07-17T15:29:58+02:00 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=9 libbeat.logstash.publish.read_bytes=1872 libbeat.logstash.publish.write_bytes=57631 libbeat.logstash.published_and_acked_events=325 libbeat.publisher.published_events=325 msg_file_cache.SystemHits=1 msg_file_cache.SystemMisses=1 msg_file_cache.SystemSize=1 published_events.ForwardedEvents=323 published_events.System=2 published_events.total=325 uptime={"server_time":"2017-07-17T13:29:58.77702Z","start_time":"2017-07-17T13:29:28.7588712Z","uptime":"30.0181488s","uptime_ms":"30018148"}
2017-07-17T15:30:05+02:00 INFO EventLog[ForwardedEvents] Successfully published 100 events

How many events are there in the ForwardedEvents log?

LogMode   MaximumSizeInBytes RecordCount LogName
-------   ------------------ ----------- -------
Circular            20971520        8996 ForwardedEvents

How many events are in Elasticsearch? What's the result of this query using the Dev Console in Kibana.

{
  "took": 66,
  "timed_out": false,
  "_shards": {
    "total": 35,
    "successful": 35,
    "failed": 0
  },
  "hits": {
    "total": 12847666,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "eventlogs": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": []
    }
  }
}

Kind Regards

The response from the Elasticsearch aggregation is not what I expected. buckets is empty, but there should be one bucket for each event log.

Can you re-run the query, but set the size such that I can inspect an event.

GET winlogbeat-*/_search
{
  "size": 3,
  "aggs": {
    "eventlogs": {
      "terms": {
        "field": "log_name",
        "size": 10,
        "order": {
          "_count": "desc"
        }
      }
    }

How are you identifying the forwarded events? I recommend adding a tag to the forwarded events to make it easy to identify them.

winlogbeat.event_logs:
  - name: ForwardedEvents
    forwarded: true
    tags: [forwarded]

Hi Andrew,

By now I only use ELK 5.5. for ForwardedEvents, but later I'll consider using tags to seperate them from possible new log sources.

Example from query with set size

    {
  "took": 84,
  "timed_out": false,
  "_shards": {
    "total": 40,
    "successful": 40,
    "failed": 0
  },
  "hits": {
    "total": 14903696,
    "max_score": 1,
    "hits": [
      {
        "_index": "winlogbeat-2017.07.12",
        "_type": "wineventlog",
        "_id": "AV02_h2ZmuQ2iIRRzkYm",
        "_score": 1,
        "_source": {
          "process_id": <ID>,
          "computer_name": "<Source Computer>",
          "level": "Information",
          "log_name": "Security",
          "record_number": "662617486",
          "event_data": {
            "ProcessName": "-",
            "LogonGuid": "{abc}",
            "LogonType": "3",
            "IpPort": "2",
            "TransmittedServices": "-",
            "SubjectLogonId": "...",
            "KeyLength": "0",
            "LmPackageName": "-",
            "TargetLogonId": "...",
            "SubjectUserName": "-",
            "WorkstationName": "-",
            "IpAddress": "www.xxx.yyy.zzz",
            "SubjectDomainName": "-",
            "ProcessId": "...",
            "TargetUserName": "<User Accountt>",
            "ImpersonationLevel": "%%1840",
            "LogonProcessName": "<ProcessName>",
            "TargetDomainName": "<Domain Name>",
            "SubjectUserSid": "<SID>",
            "TargetUserSid": "<SID>",
            "AuthenticationPackageName": "<ProcessName>"
          },
          "type": "wineventlog",
          "version": 1,
          "tags": [
            "beats_input_raw_event"
          ],
          "thread_id": 648,
          "@timestamp": "2017-07-12T13:29:55.912Z",
          "event_id": 4624,
          "provider_guid": "{<GUID>}",
          "beat": {
            "hostname": "<Computer Name Log Collector>",
            "name": "<Computer Name Log Collector>",
            "version": "5.5.0"
          },
          "@version": "1",
          "host": "<Computer Name Log Collector>",
          "source_name": "Microsoft-Windows-Security-Auditing"
        }
      },

json example from webui

    {
      "_index": "winlogbeat-2017.07.14",
      "_type": "wineventlog",
      "_id": "AV1AnYrmGcHyfHArmD_L",
      "_version": 1,
      "_score": null,
      "_source": {
        "process_id": 588,
        "computer_name": "<Computer Name - Log Source>",
        "level": "Information",
        "log_name": "Security",
        "record_number": "665978255",
        "event_data": {
          "SubjectUserName": "<Some Accountt>",
          "MemberSid": "<SID>",
          "TargetSid": "<SID>",
          "SubjectDomainName": "<Domain>",
          "TargetUserName": "<Some User>",
          "MemberName": "<Some Vlaue>",
          "SubjectLogonId": "<Some ID>",
          "TargetDomainName": "<Domain>",
          "PrivilegeList": "<Empty>",
          "SubjectUserSid": "<SID>"
        },
    "type": "wineventlog",
    "tags": [
      "beats_input_raw_event"
    ],
    "thread_id": 3528,
    "@timestamp": "2017-07-14T10:20:46.606Z",
    "event_id": 4728,
    "provider_guid": "{<GUID>}",
    "beat": {
      "hostname": "<Computer Name Log Collector>",
      "name": "<Computer Name Log Collector>",
      "version": "5.5.0"
    },
    "@version": "1",
    "host": "<Computer Name Log Collector>",
    "source_name": "Microsoft-Windows-Security-Auditing"
  },
  "fields": {
    "@timestamp": [
      1500027646606
    ]
  },
  "highlight": {
    "event_data.MemberName": [
      <Some Entry>
    ]
  },
  "sort": [
    1500027646606
  ]
}

This event shows a very long message in the log source on the log collector which forwards the "forwardedevents" source via winlogbeat to logstash and Elasticsearch.

Thank you for your support.

Kind Regards

Can you check the configuration of the windows event subscription using wecutil. You can dump the subscription config to XML using wecutil and share it here. I'm curious to see if Format has been set to Events or RenderedText. It should be RenderedText.

What Windows versions are used (remote servers and the collector machine)? (I saw a KB article affecting message on 2008.)

Can you please set include_xml: true in the config and collect an forwarded event. I'd like to see if RenderingInfo is included in the XML.

Then can you see try setting forwarded: false and see if that makes a difference. This would cause the message to be rendered on the collector machine using resources from the collector machine (this isn't always ideal because the collect machine doesn't necessarily have the same resources (DLLs, EXEs) as the machine that originated the event).

Hi Andrew,

Setting forwarded: false solved my problem it seems that the forwarder needs to render the messages.

Thank you for your support. =)

Kind Regards

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.