WinLogBeat and Windows Server 2019

Having trouble getting WinLogBeat to continue sending logs to Logstash. Prior to last week, I was running a virtual WEC (Windows Event Collector) on Windows Server 2016 with WinLogBeat 6.8 forwarding to a Logstash 6.7.2 instance. I ran that configuration for nearly 2 years with minimal issues. Last week, in an effort to upgrade to the latest/greatest and try to resolve those issues, I created a physical WEC on Windows Server 2019. Then I began to experience a new issue (which is the reason for my post). Whenever I start the WinLogBeat service I seem to get a flood of events into Logstash, but then the stream of events slows way down. I am still receiving the events via WEF (Windows Event Forwarding) on the WEC and can see/verify that in Event Viewer, but WinLogBeat only forwards a very small fraction of them. At first, I was still running WinLogBeat 6.8 on the Windows Server 2019 WEC, but after this issue arose I tried 6.8.13 and received the same result. Then I discovered that per the Compatibility Matrix, WinLogBeat <7.4.x is not supported on Windows Server 2019 and WinLogBeat > 6.8.x is not supported on Logstash 6.7.2 (which is what I was still on at the time). So I bit the bullet and formatted our ELK server (Security Onion) and moved it from 16.x to 2.3.21 which is now Logstash 7.9.3. I also upgraded WinLogBeat on my WEC to 7.11.1. So my configuration as it stands now is: a physical server running Windows Server 2019 and WinLogBeat 7.11.1 forwarding to Logstash 7.9.3 on a physical Security Onion 2.3.21(CentOS 7) server. Both servers are dedicated to these applications and have plenty of resources (each has 196GB of RAM, although I am using a portion of it on the WEC for the event logs to try to increase throughput, and each should have plenty of disk space). The servers are being routed internally, but are not going through any sort of firewall other than what is on the servers themselves. So far, I haven't found anything in the logs on the WEC or on the ELK server. Has anyone experienced this before or can anyone point me in a direction to look?

Thanks,

Branden

UPDATE: Whatever is going on seems to be with WinLogBeat or Windows Server 2019, not Logstash or anything in between. I changed the output of WinLogBeat this morning to a local file and got the same result, it fills 1 100MB file pretty quickly (within seconds) and then takes another several hours to fill another even though events are still rolling in via WEF to the event log the whole time.

Can you post a copy of your winlogbeat.yml with the username/password and server info removed?

Tried to upload the actual yml as an attachment, but apparently it is not a supported type. Not sure how else to do it, so here you go:

###################### Winlogbeat Configuration Example ########################

# This file is an example configuration file highlighting only the most common
# options. The winlogbeat.reference.yml file from the same directory contains
# all the supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/winlogbeat/index.html

# ======================== Winlogbeat specific options =========================

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig

#winlogbeat.event_logs:
#  - name: Application
#    ignore_older: 72h

#  - name: System

#  - name: Security
#    processors:
#      - script:
#          lang: javascript
#          id: security
#          file: ${path.home}/module/security/config/winlogbeat-security.js

#  - name: Microsoft-Windows-Sysmon/Operational
#    processors:
#      - script:
#          lang: javascript
#          id: sysmon
#          file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js

#  - name: Windows PowerShell
#    event_id: 400, 403, 600, 800
#   processors:
#      - script:
#          lang: javascript
#          id: powershell
#          file: ${path.home}/module/powershell/config/winlogbeat-powershell.js

#  - name: Microsoft-Windows-PowerShell/Operational
#    event_id: 4103, 4104, 4105, 4106
#    processors:
#      - script:
#          lang: javascript
#          id: powershell
#          file: ${path.home}/module/powershell/config/winlogbeat-powershell.js

#  - name: ForwardedEvents
#    tags: [forwarded]
#    processors:
#      - script:
#          when.equals.winlog.channel: Security
#          lang: javascript
#          id: security
#          file: #${path.home}/module/security/config/winlogbeat-security.js
#      - script:
#          when.equals.winlog.channel: Microsoft-Windows-Sysmon/Operational
#          lang: javascript
#          id: sysmon
#          file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
#      - script:
#          when.equals.winlog.channel: Windows PowerShell
#          lang: javascript
#          id: powershell
#          file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
#      - script:
#          when.equals.winlog.channel: Microsoft-Windows-PowerShell/Operational
#          lang: javascript
#          id: powershell
#          file: ${path.home}/module/powershell/config/winlogbeat-powershell.js

winlogbeat.event_logs:
#  - name: Application
#    ignore_older: 72h
#  - name: Security
#  - name: System
  - name: WEC-Process-Execution
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC6-Sysmon
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h    
  - name: WEC-Authentication
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC16-Test
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC-Code-Integrity
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC-EMET
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC-Powershell
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC-Services
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC-WMI
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC2-Application-Crashes
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC2-Applocker
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC2-Group-Policy-Errors
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC2-Object-Manipulation
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC2-Registry
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC2-Task-Scheduler
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC2-Windows-Defender
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC3-Account-Management
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC3-Drivers
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC3-External-Devices
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC3-Firewall
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC3-Print
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC3-Smart-Card
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC3-Windows-Diagnostics
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC4-Bits-Client
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC4-DNS
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC4-Hotpatching-Errors
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC4-Shares
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC4-System-Time-Change
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC4-Windows-Updates
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC4-Wireless
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC5-Autoruns
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC5-Certificate-Authority
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC5-Crypto-API
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC5-Log-Deletion-Security
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC5-Log-Deletion-System
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC5-MSI-Packages
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC5-Operating-System
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC6-ADFS
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC6-Device-Guard
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC6-Duo-Security
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC6-Exploit-Guard
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC6-Microsoft-Office
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC6-Software-Restriction-Policies
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC7-Active-Directory
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC7-Privilege-Use
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
  - name: WEC7-Terminal-Services
    batch_read_size: 256
    forwarded: true
    ignore_older: 72h
    
# ====================== Elasticsearch template settings =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false


# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"
  host: "REDACTED.local:5601"
  
  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Winlogbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
#  hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]
  hosts: ["REDACTED.local:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

  # Number of workers per Logstash host.
  worker: 2

  # Number of batches to be sent asynchronously to Logstash while processing
  # new batches.
  pipelining: 4

  # The maximum number of events to bulk in a single Logstash request. The
  # default is 2048.
  bulk_max_size: 4096

# -------------------------------- File Output ---------------------------------
#output.file:
#  enabled: true
#  path: D:\tmp\
#  filename: winlogbeat
#  rotate_every_kb: 100000
#  number_of_files: 50

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
#  - add_cloud_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]
logging.to_files: true
logging.level: debug
logging.files: 
  path: C:\Program Files\WinLogBeat\Logs

  name: winlogbeat

  rotateeverybytes: 10485760 # = 10MB

  keepfiles: 10

Had to remove the Instrumentation and Monitoring pieces as it was over the character limit. I don't have any of it configured so it is all commented out anyway.

Branden

So no one else is running WinLogBeat on Windows Server 2019, or no one has experienced any problems with it?

Does anyone have any tips, tricks, or learning experiences to share with getting WinLogBeat to run on Windows Server 2019?

Branden

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.