Having trouble getting WinLogBeat to continue sending logs to Logstash. Prior to last week, I was running a virtual WEC (Windows Event Collector) on Windows Server 2016 with WinLogBeat 6.8 forwarding to a Logstash 6.7.2 instance. I ran that configuration for nearly 2 years with minimal issues. Last week, in an effort to upgrade to the latest/greatest and try to resolve those issues, I created a physical WEC on Windows Server 2019. Then I began to experience a new issue (which is the reason for my post). Whenever I start the WinLogBeat service I seem to get a flood of events into Logstash, but then the stream of events slows way down. I am still receiving the events via WEF (Windows Event Forwarding) on the WEC and can see/verify that in Event Viewer, but WinLogBeat only forwards a very small fraction of them. At first, I was still running WinLogBeat 6.8 on the Windows Server 2019 WEC, but after this issue arose I tried 6.8.13 and received the same result. Then I discovered that per the Compatibility Matrix, WinLogBeat <7.4.x is not supported on Windows Server 2019 and WinLogBeat > 6.8.x is not supported on Logstash 6.7.2 (which is what I was still on at the time). So I bit the bullet and formatted our ELK server (Security Onion) and moved it from 16.x to 2.3.21 which is now Logstash 7.9.3. I also upgraded WinLogBeat on my WEC to 7.11.1. So my configuration as it stands now is: a physical server running Windows Server 2019 and WinLogBeat 7.11.1 forwarding to Logstash 7.9.3 on a physical Security Onion 2.3.21(CentOS 7) server. Both servers are dedicated to these applications and have plenty of resources (each has 196GB of RAM, although I am using a portion of it on the WEC for the event logs to try to increase throughput, and each should have plenty of disk space). The servers are being routed internally, but are not going through any sort of firewall other than what is on the servers themselves. So far, I haven't found anything in the logs on the WEC or on the ELK server. Has anyone experienced this before or can anyone point me in a direction to look?
Thanks,
Branden
UPDATE: Whatever is going on seems to be with WinLogBeat or Windows Server 2019, not Logstash or anything in between. I changed the output of WinLogBeat this morning to a local file and got the same result, it fills 1 100MB file pretty quickly (within seconds) and then takes another several hours to fill another even though events are still rolling in via WEF to the event log the whole time.
Tried to upload the actual yml as an attachment, but apparently it is not a supported type. Not sure how else to do it, so here you go:
###################### Winlogbeat Configuration Example ########################
# This file is an example configuration file highlighting only the most common
# options. The winlogbeat.reference.yml file from the same directory contains
# all the supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/winlogbeat/index.html
# ======================== Winlogbeat specific options =========================
# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig
#winlogbeat.event_logs:
# - name: Application
# ignore_older: 72h
# - name: System
# - name: Security
# processors:
# - script:
# lang: javascript
# id: security
# file: ${path.home}/module/security/config/winlogbeat-security.js
# - name: Microsoft-Windows-Sysmon/Operational
# processors:
# - script:
# lang: javascript
# id: sysmon
# file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
# - name: Windows PowerShell
# event_id: 400, 403, 600, 800
# processors:
# - script:
# lang: javascript
# id: powershell
# file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
# - name: Microsoft-Windows-PowerShell/Operational
# event_id: 4103, 4104, 4105, 4106
# processors:
# - script:
# lang: javascript
# id: powershell
# file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
# - name: ForwardedEvents
# tags: [forwarded]
# processors:
# - script:
# when.equals.winlog.channel: Security
# lang: javascript
# id: security
# file: #${path.home}/module/security/config/winlogbeat-security.js
# - script:
# when.equals.winlog.channel: Microsoft-Windows-Sysmon/Operational
# lang: javascript
# id: sysmon
# file: ${path.home}/module/sysmon/config/winlogbeat-sysmon.js
# - script:
# when.equals.winlog.channel: Windows PowerShell
# lang: javascript
# id: powershell
# file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
# - script:
# when.equals.winlog.channel: Microsoft-Windows-PowerShell/Operational
# lang: javascript
# id: powershell
# file: ${path.home}/module/powershell/config/winlogbeat-powershell.js
winlogbeat.event_logs:
# - name: Application
# ignore_older: 72h
# - name: Security
# - name: System
- name: WEC-Process-Execution
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC6-Sysmon
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC-Authentication
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC16-Test
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC-Code-Integrity
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC-EMET
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC-Powershell
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC-Services
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC-WMI
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC2-Application-Crashes
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC2-Applocker
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC2-Group-Policy-Errors
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC2-Object-Manipulation
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC2-Registry
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC2-Task-Scheduler
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC2-Windows-Defender
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC3-Account-Management
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC3-Drivers
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC3-External-Devices
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC3-Firewall
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC3-Print
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC3-Smart-Card
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC3-Windows-Diagnostics
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC4-Bits-Client
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC4-DNS
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC4-Hotpatching-Errors
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC4-Shares
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC4-System-Time-Change
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC4-Windows-Updates
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC4-Wireless
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC5-Autoruns
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC5-Certificate-Authority
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC5-Crypto-API
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC5-Log-Deletion-Security
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC5-Log-Deletion-System
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC5-MSI-Packages
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC5-Operating-System
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC6-ADFS
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC6-Device-Guard
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC6-Duo-Security
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC6-Exploit-Guard
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC6-Microsoft-Office
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC6-Software-Restriction-Policies
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC7-Active-Directory
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC7-Privilege-Use
batch_read_size: 256
forwarded: true
ignore_older: 72h
- name: WEC7-Terminal-Services
batch_read_size: 256
forwarded: true
ignore_older: 72h
# ====================== Elasticsearch template settings =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
#setup.dashboards.enabled: false
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
#host: "localhost:5601"
host: "REDACTED.local:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# =============================== Elastic Cloud ================================
# These settings simplify using Winlogbeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
# Array of hosts to connect to.
# hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
#hosts: ["localhost:5044"]
hosts: ["REDACTED.local:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
# Number of workers per Logstash host.
worker: 2
# Number of batches to be sent asynchronously to Logstash while processing
# new batches.
pipelining: 4
# The maximum number of events to bulk in a single Logstash request. The
# default is 2048.
bulk_max_size: 4096
# -------------------------------- File Output ---------------------------------
#output.file:
# enabled: true
# path: D:\tmp\
# filename: winlogbeat
# rotate_every_kb: 100000
# number_of_files: 50
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
# - add_cloud_metadata: ~
# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publisher", "service".
#logging.selectors: ["*"]
logging.to_files: true
logging.level: debug
logging.files:
path: C:\Program Files\WinLogBeat\Logs
name: winlogbeat
rotateeverybytes: 10485760 # = 10MB
keepfiles: 10
Had to remove the Instrumentation and Monitoring pieces as it was over the character limit. I don't have any of it configured so it is all commented out anyway.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.