Winlogbeat will no longer start

[popa]

I made a configuration change to winlogbeat.yml then issued a Restart-Service winlogbeat. I've done this several times in the past when updating my configuration file without any issues.

However, this time the winlogbeat service briefly starts then stops. I reverted back to a working configuration, testing it using winlogbeat.exe test config -c winlogbeat.yml -e and the status is Config OK, so I know the syntax is okay.

Here's the only error log that I'm able to come across in C:\ProgramData\winlogbeat\logs:

2018-10-03T14:37:35.081+0900	INFO	instance/beat.go:544	Home path: [C:\Program Files\winlogbeat-6.4.0] Config path: [C:\Program Files\winlogbeat-6.4.0] Data path: [C:\ProgramData\winlogbeat] Logs path: [C:\ProgramData\winlogbeat\logs]
2018-10-03T14:37:35.261+0900	INFO	instance/beat.go:551	Beat UUID: <UUID HERE>
2018-10-03T14:37:35.261+0900	INFO	[beat]	instance/beat.go:768	Beat info	{"system_info": {"beat": {"path": {"config": "C:\\Program Files\\winlogbeat-6.4.0", "data": "C:\\ProgramData\\winlogbeat", "home": "C:\\Program Files\\winlogbeat-6.4.0", "logs": "C:\\ProgramData\\winlogbeat\\logs"}, "type": "winlogbeat", "uuid": "fc0c37c0-038f-42a7-ae87-984fa11d6b92"}}}
2018-10-03T14:37:35.261+0900	INFO	[beat]	instance/beat.go:777	Build info	{"system_info": {"build": {"commit": "34b4e2cc75fbbee5e7149f3916de72fb8892d070", "libbeat": "6.4.0", "time": "2018-08-17T22:29:02.000Z", "version": "6.4.0"}}}
2018-10-03T14:37:35.262+0900	INFO	[beat]	instance/beat.go:780	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":2,"version":"go1.10.3"}}}
2018-10-03T14:37:35.272+0900	INFO	[beat]	instance/beat.go:784	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-08-29T11:36:02.01+09:00","hostname":"WINLOGBEATHOSTNAME","ips":["172.X.X.X/24","::1/128","127.0.0.1/8"],"kernel_version":"6.3.9600.19101 (winblue_ltsb_escrow.180718-1800)","mac_addresses":["00:XX:XX:XX:XX:XX"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Standard","version":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19102"},"timezone":"JST","timezone_offset_sec":32400,"id":"XXXXXXXXXXXXXXXXX"}}}
2018-10-03T14:37:35.272+0900	INFO	instance/beat.go:273	Setup Beat: winlogbeat; Version: 6.4.0
2018-10-03T14:37:35.275+0900	INFO	pipeline/module.go:98	Beat name: WINLOGBEATHOSTNAME
2018-10-03T14:37:35.275+0900	INFO	beater/winlogbeat.go:68	State will be read from and persisted to C:\ProgramData\winlogbeat\.winlogbeat.yml
2018-10-03T14:37:35.276+0900	INFO	instance/beat.go:367	winlogbeat start running.
2018-10-03T14:37:35.279+0900	INFO	[monitoring]	log/log.go:114	Starting metrics logging every 30s
2018-10-03T14:37:37.398+0900	INFO	[monitoring]	log/log.go:149	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":{"ms":203}},"total":{"ticks":171,"time":{"ms":249},"value":0},"user":{"ticks":31,"time":{"ms":46}}},"info":{"ephemeral_id":"XXXXXXXXXXXXXXXXX","uptime":{"ms":3123}},"memstats":{"gc_next":4194304,"memory_alloc":2219528,"memory_total":3799136,"rss":17883136}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":2}}}}}
2018-10-03T14:37:37.398+0900	INFO	[monitoring]	log/log.go:150	Uptime: 3.123263s
2018-10-03T14:37:37.398+0900	INFO	[monitoring]	log/log.go:127	Stopping metrics logging.
2018-10-03T14:37:37.398+0900	INFO	instance/beat.go:373	winlogbeat stopped.
2018-10-03T14:37:37.398+0900	ERROR	instance/beat.go:743	Exiting: yaml: line 56: found unexpected end of stream

I'm a bit confused and running out of troubleshooting steps.

Thanks for your time.

[kvch (Noémi Ványi)]

Could you please share your full config formatted using </>?
Unfortunately, test config does not do an extensive check, so it's possible that something is still wrong in your config.

[popa]

Hi @kvch,

Sure, I will share the configuration with you. Just as a side note, I tried both a new configuration and a previous config that was working. Both of them resulted in the same issue.

Ultimately I was able to fix my issue by uninstalling the service and deleting all remnants of winlogbeat on disk.

Here is a working configuration:

###################### Winlogbeat Configuration Example ##########################

# This file is an example configuration file highlighting only the most common
# options. The winlogbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/winlogbeat/index.html
# 
# You can find a good configuration here:
# https://www.elastic.co/guide/en/beats/winlogbeat/master/winlogbeat-reference-yml.html

#======================= Winlogbeat specific options ==========================

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig

# The following items will be accomplished using the Winlogbeat service / agent:
# 1. Forward logs being forwardered to the WEC Server from WEF Clients
# 2. Forward logs generated by the WEC Server

winlogbeat.event_logs:
  - name: WEC-Authentication
    ignore_older: 24h
#  - name: WEC-Code-Integrity
#  - name: WEC-EMET  
  - name: WEC-Powershell
    ignore_older: 24h
  - name: WEC-Process-Execution
    ignore_older: 24h
  - name: WEC-Services
    ignore_older: 24h
  - name: WEC-WMI
    ignore_older: 24h
  - name: WEC2-Application-Crashes
    ignore_older: 24h
#  - name: WEC2-Applocker
  - name: WEC2-Group-Policy-Errors
    ignore_older: 24h
  - name: WEC2-Object-Manipulation
    ignore_older: 24h
  - name: WEC2-Registry
    ignore_older: 24h
  - name: WEC2-Task-Scheduler
    ignore_older: 24h
#  - name: WEC2-Windows-Defender
##  - name: WEC3-Account-Management
    ##ignore_older: 24h
##  - name: WEC3-Drivers
    ##ignore_older: 24h
##  - name: WEC3-External-Devices
    ##ignore_older: 24h
##  - name: WEC3-Firewall
    ##ignore_older: 24h
#  - name: WEC3-Print
#  - name: WEC3-Smart-Card
#  - name: WEC3-Windows-Diagnostics
#  - name: WEC4-Bits-Client
##  - name: WEC4-DNS
    ##ignore_older: 24h
##  - name: WEC4-Hotpatching-Errors
    ##ignore_older: 24h
##  - name: WEC4-Shares
    ##ignore_older: 24h
##  - name: WEC4-System-Time-Change
    ##ignore_older: 24h
##  - name: WEC4-Windows-Updates
    ##ignore_older: 24h
##  - name: WEC4-Wireless
    ##ignore_older: 24h
#  - name: WEC5-Autoruns
##  - name: WEC5-Certificate-Authority
    ##ignore_older: 24h
##  - name: WEC5-Crypto-API
    ##ignore_older: 24h
##  - name: WEC5-Log-Deletion-Security
    ##ignore_older: 24h
##  - name: WEC5-Log-Deletion-System
    ##ignore_older: 24h
##  - name: WEC5-MSI-Packages
    ##ignore_older: 24h
##  - name: WEC5-Operating-System
    ##ignore_older: 24h
#  - name: WEC6-ADFS
#  - name: WEC6-Device-Guard
#  - name: WEC6-Duo-Security
#  - name: WEC6-Exploit-Guard
##  - name: WEC6-Microsoft-Office
    ##ignore_older: 24h
##  - name: WEC6-Software-Restriction-Policies
  - name: WEC6-Sysmon
    ignore_older: 24h
##  - name: WEC7-Active-Directory
    ##ignore_older: 24h
##  - name: WEC7-Privilege-Use
    ##ignore_older: 24h
##  - name: WEC7-Terminal-Services
    ##ignore_older: 24h

#==================== Elasticsearch template setting ==========================

#================================ General =====================================

# The tags of the shipper are included in their own field with each
# transaction published.
tags: ["wec-server", "WINLOGBEATHOSTNAME"]

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["172.X.X.X:5044"]
  
  # The maximum number of events to bulk in a single Logstash request. 
  # Winlogbeat can process around 2900 eps
  # The default is 2048:
  bulk_max_size: 2048

The confusing part is that I reloaded the working configuration and winlogbeat still failed to start.

[popa]

Here is the config that broke the winlogbeat service:

###################### Winlogbeat Configuration Example ##########################

# This file is an example configuration file highlighting only the most common
# options. The winlogbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/winlogbeat/index.html
# 
# You can find a good configuration here:
# https://www.elastic.co/guide/en/beats/winlogbeat/master/winlogbeat-reference-yml.html

#======================= Winlogbeat specific options ==========================

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig

# The following items will be accomplished using the Winlogbeat service / agent:
# 1. Forward logs being forwardered to the WEC Server from WEF Clients
# 2. Forward logs generated by the WEC Server

winlogbeat.event_logs:
  - name: WEC-Authentication
    ignore_older: 72h
#  - name: WEC-Code-Integrity
#  - name: WEC-EMET  
  - name: WEC-Powershell
    ignore_older: 72h
  - name: WEC-Process-Execution
    ignore_older: 72h
  - name: WEC-Services
    ignore_older: 72h
  - name: WEC-WMI
    ignore_older: 72h
  - name: WEC2-Application-Crashes
    ignore_older: 72h
#  - name: WEC2-Applocker
  - name: WEC2-Group-Policy-Errors
    ignore_older: 72h
  - name: WEC2-Object-Manipulation
    ignore_older: 72h
  - name: WEC2-Registry
    ignore_older: 72h
  - name: WEC2-Task-Scheduler
    ignore_older: 72h
#  - name: WEC2-Windows-Defender
##  - name: WEC3-Account-Management
    ##ignore_older: 72h
##  - name: WEC3-Drivers
    ##ignore_older: 72h
##  - name: WEC3-External-Devices
    ##ignore_older: 72h
##  - name: WEC3-Firewall
    ##ignore_older: 72h
#  - name: WEC3-Print
#  - name: WEC3-Smart-Card
#  - name: WEC3-Windows-Diagnostics
#  - name: WEC4-Bits-Client
##  - name: WEC4-DNS
    ##ignore_older: 72h
##  - name: WEC4-Hotpatching-Errors
    ##ignore_older: 72h
##  - name: WEC4-Shares
    ##ignore_older: 72h
##  - name: WEC4-System-Time-Change
    ##ignore_older: 72h
##  - name: WEC4-Windows-Updates
    ##ignore_older: 72h
##  - name: WEC4-Wireless
    ##ignore_older: 72h
#  - name: WEC5-Autoruns
##  - name: WEC5-Certificate-Authority
    ##ignore_older: 72h
##  - name: WEC5-Crypto-API
    ##ignore_older: 72h
##  - name: WEC5-Log-Deletion-Security
    ##ignore_older: 72h
##  - name: WEC5-Log-Deletion-System
    ##ignore_older: 72h
##  - name: WEC5-MSI-Packages
    ##ignore_older: 72h
##  - name: WEC5-Operating-System
    ##ignore_older: 72h
#  - name: WEC6-ADFS
#  - name: WEC6-Device-Guard
#  - name: WEC6-Duo-Security
#  - name: WEC6-Exploit-Guard
##  - name: WEC6-Microsoft-Office
    ##ignore_older: 72h
##  - name: WEC6-Software-Restriction-Policies
  - name: WEC6-Sysmon
    ignore_older: 72h
##  - name: WEC7-Active-Directory
    ##ignore_older: 72h
##  - name: WEC7-Privilege-Use
    ##ignore_older: 72h
##  - name: WEC7-Terminal-Services
    ##ignore_older: 72h

#==================== Processors ==========================

# The following processor *should* drop all McAfee noise:

processors:
- drop_event.when.and:
  - equals.event_data.ProcessName: 'C:\Program Files\Common Files\McAfee\SystemCore\mcshield.exe'
  - equals.event_id: '4663'
- drop_event.when.and:
  - equals.event_data.ProcessName: 'C:\Program Files (x86)\McAfee\VirusScan Enterprise\x64\Scan64.Exe'
  - equals.event_id: '4656'

#================================ General =====================================

# The tags of the shipper are included in their own field with each
# transaction published.
tags: ["wec-server", "WINLOGBEATHOSTNAME"]

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["172.X.X.X:5044"]
  
    # Set gzip compression level.
  compression_level: 3

  # The maximum number of events to bulk in a single Logstash request. 
  # Winlogbeat can process around 2900 eps
  # The default is 2048:
  bulk_max_size: 2048

To add a little more context on the steps I took when modifying winlogbeat.yml:

Step 1: Modify winlogbeat.yml and make changes.

Step 2: Run a test to make sure the configuration file is formatted correctly: PS C:\Program Files\Winlogbeat> .\winlogbeat.exe test config -c .\winlogbeat.yml -e

Step 3: Restart the Winlogbeat service: PS C:\Program Files\Winlogbeat> Restart-Service winlogbeat

One thing that I don't do is stop the service before modifying the file. Should I stop the service first and then modify, or is my process okay?