Logstash stopped accepting events from winlogbeat since the new year

Posting this here as I think this is a logstash problem although I'm not really sure.
ELK version 6.7.2
Winlogbeat version is a mix of 6.7.2 and 6.3.2

At midnight on January 1st, all winlogbeat events stopped going in to logstash and nothing has changed in our environment. I went from 26 million logs on December 31 to 0 on January 1. I have had a few events here and there sneak in but nothing consistent. Maybe a thousand in 1 day.

Logstash logs show a 403 forbidden/8.index write error, however it looks like those started around the 5th of December so I don't think that error is related to my issue. Elasticsearch shows nothing.

I have tried sending the events directly to elasticsearch and that works so I don't think it is a client issue.

Logs from the client:

C:\Program Files\Winlogbeat>winlogbeat.exe -e -c winlogbeat.yml
2020-01-21T14:32:23.982-0500 INFO instance/beat.go:611 Home path: [C:\Program Files\Winlogbeat] Config path: [C:\Program Files\Winlogbeat] Data path: [C:\Program Files\Winlogbeat\data] Logs path: [C:\Program Files\Winlogbeat\logs]
2020-01-21T14:32:23.984-0500 INFO instance/beat.go:618 Beat UUID: b36cbe30-cf3e-4de7-aed9-16c3f4cdc7f3
2020-01-21T14:32:23.987-0500 INFO [beat] instance/beat.go:931 Beat info {"system_info": {"beat": {"path": {"config": "C:\Program Files\Winlogbeat", "data": "C:\Program Files\Winlogbeat\data", "home": "C:\Program Files\Winlogbeat", "logs": "C:\Program Files\Winlogbeat\logs"}, "type": "winlogbeat", "uuid": "b36cbe30-cf3e-4de7-aed9-16c3f4cdc7f3"}}}
2020-01-21T14:32:23.987-0500 INFO [beat] instance/beat.go:940 Build info {"system_info": {"build": {"commit": "a8ab26dd1f818d27c17c3049f643652c6a789d88", "libbeat": "6.7.2", "time": "2019-04-29T08:23:56.000Z", "version": "6.7.2"}}}
2020-01-21T14:32:23.988-0500 INFO [beat] instance/beat.go:943 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":2,"version":"go1.10.8"}}}
2020-01-21T14:32:23.996-0500 INFO [beat] instance/beat.go:947 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-10-30T12:02:09.92-04:00","name":"BAPM31001","ip":["172.32.7.60/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17134.1 (WinBuild.160101.0800)","mac":["00:50:56:a6:89:dd"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"17134.1"},"timezone":"EST","timezone_offset_sec":-18000,"id":"e1c2f043-6a3f-4945-ab0b-53307a97606b"}}}
2020-01-21T14:32:24.004-0500 INFO [beat] instance/beat.go:976 Process info {"system_info": {"process": {"cwd": "C:\Program Files\Winlogbeat", "exe": "C:\Program Files\Winlogbeat\winlogbeat.exe", "name": "winlogbeat.exe", "pid": 42312, "ppid": 42160, "start_time": "2020-01-21T14:32:23.883-0500"}}}
2020-01-21T14:32:24.006-0500 INFO instance/beat.go:280 Setup Beat: winlogbeat; Version: 6.7.2
2020-01-21T14:32:24.007-0500 INFO [publisher] pipeline/module.go:110 Beat name: BAPM31001
2020-01-21T14:32:24.011-0500 INFO beater/winlogbeat.go:68 State will be read from and persisted to C:\Program Files\Winlogbeat\data.winlogbeat.yml
2020-01-21T14:32:24.013-0500 INFO elasticsearch/client.go:164 Elasticsearch url: http://elasticsearch_address_here:9200
2020-01-21T14:32:24.013-0500 INFO instance/beat.go:402 winlogbeat start running.
2020-01-21T14:32:24.034-0500 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2020-01-21T14:32:24.100-0500 INFO [monitoring] elasticsearch/elasticsearch.go:247 Successfully connected to X-Pack Monitoring endpoint.
2020-01-21T14:32:24.115-0500 INFO [monitoring] elasticsearch/elasticsearch.go:261 Start monitoring stats metrics snapshot loop with period 10s.
2020-01-21T14:32:24.138-0500 INFO [monitoring] elasticsearch/elasticsearch.go:261 Start monitoring state metrics snapshot loop with period 1m0s.
2020-01-21T14:32:25.378-0500 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://logstash_address_here:5045))
2020-01-21T14:32:25.460-0500 INFO pipeline/output.go:105 Connection to backoff(async(tcp://logstash_address_here:5045)) established
2020-01-21T14:32:34.285-0500 INFO pipeline/output.go:95 Connecting to backoff(publish(elasticsearch(http://logstash_address_here:9200)))
2020-01-21T14:32:34.319-0500 INFO pipeline/output.go:105 Connection to backoff(publish(elasticsearch(http://logstash_address_here:9200))) established
2020-01-21T14:32:54.042-0500 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":375,"time":{"ms":375}},"total":{"ticks":2375,"time":{"ms":2375},"value":2375},"user":{"ticks":2000,"time":{"ms":2000}}},"handles":{"open":370},"info":{"ephemeral_id":"d7e61c20-a170-4fcd-bd0e-d60f89d068f6","uptime":{"ms":30135}},"memstats":{"gc_next":24508304,"memory_alloc":12812480,"memory_total":89719496,"rss":42831872}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":2542,"batches":3,"total":2542},"read":{"bytes":3074},"type":"logstash","write":{"bytes":317548}},"pipeline":{"clients":3,"events":{"active":2543,"published":2543,"retry":1404,"total":2543}}},"msg_file_cache":{"ApplicationHits":191,"ApplicationMisses":10,"ApplicationSize":10,"SecurityHits":1838,"SecurityMisses":1,"SecuritySize":1,"SystemHits":497,"SystemMisses":7,"SystemSize":7},"system":{"cpu":{"cores":2}}}}}

Debug log from the client adds the following:

2020-01-21T14:34:37.996-0500 DEBUG [publish] pipeline/client.go:201 Pipeline client receives callback 'onDroppedOnPublish' for event: %+v{2020-01-19 12:41:55.5771433 +0000 UTC null {"activity_id":"{2AAD78B0-8F3B-0000-3179-AD2A3B8FD501}","computer_name":"computer.bapm.com","event_data":{"AuditPolicyChanges":"%%8451","CategoryId":"%%8279","SubcategoryGuid":"{0CCE923E-69AE-11D9-BED3-505054503030}","SubcategoryId":"%%14083","SubjectDomainName":"BAPM.COM","SubjectLogonId":"0x3e7","SubjectUserName":"Computer$","SubjectUserSid":"S-1-5-18"},"event_id":4719,"keywords":["Audit Success"],"level":"Information","log_name":"Security","message":"System audit policy was changed.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tcomputer$\n\tAccount Domain:\t\tBAPM.COM\n\tLogon ID:\t\t0x3E7\n\nAudit Policy Change:\n\tCategory:\t\tDS Access\n\tSubcategory:\t\tDetailed Directory Service Replication\n\tSubcategory GUID:\t{0CCE923E-69AE-11D9-BED3-505054503030}\n\tChanges:\t\tFailure added","opcode":"Info","process_id":684,"provider_guid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","record_number":"309277","source_name":"Microsoft-Windows-Security-Auditing","task":"Audit Policy Change","thread_id":38148,"type":"wineventlog"} {Security 309277 2020-01-19 12:41:55.5771433 +0000 UTC

}}

Config file for winlogbeat:
>
> #======================= Winlogbeat specific options ==========================
>
> # event_logs specifies a list of event logs to monitor as well as any
> # accompanying options. The YAML data type of event_logs is a list of
> # dictionaries.
> #
> # The supported keys are name (required), tags, fields, fields_under_root,
> # forwarded, ignore_older, level, event_id, provider, and include_xml. Please
> # visit the documentation for the complete details of each option.
> # https://go.es.io/WinlogbeatConfig
> winlogbeat.event_logs:
> - name: Application
> ignore_older: 72h
> - name: Security
> ignore_older: 72h
> - name: System
> ignore_older: 72h
>
> #==================== Elasticsearch template setting ==========================
>
> #setup.template.settings:
> # index.number_of_shards: 3
> #index.codec: best_compression
> #_source.enabled: false
>
> #================================ General =====================================
>
> # The name of the shipper that publishes the network data. It can be used to group
> # all the transactions sent by a single shipper in the web interface.
> #name:
>
> # The tags of the shipper are included in their own field with each
> # transaction published.
> #tags: ["service-X", "web-tier"]
>
> # Optional fields that you can specify to add additional information to the
> # output.
> #fields:
> # env: staging
>
>
> #============================== Dashboards =====================================
> # These settings control loading the sample dashboards to the Kibana index. Loading
> # the dashboards is disabled by default and can be enabled either by setting the
> # options here, or by using the -setup CLI flag or the setup command.
> #setup.dashboards.enabled: false
>
> # The URL from where to download the dashboards archive. By default this URL
> # has a value which is computed based on the Beat name and version. For released
> # versions, this URL points to the dashboard archive on the artifacts.elastic.co
> # website.
> #setup.dashboards.url:
>
> #============================== Kibana =====================================
>
> # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
> # This requires a Kibana endpoint configuration.
> setup.kibana:
>
> # Kibana Host
> # Scheme and port can be left out and will be set to the default (http and 5601)
> # In case you specify and additional path, the scheme is required: http://localhost:5601/path
> # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
> #host: "localhost:5601"
>
> #============================= Elastic Cloud ==================================
>
> # These settings simplify using winlogbeat with the Elastic Cloud (https://cloud.elastic.co/).
>
> # The cloud.id setting overwrites the output.elasticsearch.hosts and
> # setup.kibana.host options.
> # You can find the cloud.id in the Elastic Cloud web UI.
> #cloud.id:
>
> # The cloud.auth setting overwrites the output.elasticsearch.username and
> # output.elasticsearch.password settings. The format is <user>:<pass>.
> #cloud.auth:
>
> #================================ Outputs =====================================
>
> # Configure what output to use when sending the data collected by the beat.
>
> #-------------------------- Elasticsearch output ------------------------------
> #output.elasticsearch:
> # Array of hosts to connect to.
> # hosts: ["localhost:9200"]
>
> # Optional protocol and basic auth credentials.
> #protocol: "https"
> #username: "elastic"
> #password: "changeme"
>
> #----------------------------- Logstash output --------------------------------
> output.logstash:
> # The Logstash hosts
> # 5044 is unsecured and Servers
> # 5045 is secured and for workstations.
> hosts: ["logstash_address:5045"]
>
> # Enable SSL support. SSL is automatically enabled, if any SSL setting is set.
> ssl.enabled: true
>
> # Configure SSL verification mode. If none is configured, all server hosts
> # and certificates will be accepted. In this mode, SSL based connections are
> # susceptible to man-in-the-middle attacks. Use only for testing. Default is
> # full.
> ssl.verification_mode: full
>
> # Optional SSL. By default is off.
> # List of root certificates for HTTPS server verifications
> #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
>
> # Certificate for SSL client authentication
> #ssl.certificate: "/etc/pki/client/cert.pem"
>
> # Client Certificate Key
> #ssl.key: "/etc/pki/client/cert.key"
>
> #================================ Logging =====================================
>
> # Sets log level. The default log level is info.
> # Available log levels are: critical, error, warning, info, debug
> #logging.level: debug
>
> # At debug level, you can selectively enable logging only for some components.
> # To enable all selectors use [""]. Examples of other selectors are "beat",
> # "publish", "service".
> #logging.selectors: ["
"]
> #============================== Xpack Monitoring =====================================
> xpack.monitoring:
> enabled: true
> elasticsearch:
> hosts: ["http://elasticsearch_address:9200"]
> username: username
> password: password

Also, here is the pipeline configuration for winlogbeat on logstash.

input {
    beats {
	port => 5044
	type => "wineventlog"
    }    
    beats {
	port => 5045
	type => "wineventlog"
    ssl => true
#    ssl_certificate_authorities => ["/etc/logstash/Cert.cer", "/etc/logstash/Cert.cer"]
    ssl_certificate => "/etc/certs/cert.pem"
    ssl_key => "/etc/certs/cert.pem"
#    ssl_verify_mode => "force_peer"
    }
} # End of Input

filter {
	grok {
	    match => [ 'TimeCreated', "Date\(%{NUMBER:timestamp}\)" ]
	}
	date {
	    match => [ 'timestamp', 'UNIX_MS' ]
	}
} #End of Filter

output{
    elasticsearch { 
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        hosts => ["logstash_address"]
        user =>  "user"
        password => "password"
    }      
} #End of Output

Here is a great visual of what is happening. These are my winlogbeat indices. You can see the doc count fall off a cliff after the first of the year. New indices are not even being created so it is not a read only issue.

health status index                              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   shrink-winlogbeat-6.7.2-2019.11.27 uzSyEzAZTFqeaAEdrF9fZQ   1   0          3            0     36.1kb         36.1kb
green  open   shrink-winlogbeat-6.7.2-2019.12.23 GWMRwTmWSkqrAXGBGyK7jg   1   0   35293503            0     15.5gb         15.5gb
green  open   shrink-winlogbeat-6.7.2-2019.12.24 3KiTDImaQiOgh4ZeNDpHEA   1   0   34023459            0     14.7gb         14.7gb
green  open   shrink-winlogbeat-6.7.2-2019.12.25 jm6FOJxyQoWgi8OOmLd1Cw   1   0   34008322            0     14.7gb         14.7gb
green  open   shrink-winlogbeat-6.7.2-2019.12.26 cg2_X_SPSfm5N8gElKjg4Q   1   0   34392820            0     14.9gb         14.9gb
green  open   shrink-winlogbeat-6.7.2-2019.12.27 4bUvzjd-SHWd20qr-T0Rag   1   0   34430700            0       15gb           15gb
green  open   shrink-winlogbeat-6.7.2-2019.12.28 _Xb40jY8QGaEiRIM66IEaA   1   0   40165781            0     17.8gb         17.8gb
green  open   shrink-winlogbeat-6.7.2-2019.12.29 YDpWmq0uSruY-DiXg6Tncg   1   0   34179318            0     14.8gb         14.8gb
green  open   shrink-winlogbeat-6.7.2-2019.12.30 xD5l4Df-TDGu4lhd2xhNHg   1   0   34196816            0     14.9gb         14.9gb
green  open   shrink-winlogbeat-6.7.2-2019.12.31 vaEyngFeRQeCzecwXWTZIQ   1   0   31405106            0     13.5gb         13.5gb
green  open   shrink-winlogbeat-6.7.2-2020.01.02 uRinwYKDTqyd5pqqp8NSUg   1   0       4141            0      2.9mb          2.9mb
green  open   shrink-winlogbeat-6.7.2-2020.01.03 2p-XZ7PZTWyDbUkIU4pNSA   1   0        308            0    374.7kb        374.7kb
green  open   shrink-winlogbeat-6.7.2-2020.01.04 J6vVy9BlSLOvPErkZUVwpw   1   0        476            0    655.3kb        655.3kb
green  open   shrink-winlogbeat-6.7.2-2020.01.05 yosgHQnMSeqzOIOq1HCb-A   1   0        695            0    898.9kb        898.9kb
green  open   shrink-winlogbeat-6.7.2-2020.01.07 gFFH5ZHFRV2GpzWXJT85nQ   1   0         52            0    156.8kb        156.8kb
yellow open   winlogbeat-6.3.2-2019.12.23        vwRq3V6tSwWMufJ0Mkom0g   5   1    3997515            0      3.4gb          3.4gb
yellow open   winlogbeat-6.3.2-2019.12.24        GhbSRNlpSFCZa7QpBCdFqA   5   1    3161797            0      2.6gb          2.6gb
yellow open   winlogbeat-6.3.2-2019.12.25        DjUSiIeCTlmGSA492jhCqg   5   1    2927893            0      2.4gb          2.4gb
yellow open   winlogbeat-6.3.2-2019.12.26        SaNWnXjLTouzVWNpisuVXA   5   1    3162800            0      2.6gb          2.6gb
yellow open   winlogbeat-6.3.2-2019.12.27        zzDAAzz3TOOMSU-9w2E9lA   5   1    3290534            0      2.8gb          2.8gb
yellow open   winlogbeat-6.3.2-2019.12.28        K7zlp4qnSGKxwlPbFTzXhw   5   1    2957441            0      2.5gb          2.5gb
yellow open   winlogbeat-6.3.2-2019.12.29        mCj92_3zQ1a6-2RPm28wmA   5   1    2611337            0      2.1gb          2.1gb
yellow open   winlogbeat-6.3.2-2019.12.30        SfeYmT1lQ9GsabKDuV417g   5   1    2563662            0      2.1gb          2.1gb
yellow open   winlogbeat-6.3.2-2019.12.31        x9W81kIVTK6E5qSwv9EWDg   5   1    2015530            0      1.6gb          1.6gb
yellow open   winlogbeat-6.3.2-2020.01.01        AICHbm1DQfiV9mFUQrMDNA   5   1       3969            0      3.8mb          3.8mb
yellow open   winlogbeat-6.3.2-2020.01.02        X6sFQF6BQnutpBtJhiqbIw   5   1        315            0      1.4mb          1.4mb
yellow open   winlogbeat-6.3.2-2020.01.03        -a26yxLSR0qVopTGVbyJGQ   5   1        172            0    642.7kb        642.7kb
yellow open   winlogbeat-6.3.2-2020.01.04        nir5lgB7Tt27akcDg9Xzpg   5   1        136            0    613.5kb        613.5kb
yellow open   winlogbeat-6.3.2-2020.01.05        ChlulbzATr2SVOqiLmFaWw   5   1        230            0      429kb          429kb
yellow open   winlogbeat-6.3.2-2020.01.06        mHQH75kWQXu0tpAZmRoMaQ   5   1       4485            0      5.9mb          5.9mb
yellow open   winlogbeat-6.3.2-2020.01.07        XRxnNeUzSWqFIcsYzt06FQ   5   1       4058            0      5.3mb          5.3mb
yellow open   winlogbeat-6.3.2-2020.01.08        _RgNqtNxTdiERYfabJIifw   5   1       2203            0      3.2mb          3.2mb
yellow open   winlogbeat-6.3.2-2020.01.09        06dP4jrSTKOv8VRtVhl1QA   5   1       1682            0      2.7mb          2.7mb
yellow open   winlogbeat-6.3.2-2020.01.10        SQImvDprQnyQSlRHr1bCRg   5   1       1177            0      2.7mb          2.7mb
yellow open   winlogbeat-6.3.2-2020.01.11        xCMBIz5SSFKxH1MxSYEVMw   5   1        672            0      1.3mb          1.3mb
green  open   winlogbeat-6.7.2-2019.10.24        g5w7wUtWSAmfgrobpWQvCg   5   0   34679405            0     15.8gb         15.8gb
green  open   winlogbeat-6.7.2-2020.01.14        zRQ0ZqAUR0yzN0MgjbKUKQ   5   0       1181            0      1.2mb          1.2mb
green  open   winlogbeat-6.7.2-2020.01.15        ikMNhJRHQL2uISs_IXgW-A   5   0       2601            0      2.2mb          2.2mb
green  open   winlogbeat-6.7.2-2020.01.16        fqr-G2FiRwKZtDE-88m69w   5   0       4749            0      4.2mb          4.2mb
green  open   winlogbeat-6.7.2-2020.01.17        _OkA3FNVRcOkNgMQdTiBCw   5   0       2695            0      2.4mb          2.4mb

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.