# Logstash stopped accepting events from winlogbeat since the new year **URL:** \ **Category:** Logstash\ **Created:** [January 21, 2020, 7:41pm UTC](https://discuss.elastic.co/t/logstash-stopped-accepting-events-from-winlogbeat-since-the-new-year/215941 "2020-01-21T19:41:13Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![albbapm](https://avatars.discourse-cdn.com/v4/letter/a/90ced4/32.png) [@albbapm](https://discuss.elastic.co/u/albbapm)\ **Post date:** [January 21, 2020, 7:41pm UTC](https://discuss.elastic.co/t/logstash-stopped-accepting-events-from-winlogbeat-since-the-new-year/215941/1 "2020-01-21T19:41:13Z")
Posting this here as I think this is a logstash problem although I'm not really sure. ELK version 6.7.2 Winlogbeat version is a mix of 6.7.2 and 6.3.2 At midnight on January 1st, all winlogbeat events stopped going in to logstash and nothing has changed in our environment. I went from 26 million logs on December 31 to 0 on January 1. I have had a few events here and there sneak in but nothing consistent. Maybe a thousand in 1 day. Logstash logs show a 403 forbidden/8.index write error, however it looks like those started around the 5th of December so I don't think that error is related to my issue. Elasticsearch shows nothing. I have tried sending the events directly to elasticsearch and that works so I don't think it is a client issue. Logs from the client: > C:\Program Files\Winlogbeat\>winlogbeat.exe -e -c winlogbeat.yml > 2020-01-21T14:32:23.982-0500 INFO instance/beat.go:611 Home path: [C:\Program Files\Winlogbeat] Config path: [C:\Program Files\Winlogbeat] Data path: [C:\Program Files\Winlogbeat\data] Logs path: [C:\Program Files\Winlogbeat\logs] > 2020-01-21T14:32:23.984-0500 INFO instance/beat.go:618 Beat UUID: b36cbe30-cf3e-4de7-aed9-16c3f4cdc7f3 > 2020-01-21T14:32:23.987-0500 INFO [beat] instance/beat.go:931 Beat info {"system\_info": {"beat": {"path": {"config": "C:\Program Files\Winlogbeat", "data": "C:\Program Files\Winlogbeat\data", "home": "C:\Program Files\Winlogbeat", "logs": "C:\Program Files\Winlogbeat\logs"}, "type": "winlogbeat", "uuid": "b36cbe30-cf3e-4de7-aed9-16c3f4cdc7f3"}}} > 2020-01-21T14:32:23.987-0500 INFO [beat] instance/beat.go:940 Build info {"system\_info": {"build": {"commit": "a8ab26dd1f818d27c17c3049f643652c6a789d88", "libbeat": "6.7.2", "time": "2019-04-29T08:23:56.000Z", "version": "6.7.2"}}} > 2020-01-21T14:32:23.988-0500 INFO [beat] instance/beat.go:943 Go runtime info {"system\_info": {"go": {"os":"windows","arch":"amd64","max\_procs":2,"version":"go1.10.8"}}} > 2020-01-21T14:32:23.996-0500 INFO [beat] instance/beat.go:947 Host info {"system\_info": {"host": {"architecture":"x86\_64","boot\_time":"2019-10-30T12:02:09.92-04:00","name":"BAPM31001","ip":["172.32.7.60/24","::1/128","127.0.0.1/8"],"kernel\_version":"10.0.17134.1 (WinBuild.160101.0800)","mac":["00:50:56:a6:89:dd"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"17134.1"},"timezone":"EST","timezone\_offset\_sec":-18000,"id":"e1c2f043-6a3f-4945-ab0b-53307a97606b"}}} > 2020-01-21T14:32:24.004-0500 INFO [beat] instance/beat.go:976 Process info {"system\_info": {"process": {"cwd": "C:\Program Files\Winlogbeat", "exe": "C:\Program Files\Winlogbeat\winlogbeat.exe", "name": "winlogbeat.exe", "pid": 42312, "ppid": 42160, "start\_time": "2020-01-21T14:32:23.883-0500"}}} > 2020-01-21T14:32:24.006-0500 INFO instance/beat.go:280 Setup Beat: winlogbeat; Version: 6.7.2 > 2020-01-21T14:32:24.007-0500 INFO [publisher] pipeline/module.go:110 Beat name: BAPM31001 > 2020-01-21T14:32:24.011-0500 INFO beater/winlogbeat.go:68 State will be read from and persisted to C:\Program Files\Winlogbeat\data.winlogbeat.yml > 2020-01-21T14:32:24.013-0500 INFO elasticsearch/client.go:164 Elasticsearch url: http://elasticsearch\_address\_here:9200 > 2020-01-21T14:32:24.013-0500 INFO instance/beat.go:402 winlogbeat start running. > 2020-01-21T14:32:24.034-0500 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s > 2020-01-21T14:32:24.100-0500 INFO [monitoring] elasticsearch/elasticsearch.go:247 Successfully connected to X-Pack Monitoring endpoint. > 2020-01-21T14:32:24.115-0500 INFO [monitoring] elasticsearch/elasticsearch.go:261 Start monitoring stats metrics snapshot loop with period 10s. > 2020-01-21T14:32:24.138-0500 INFO [monitoring] elasticsearch/elasticsearch.go:261 Start monitoring state metrics snapshot loop with period 1m0s. > 2020-01-21T14:32:25.378-0500 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://logstash\_address\_here:5045)) > 2020-01-21T14:32:25.460-0500 INFO pipeline/output.go:105 Connection to backoff(async(tcp://logstash\_address\_here:5045)) established > 2020-01-21T14:32:34.285-0500 INFO pipeline/output.go:95 Connecting to backoff(publish(elasticsearch(http://logstash\_address\_here:9200))) > 2020-01-21T14:32:34.319-0500 INFO pipeline/output.go:105 Connection to backoff(publish(elasticsearch(http://logstash\_address\_here:9200))) established > 2020-01-21T14:32:54.042-0500 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":375,"time":{"ms":375}},"total":{"ticks":2375,"time":{"ms":2375},"value":2375},"user":{"ticks":2000,"time":{"ms":2000}}},"handles":{"open":370},"info":{"ephemeral\_id":"d7e61c20-a170-4fcd-bd0e-d60f89d068f6","uptime":{"ms":30135}},"memstats":{"gc\_next":24508304,"memory\_alloc":12812480,"memory\_total":89719496,"rss":42831872}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":2542,"batches":3,"total":2542},"read":{"bytes":3074},"type":"logstash","write":{"bytes":317548}},"pipeline":{"clients":3,"events":{"active":2543,"published":2543,"retry":1404,"total":2543}}},"msg\_file\_cache":{"ApplicationHits":191,"ApplicationMisses":10,"ApplicationSize":10,"SecurityHits":1838,"SecurityMisses":1,"SecuritySize":1,"SystemHits":497,"SystemMisses":7,"SystemSize":7},"system":{"cpu":{"cores":2}}}}} Debug log from the client adds the following: > 2020-01-21T14:34:37.996-0500 DEBUG [publish] pipeline/client.go:201 Pipeline client receives callback 'onDroppedOnPublish' for event: %+v{2020-01-19 12:41:55.5771433 +0000 UTC null {"activity\_id":"{2AAD78B0-8F3B-0000-3179-AD2A3B8FD501}","computer\_name":"[computer.bapm.com](http://computer.bapm.com)","event\_data":{"AuditPolicyChanges":"%%8451","CategoryId":"%%8279","SubcategoryGuid":"{0CCE923E-69AE-11D9-BED3-505054503030}","SubcategoryId":"%%14083","SubjectDomainName":"[BAPM.COM](http://BAPM.COM)","SubjectLogonId":"0x3e7","SubjectUserName":"Computer$","SubjectUserSid":"S-1-5-18"},"event\_id":4719,"keywords":["Audit Success"],"level":"Information","log\_name":"Security","message":"System audit policy was changed.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tcomputer$\n\tAccount Domain:\t\tBAPM.COM\n\tLogon ID:\t\t0x3E7\n\nAudit Policy Change:\n\tCategory:\t\tDS Access\n\tSubcategory:\t\tDetailed Directory Service Replication\n\tSubcategory GUID:\t{0CCE923E-69AE-11D9-BED3-505054503030}\n\tChanges:\t\tFailure added","opcode":"Info","process\_id":684,"provider\_guid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","record\_number":"309277","source\_name":"Microsoft-Windows-Security-Auditing","task":"Audit Policy Change","thread\_id":38148,"type":"wineventlog"} {Security 309277 2020-01-19 12:41:55.5771433 +0000 UTC > > }} Config file for winlogbeat: \> \> #======================= Winlogbeat specific options ========================== \> \> # event\_logs specifies a list of event logs to monitor as well as any \> # accompanying options. The YAML data type of event\_logs is a list of \> # dictionaries. \> # \> # The supported keys are name (required), tags, fields, fields\_under\_root, \> # forwarded, ignore\_older, level, event\_id, provider, and include\_xml. Please \> # visit the documentation for the complete details of each option. \> # [Configure Winlogbeat | Winlogbeat Reference [8.11] | Elastic](https://go.es.io/WinlogbeatConfig) \> winlogbeat.event\_logs: \> - name: Application \> ignore\_older: 72h \> - name: Security \> ignore\_older: 72h \> - name: System \> ignore\_older: 72h \> \> #==================== Elasticsearch template setting ========================== \> \> #setup.template.settings: \> # index.number\_of\_shards: 3 \> #index.codec: best\_compression \> #\_source.enabled: false \> \> #================================ General ===================================== \> \> # The name of the shipper that publishes the network data. It can be used to group \> # all the transactions sent by a single shipper in the web interface. \> #name: \> \> # The tags of the shipper are included in their own field with each \> # transaction published. \> #tags: ["service-X", "web-tier"] \> \> # Optional fields that you can specify to add additional information to the \> # output. \> #fields: \> # env: staging \> \> \> #============================== Dashboards ===================================== \> # These settings control loading the sample dashboards to the Kibana index. Loading \> # the dashboards is disabled by default and can be enabled either by setting the \> # options here, or by using the `-setup` CLI flag or the `setup` command. \> #setup.dashboards.enabled: false \> \> # The URL from where to download the dashboards archive. By default this URL \> # has a value which is computed based on the Beat name and version. For released \> # versions, this URL points to the dashboard archive on the [artifacts.elastic.co](http://artifacts.elastic.co) \> # website. \> #setup.dashboards.url: \> \> #============================== Kibana ===================================== \> \> # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API. \> # This requires a Kibana endpoint configuration. \> setup.kibana: \> \> # Kibana Host \> # Scheme and port can be left out and will be set to the default (http and 5601) \> # In case you specify and additional path, the scheme is required: [http://localhost:5601/path](http://localhost:5601/path) \> # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601 \> #host: "localhost:5601" \> \> #============================= Elastic Cloud ================================== \> \> # These settings simplify using winlogbeat with the Elastic Cloud ([https://cloud.elastic.co/](https://cloud.elastic.co/)). \> \> # The cloud.id setting overwrites the `output.elasticsearch.hosts` and \> # `setup.kibana.host` options. \> # You can find the `cloud.id` in the Elastic Cloud web UI. \> #cloud.id: \> \> # The cloud.auth setting overwrites the `output.elasticsearch.username` and \> # `output.elasticsearch.password` settings. The format is `:`. \> #cloud.auth: \> \> #================================ Outputs ===================================== \> \> # Configure what output to use when sending the data collected by the beat. \> \> #-------------------------- Elasticsearch output ------------------------------ \> #output.elasticsearch: \> # Array of hosts to connect to. \> # hosts: ["localhost:9200"] \> \> # Optional protocol and basic auth credentials. \> #protocol: "https" \> #username: "elastic" \> #password: "changeme" \> \> #----------------------------- Logstash output -------------------------------- \> output.logstash: \> # The Logstash hosts \> # 5044 is unsecured and Servers \> # 5045 is secured and for workstations. \> hosts: ["logstash\_address:5045"] \> \> # Enable SSL support. SSL is automatically enabled, if any SSL setting is set. \> ssl.enabled: true \> \> # Configure SSL verification mode. If `none` is configured, all server hosts \> # and certificates will be accepted. In this mode, SSL based connections are \> # susceptible to man-in-the-middle attacks. Use only for testing. Default is \> # `full`. \> ssl.verification\_mode: full \> \> # Optional SSL. By default is off. \> # List of root certificates for HTTPS server verifications \> #ssl.certificate\_authorities: ["/etc/pki/root/ca.pem"] \> \> # Certificate for SSL client authentication \> #ssl.certificate: "/etc/pki/client/cert.pem" \> \> # Client Certificate Key \> #ssl.key: "/etc/pki/client/cert.key" \> \> #================================ Logging ===================================== \> \> # Sets log level. The default log level is info. \> # Available log levels are: critical, error, warning, info, debug \> #logging.level: debug \> \> # At debug level, you can selectively enable logging only for some components. \> # To enable all selectors use ["_"]. Examples of other selectors are "beat", \> # "publish", "service". \> #logging.selectors: ["_"] \> #============================== Xpack Monitoring ===================================== \> xpack.monitoring: \> enabled: true \> elasticsearch: \> hosts: ["http://elasticsearch\_address:9200"] \> username: username \> password: password ---
**Author:** ![albbapm](https://avatars.discourse-cdn.com/v4/letter/a/90ced4/32.png) [@albbapm](https://discuss.elastic.co/u/albbapm)\ **Post date:** [January 21, 2020, 7:51pm UTC](https://discuss.elastic.co/t/logstash-stopped-accepting-events-from-winlogbeat-since-the-new-year/215941/2 "2020-01-21T19:51:51Z")
Also, here is the pipeline configuration for winlogbeat on logstash. ``` input { beats { port => 5044 type => "wineventlog" } beats { port => 5045 type => "wineventlog" ssl => true # ssl_certificate_authorities => ["/etc/logstash/Cert.cer", "/etc/logstash/Cert.cer"] ssl_certificate => "/etc/certs/cert.pem" ssl_key => "/etc/certs/cert.pem" # ssl_verify_mode => "force_peer" } } # End of Input filter { grok { match => ['TimeCreated', "Date\(%{NUMBER:timestamp}\)"] } date { match => ['timestamp', 'UNIX_MS'] } } #End of Filter output{ elasticsearch { index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" hosts => ["logstash_address"] user => "user" password => "password" } } #End of Output ``` ---
**Author:** ![albbapm](https://avatars.discourse-cdn.com/v4/letter/a/90ced4/32.png) [@albbapm](https://discuss.elastic.co/u/albbapm)\ **Post date:** [January 21, 2020, 8:37pm UTC](https://discuss.elastic.co/t/logstash-stopped-accepting-events-from-winlogbeat-since-the-new-year/215941/3 "2020-01-21T20:37:45Z")
Here is a great visual of what is happening. These are my winlogbeat indices. You can see the doc count fall off a cliff after the first of the year. New indices are not even being created so it is not a read only issue. ``` health status index uuid pri rep docs.count docs.deleted store.size pri.store.size green open shrink-winlogbeat-6.7.2-2019.11.27 uzSyEzAZTFqeaAEdrF9fZQ 1 0 3 0 36.1kb 36.1kb green open shrink-winlogbeat-6.7.2-2019.12.23 GWMRwTmWSkqrAXGBGyK7jg 1 0 35293503 0 15.5gb 15.5gb green open shrink-winlogbeat-6.7.2-2019.12.24 3KiTDImaQiOgh4ZeNDpHEA 1 0 34023459 0 14.7gb 14.7gb green open shrink-winlogbeat-6.7.2-2019.12.25 jm6FOJxyQoWgi8OOmLd1Cw 1 0 34008322 0 14.7gb 14.7gb green open shrink-winlogbeat-6.7.2-2019.12.26 cg2_X_SPSfm5N8gElKjg4Q 1 0 34392820 0 14.9gb 14.9gb green open shrink-winlogbeat-6.7.2-2019.12.27 4bUvzjd-SHWd20qr-T0Rag 1 0 34430700 0 15gb 15gb green open shrink-winlogbeat-6.7.2-2019.12.28 _Xb40jY8QGaEiRIM66IEaA 1 0 40165781 0 17.8gb 17.8gb green open shrink-winlogbeat-6.7.2-2019.12.29 YDpWmq0uSruY-DiXg6Tncg 1 0 34179318 0 14.8gb 14.8gb green open shrink-winlogbeat-6.7.2-2019.12.30 xD5l4Df-TDGu4lhd2xhNHg 1 0 34196816 0 14.9gb 14.9gb green open shrink-winlogbeat-6.7.2-2019.12.31 vaEyngFeRQeCzecwXWTZIQ 1 0 31405106 0 13.5gb 13.5gb green open shrink-winlogbeat-6.7.2-2020.01.02 uRinwYKDTqyd5pqqp8NSUg 1 0 4141 0 2.9mb 2.9mb green open shrink-winlogbeat-6.7.2-2020.01.03 2p-XZ7PZTWyDbUkIU4pNSA 1 0 308 0 374.7kb 374.7kb green open shrink-winlogbeat-6.7.2-2020.01.04 J6vVy9BlSLOvPErkZUVwpw 1 0 476 0 655.3kb 655.3kb green open shrink-winlogbeat-6.7.2-2020.01.05 yosgHQnMSeqzOIOq1HCb-A 1 0 695 0 898.9kb 898.9kb green open shrink-winlogbeat-6.7.2-2020.01.07 gFFH5ZHFRV2GpzWXJT85nQ 1 0 52 0 156.8kb 156.8kb yellow open winlogbeat-6.3.2-2019.12.23 vwRq3V6tSwWMufJ0Mkom0g 5 1 3997515 0 3.4gb 3.4gb yellow open winlogbeat-6.3.2-2019.12.24 GhbSRNlpSFCZa7QpBCdFqA 5 1 3161797 0 2.6gb 2.6gb yellow open winlogbeat-6.3.2-2019.12.25 DjUSiIeCTlmGSA492jhCqg 5 1 2927893 0 2.4gb 2.4gb yellow open winlogbeat-6.3.2-2019.12.26 SaNWnXjLTouzVWNpisuVXA 5 1 3162800 0 2.6gb 2.6gb yellow open winlogbeat-6.3.2-2019.12.27 zzDAAzz3TOOMSU-9w2E9lA 5 1 3290534 0 2.8gb 2.8gb yellow open winlogbeat-6.3.2-2019.12.28 K7zlp4qnSGKxwlPbFTzXhw 5 1 2957441 0 2.5gb 2.5gb yellow open winlogbeat-6.3.2-2019.12.29 mCj92_3zQ1a6-2RPm28wmA 5 1 2611337 0 2.1gb 2.1gb yellow open winlogbeat-6.3.2-2019.12.30 SfeYmT1lQ9GsabKDuV417g 5 1 2563662 0 2.1gb 2.1gb yellow open winlogbeat-6.3.2-2019.12.31 x9W81kIVTK6E5qSwv9EWDg 5 1 2015530 0 1.6gb 1.6gb yellow open winlogbeat-6.3.2-2020.01.01 AICHbm1DQfiV9mFUQrMDNA 5 1 3969 0 3.8mb 3.8mb yellow open winlogbeat-6.3.2-2020.01.02 X6sFQF6BQnutpBtJhiqbIw 5 1 315 0 1.4mb 1.4mb yellow open winlogbeat-6.3.2-2020.01.03 -a26yxLSR0qVopTGVbyJGQ 5 1 172 0 642.7kb 642.7kb yellow open winlogbeat-6.3.2-2020.01.04 nir5lgB7Tt27akcDg9Xzpg 5 1 136 0 613.5kb 613.5kb yellow open winlogbeat-6.3.2-2020.01.05 ChlulbzATr2SVOqiLmFaWw 5 1 230 0 429kb 429kb yellow open winlogbeat-6.3.2-2020.01.06 mHQH75kWQXu0tpAZmRoMaQ 5 1 4485 0 5.9mb 5.9mb yellow open winlogbeat-6.3.2-2020.01.07 XRxnNeUzSWqFIcsYzt06FQ 5 1 4058 0 5.3mb 5.3mb yellow open winlogbeat-6.3.2-2020.01.08 _RgNqtNxTdiERYfabJIifw 5 1 2203 0 3.2mb 3.2mb yellow open winlogbeat-6.3.2-2020.01.09 06dP4jrSTKOv8VRtVhl1QA 5 1 1682 0 2.7mb 2.7mb yellow open winlogbeat-6.3.2-2020.01.10 SQImvDprQnyQSlRHr1bCRg 5 1 1177 0 2.7mb 2.7mb yellow open winlogbeat-6.3.2-2020.01.11 xCMBIz5SSFKxH1MxSYEVMw 5 1 672 0 1.3mb 1.3mb green open winlogbeat-6.7.2-2019.10.24 g5w7wUtWSAmfgrobpWQvCg 5 0 34679405 0 15.8gb 15.8gb green open winlogbeat-6.7.2-2020.01.14 zRQ0ZqAUR0yzN0MgjbKUKQ 5 0 1181 0 1.2mb 1.2mb green open winlogbeat-6.7.2-2020.01.15 ikMNhJRHQL2uISs_IXgW-A 5 0 2601 0 2.2mb 2.2mb green open winlogbeat-6.7.2-2020.01.16 fqr-G2FiRwKZtDE-88m69w 5 0 4749 0 4.2mb 4.2mb green open winlogbeat-6.7.2-2020.01.17 _OkA3FNVRcOkNgMQdTiBCw 5 0 2695 0 2.4mb 2.4mb ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 18, 2020, 8:37pm UTC](https://discuss.elastic.co/t/logstash-stopped-accepting-events-from-winlogbeat-since-the-new-year/215941/4 "2020-02-18T20:37:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.