Posting this here as I think this is a logstash problem although I'm not really sure.
ELK version 6.7.2
Winlogbeat version is a mix of 6.7.2 and 6.3.2
At midnight on January 1st, all winlogbeat events stopped going in to logstash and nothing has changed in our environment. I went from 26 million logs on December 31 to 0 on January 1. I have had a few events here and there sneak in but nothing consistent. Maybe a thousand in 1 day.
Logstash logs show a 403 forbidden/8.index write error, however it looks like those started around the 5th of December so I don't think that error is related to my issue. Elasticsearch shows nothing.
I have tried sending the events directly to elasticsearch and that works so I don't think it is a client issue.
Logs from the client:
C:\Program Files\Winlogbeat>winlogbeat.exe -e -c winlogbeat.yml
2020-01-21T14:32:23.982-0500 INFO instance/beat.go:611 Home path: [C:\Program Files\Winlogbeat] Config path: [C:\Program Files\Winlogbeat] Data path: [C:\Program Files\Winlogbeat\data] Logs path: [C:\Program Files\Winlogbeat\logs]
2020-01-21T14:32:23.984-0500 INFO instance/beat.go:618 Beat UUID: b36cbe30-cf3e-4de7-aed9-16c3f4cdc7f3
2020-01-21T14:32:23.987-0500 INFO [beat] instance/beat.go:931 Beat info {"system_info": {"beat": {"path": {"config": "C:\Program Files\Winlogbeat", "data": "C:\Program Files\Winlogbeat\data", "home": "C:\Program Files\Winlogbeat", "logs": "C:\Program Files\Winlogbeat\logs"}, "type": "winlogbeat", "uuid": "b36cbe30-cf3e-4de7-aed9-16c3f4cdc7f3"}}}
2020-01-21T14:32:23.987-0500 INFO [beat] instance/beat.go:940 Build info {"system_info": {"build": {"commit": "a8ab26dd1f818d27c17c3049f643652c6a789d88", "libbeat": "6.7.2", "time": "2019-04-29T08:23:56.000Z", "version": "6.7.2"}}}
2020-01-21T14:32:23.988-0500 INFO [beat] instance/beat.go:943 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":2,"version":"go1.10.8"}}}
2020-01-21T14:32:23.996-0500 INFO [beat] instance/beat.go:947 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-10-30T12:02:09.92-04:00","name":"BAPM31001","ip":["172.32.7.60/24","::1/128","127.0.0.1/8"],"kernel_version":"10.0.17134.1 (WinBuild.160101.0800)","mac":["00:50:56:a6:89:dd"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Pro","version":"10.0","major":10,"minor":0,"patch":0,"build":"17134.1"},"timezone":"EST","timezone_offset_sec":-18000,"id":"e1c2f043-6a3f-4945-ab0b-53307a97606b"}}}
2020-01-21T14:32:24.004-0500 INFO [beat] instance/beat.go:976 Process info {"system_info": {"process": {"cwd": "C:\Program Files\Winlogbeat", "exe": "C:\Program Files\Winlogbeat\winlogbeat.exe", "name": "winlogbeat.exe", "pid": 42312, "ppid": 42160, "start_time": "2020-01-21T14:32:23.883-0500"}}}
2020-01-21T14:32:24.006-0500 INFO instance/beat.go:280 Setup Beat: winlogbeat; Version: 6.7.2
2020-01-21T14:32:24.007-0500 INFO [publisher] pipeline/module.go:110 Beat name: BAPM31001
2020-01-21T14:32:24.011-0500 INFO beater/winlogbeat.go:68 State will be read from and persisted to C:\Program Files\Winlogbeat\data.winlogbeat.yml
2020-01-21T14:32:24.013-0500 INFO elasticsearch/client.go:164 Elasticsearch url: http://elasticsearch_address_here:9200
2020-01-21T14:32:24.013-0500 INFO instance/beat.go:402 winlogbeat start running.
2020-01-21T14:32:24.034-0500 INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
2020-01-21T14:32:24.100-0500 INFO [monitoring] elasticsearch/elasticsearch.go:247 Successfully connected to X-Pack Monitoring endpoint.
2020-01-21T14:32:24.115-0500 INFO [monitoring] elasticsearch/elasticsearch.go:261 Start monitoring stats metrics snapshot loop with period 10s.
2020-01-21T14:32:24.138-0500 INFO [monitoring] elasticsearch/elasticsearch.go:261 Start monitoring state metrics snapshot loop with period 1m0s.
2020-01-21T14:32:25.378-0500 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://logstash_address_here:5045))
2020-01-21T14:32:25.460-0500 INFO pipeline/output.go:105 Connection to backoff(async(tcp://logstash_address_here:5045)) established
2020-01-21T14:32:34.285-0500 INFO pipeline/output.go:95 Connecting to backoff(publish(elasticsearch(http://logstash_address_here:9200)))
2020-01-21T14:32:34.319-0500 INFO pipeline/output.go:105 Connection to backoff(publish(elasticsearch(http://logstash_address_here:9200))) established
2020-01-21T14:32:54.042-0500 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":375,"time":{"ms":375}},"total":{"ticks":2375,"time":{"ms":2375},"value":2375},"user":{"ticks":2000,"time":{"ms":2000}}},"handles":{"open":370},"info":{"ephemeral_id":"d7e61c20-a170-4fcd-bd0e-d60f89d068f6","uptime":{"ms":30135}},"memstats":{"gc_next":24508304,"memory_alloc":12812480,"memory_total":89719496,"rss":42831872}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":2542,"batches":3,"total":2542},"read":{"bytes":3074},"type":"logstash","write":{"bytes":317548}},"pipeline":{"clients":3,"events":{"active":2543,"published":2543,"retry":1404,"total":2543}}},"msg_file_cache":{"ApplicationHits":191,"ApplicationMisses":10,"ApplicationSize":10,"SecurityHits":1838,"SecurityMisses":1,"SecuritySize":1,"SystemHits":497,"SystemMisses":7,"SystemSize":7},"system":{"cpu":{"cores":2}}}}}
Debug log from the client adds the following:
2020-01-21T14:34:37.996-0500 DEBUG [publish] pipeline/client.go:201 Pipeline client receives callback 'onDroppedOnPublish' for event: %+v{2020-01-19 12:41:55.5771433 +0000 UTC null {"activity_id":"{2AAD78B0-8F3B-0000-3179-AD2A3B8FD501}","computer_name":"computer.bapm.com","event_data":{"AuditPolicyChanges":"%%8451","CategoryId":"%%8279","SubcategoryGuid":"{0CCE923E-69AE-11D9-BED3-505054503030}","SubcategoryId":"%%14083","SubjectDomainName":"BAPM.COM","SubjectLogonId":"0x3e7","SubjectUserName":"Computer$","SubjectUserSid":"S-1-5-18"},"event_id":4719,"keywords":["Audit Success"],"level":"Information","log_name":"Security","message":"System audit policy was changed.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tcomputer$\n\tAccount Domain:\t\tBAPM.COM\n\tLogon ID:\t\t0x3E7\n\nAudit Policy Change:\n\tCategory:\t\tDS Access\n\tSubcategory:\t\tDetailed Directory Service Replication\n\tSubcategory GUID:\t{0CCE923E-69AE-11D9-BED3-505054503030}\n\tChanges:\t\tFailure added","opcode":"Info","process_id":684,"provider_guid":"{54849625-5478-4994-A5BA-3E3B0328C30D}","record_number":"309277","source_name":"Microsoft-Windows-Security-Auditing","task":"Audit Policy Change","thread_id":38148,"type":"wineventlog"} {Security 309277 2020-01-19 12:41:55.5771433 +0000 UTC
}}
Config file for winlogbeat:
>
> #======================= Winlogbeat specific options ==========================
>
> # event_logs specifies a list of event logs to monitor as well as any
> # accompanying options. The YAML data type of event_logs is a list of
> # dictionaries.
> #
> # The supported keys are name (required), tags, fields, fields_under_root,
> # forwarded, ignore_older, level, event_id, provider, and include_xml. Please
> # visit the documentation for the complete details of each option.
> # Configure Winlogbeat | Winlogbeat Reference [8.11] | Elastic
> winlogbeat.event_logs:
> - name: Application
> ignore_older: 72h
> - name: Security
> ignore_older: 72h
> - name: System
> ignore_older: 72h
>
> #==================== Elasticsearch template setting ==========================
>
> #setup.template.settings:
> # index.number_of_shards: 3
> #index.codec: best_compression
> #_source.enabled: false
>
> #================================ General =====================================
>
> # The name of the shipper that publishes the network data. It can be used to group
> # all the transactions sent by a single shipper in the web interface.
> #name:
>
> # The tags of the shipper are included in their own field with each
> # transaction published.
> #tags: ["service-X", "web-tier"]
>
> # Optional fields that you can specify to add additional information to the
> # output.
> #fields:
> # env: staging
>
>
> #============================== Dashboards =====================================
> # These settings control loading the sample dashboards to the Kibana index. Loading
> # the dashboards is disabled by default and can be enabled either by setting the
> # options here, or by using the -setup CLI flag or the setup command.
> #setup.dashboards.enabled: false
>
> # The URL from where to download the dashboards archive. By default this URL
> # has a value which is computed based on the Beat name and version. For released
> # versions, this URL points to the dashboard archive on the artifacts.elastic.co
> # website.
> #setup.dashboards.url:
>
> #============================== Kibana =====================================
>
> # Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
> # This requires a Kibana endpoint configuration.
> setup.kibana:
>
> # Kibana Host
> # Scheme and port can be left out and will be set to the default (http and 5601)
> # In case you specify and additional path, the scheme is required: http://localhost:5601/path
> # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
> #host: "localhost:5601"
>
> #============================= Elastic Cloud ==================================
>
> # These settings simplify using winlogbeat with the Elastic Cloud (https://cloud.elastic.co/).
>
> # The cloud.id setting overwrites the output.elasticsearch.hosts and
> # setup.kibana.host options.
> # You can find the cloud.id in the Elastic Cloud web UI.
> #cloud.id:
>
> # The cloud.auth setting overwrites the output.elasticsearch.username and
> # output.elasticsearch.password settings. The format is <user>:<pass>.
> #cloud.auth:
>
> #================================ Outputs =====================================
>
> # Configure what output to use when sending the data collected by the beat.
>
> #-------------------------- Elasticsearch output ------------------------------
> #output.elasticsearch:
> # Array of hosts to connect to.
> # hosts: ["localhost:9200"]
>
> # Optional protocol and basic auth credentials.
> #protocol: "https"
> #username: "elastic"
> #password: "changeme"
>
> #----------------------------- Logstash output --------------------------------
> output.logstash:
> # The Logstash hosts
> # 5044 is unsecured and Servers
> # 5045 is secured and for workstations.
> hosts: ["logstash_address:5045"]
>
> # Enable SSL support. SSL is automatically enabled, if any SSL setting is set.
> ssl.enabled: true
>
> # Configure SSL verification mode. If none is configured, all server hosts
> # and certificates will be accepted. In this mode, SSL based connections are
> # susceptible to man-in-the-middle attacks. Use only for testing. Default is
> # full.
> ssl.verification_mode: full
>
> # Optional SSL. By default is off.
> # List of root certificates for HTTPS server verifications
> #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
>
> # Certificate for SSL client authentication
> #ssl.certificate: "/etc/pki/client/cert.pem"
>
> # Client Certificate Key
> #ssl.key: "/etc/pki/client/cert.key"
>
> #================================ Logging =====================================
>
> # Sets log level. The default log level is info.
> # Available log levels are: critical, error, warning, info, debug
> #logging.level: debug
>
> # At debug level, you can selectively enable logging only for some components.
> # To enable all selectors use [""]. Examples of other selectors are "beat",
> # "publish", "service".
> #logging.selectors: [""]
> #============================== Xpack Monitoring =====================================
> xpack.monitoring:
> enabled: true
> elasticsearch:
> hosts: ["http://elasticsearch_address:9200"]
> username: username
> password: password