Hello everyone,
I recently came across the topic of Elastic Search, following the motto: "Can you take a look at this" We have been having an issue since November where the transmission on our Active Directory Domain Controllers freezes after some time. Unfortunately, I am not getting any useful information from the log files that could help me solve the problem. Could you give me a few tips here? Windows Server 2022 Elastic Kibana v 8.14.3 Winlogbeat v8.16.1
Can you share your winlogbeat configuration as well as the log?
# ======================== Winlogbeat specific options =========================
winlogbeat.event_logs:
- name: Application
ignore_older: 72h
- name: System
- name: Security
- name: Microsoft-Windows-Sysmon/Operational
- name: Windows PowerShell
event_id: 400, 403, 600, 800
- name: Microsoft-Windows-PowerShell/Operational
event_id: 4103, 4104, 4105, 4106
- name: ForwardedEvents
tags: [forwarded]
# ====================== Elasticsearch template settings =======================
setup.template.settings:
index.number_of_shards: 1
# ================================== General ===================================
write_ahead: 2048
# =================================== Kibana ===================================
setup.kibana:
host: 'https://xxx.xxx.xxx.xxx:xxxx'
ssl.enabled: true
username: "username"
password: "password"
# ================================== Outputs ===================================
hosts: ['https://xxx.xxx.xxx.xxx:xxxx', 'https://xxx.xxx.xxx.xxx:xxxx', 'https://xxx.xxx.xxx.xxx:xxxx']
protocol: "https"
username: "username"
password: "password"
index: "index-name"
# ============================= X-Pack Monitoring ==============================
monitoring.enabled: false
setup.template.name: "template-name"
setup.template.name: "template-pattern"
output.elasticsearch.ssl.certificate_authorities: ["path to pem\\elasticsearch-ca.pem"]
setup.ilm.enabled: true
setup.ilm.policy_name: "policy-name"
and all log entries end with:
{"log.level":"info","@timestamp":"2024-12-21T06:03:59.329+0100","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"winlogbeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":393468,"time":{"ms":250}},"total":{"ticks":1038780,"time":{"ms":297},"value":1038780},"user":{"ticks":645312,"time":{"ms":47}}},"info":{"ephemeral_id":"347f1cb7-99a5-41d5-982a-4410e72c04d6","uptime":{"ms":54600122},"version":"8.16.1"},"memstats":{"gc_next":53790064,"memory_alloc":44754496,"memory_total":88799208520,"rss":105578496},"runtime":{"goroutines":46}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":450,"active":0,"batches":3,"total":450},"read":{"bytes":12155,"errors":3},"write":{"bytes":165398,"latency":{"histogram":{"count":5802,"max":792,"mean":109.5810546875,"median":78,"min":15,"p75":104,"p95":404.5,"p99":592.25,"p999":791.0750000000008,"stddev":117.12300422051013}}}},"pipeline":{"clients":7,"events":{"active":133,"published":414,"total":414},"queue":{"acked":450,"added":{"bytes":1197830,"events":414},"consumed":{"bytes":1359364,"events":450},"filled":{"bytes":372462,"events":133,"pct":0.0415625},"max_bytes":0,"max_events":3200,"removed":{"bytes":1359364,"events":450}}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-12-21T06:04:29.333+0100","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"winlogbeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":393703,"time":{"ms":235}},"total":{"ticks":1039218,"time":{"ms":438},"value":1039218},"user":{"ticks":645515,"time":{"ms":203}}},"info":{"ephemeral_id":"347f1cb7-99a5-41d5-982a-4410e72c04d6","uptime":{"ms":54630125},"version":"8.16.1"},"memstats":{"gc_next":54838120,"memory_alloc":41856960,"memory_total":88821422528,"rss":105521152},"runtime":{"goroutines":46}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":473,"active":0,"batches":3,"total":473},"read":{"bytes":12752,"errors":3},"write":{"bytes":162920,"latency":{"histogram":{"count":5805,"max":792,"mean":109.384765625,"median":78,"min":15,"p75":104,"p95":404.5,"p99":592.25,"p999":791.0750000000008,"stddev":116.93512157833}}}},"pipeline":{"clients":7,"events":{"active":94,"published":434,"total":434},"queue":{"acked":473,"added":{"bytes":1298257,"events":434},"consumed":{"bytes":1388400,"events":473},"filled":{"bytes":282319,"events":94,"pct":0.029375},"max_bytes":0,"max_events":3200,"removed":{"bytes":1388400,"events":473}}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-12-21T06:04:59.326+0100","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"winlogbeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":394031,"time":{"ms":328}},"total":{"ticks":1039656,"time":{"ms":438},"value":1039656},"user":{"ticks":645625,"time":{"ms":110}}},"info":{"ephemeral_id":"347f1cb7-99a5-41d5-982a-4410e72c04d6","uptime":{"ms":54660119},"version":"8.16.1"},"memstats":{"gc_next":53719600,"memory_alloc":39825336,"memory_total":88845631216,"rss":105521152},"runtime":{"goroutines":46}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":435,"active":0,"batches":3,"total":435},"read":{"bytes":11765,"errors":3},"write":{"bytes":156697,"latency":{"histogram":{"count":5808,"max":792,"mean":109.3271484375,"median":78,"min":15,"p75":103.75,"p95":404.5,"p99":592.25,"p999":791.0750000000008,"stddev":116.95130950191341}}}},"pipeline":{"clients":7,"events":{"active":131,"published":472,"total":472},"queue":{"acked":435,"added":{"bytes":1429793,"events":472},"consumed":{"bytes":1305270,"events":435},"filled":{"bytes":406842,"events":131,"pct":0.0409375},"max_bytes":0,"max_events":3200,"removed":{"bytes":1305270,"events":435}}}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-12-21T06:05:29.327+0100","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"winlogbeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":394218,"time":{"ms":187}},"total":{"ticks":1040014,"time":{"ms":358},"value":1040014},"user":{"ticks":645796,"time":{"ms":171}}},"info":{"ephemeral_id":"347f1cb7-99a5-41d5-982a-4410e72c04d6","uptime":{"ms":54690120},"version":"8.16.1"},"memstats":{"gc_next":55336880,"memory_alloc":35120736,"memory_total":88865057112,"rss":105209856},"runtime":{"goroutines":46}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":428,"active":0,"batches":3,"total":428},"read":{"bytes":11582,"errors":3},"write":{"bytes":195075,"latency":{"histogram":{"count":5811,"max":792,"mean":109.3271484375,"median":78,"min":15,"p75":103.75,"p95":404.5,"p99":592.25,"p999":791.0750000000008,"stddev":116.95130950191341}}}},"pipeline":{"clients":7,"events":{"active":36,"published":333,"total":333},"queue":{"acked":428,"added":{"bytes":1516022,"events":333},"consumed":{"bytes":1808161,"events":428},"filled":{"bytes":114703,"events":36,"pct":0.01125},"max_bytes":0,"max_events":3200,"removed":{"bytes":1808161,"events":428}}}},"system":{"handles":{"open":-2}}},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2024-12-21T06:05:59.324+0100","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"winlogbeat","monitoring":{"metrics":{"beat":{"cpu":{"system":{"ticks":394421,"time":{"ms":203}},"total":{"ticks":1040374,"time":{"ms":360},"value":1040374},"user":{"ticks":645953,"time":{"ms":157}}},"info":{"ephemeral_id":"347f1cb7-99a5-41d5-982a-4410e72c04d6","uptime":{"ms":54720117},"version":"8.16.1"},"memstats":{"gc_next":55336880,"memory_alloc":50128312,"memory_total":88880064688,"rss":105324544},"runtime":{"goroutines":46}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":210,"active":0,"batches":3,"total":210},"read":{"bytes":5913,"errors":3},"write":{"bytes":80814,"latency":{"histogram":{"count":5814,"max":792,"mean":109.3271484375,"median":78,"min":15,"p75":103.75,"p95":404.5,"p99":592.25,"p999":791.0750000000008,"stddev":116.95130950191341}}}},"pipeline":{"clients":7,"events":{"active":119,"published":293,"total":293},"queue":{"acked":210,"added":{"bytes":852813,"events":293},"consumed":{"bytes":618383,"events":210},"filled":{"bytes":349133,"events":119,"pct":0.0371875},"max_bytes":0,"max_events":3200,"removed":{"bytes":618383,"events":210}}}},"system":{"handles":{"open":4}}},"ecs.version":"1.6.0"}}
Does it stop printing those nonzero metrics logs? Or does it keep printing those to the log?
Asked another way, is winlogbeat still running long after the "2024-12-21T06:05:59.324" was printed or does it keep printing those nonzero metric messages but you're not seeing anything in Elasticsearch?
The problem was caused by the installed virus scanner, presumably the DeepInspection module.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.