Hello I am looking at a host running Ubuntu Xenial, Logging goes to the /var/log/filebeat/filebeat fine, until an index it is writing to goes read only. Then filebeat spams /var/log/syslog with messages like the following until the disk fills to 100%. This behavior is also present with the other beats we run, auditbeat, metricbeat, packetbeat etc.
Apr 29 18:06:39 SYSTEM filebeat[11667]: 2019-04-29T18:06:39.367-0400#011ERROR#011pipeline/output.go:121#011Failed to publish events: 500 Internal Server Error: {"took":488,"ignored":false,"errors":true,"error":{"type":"export_exception","reason":"Exception when closing export bulk","caused_by":{"type":"export_exception","reason":"failed to flush export bulks","caused_by":{"type":"export_exception","reason":"bulk [default_local] reports failures when exporting documents","exceptions":[{"type":"export_exception","reason":"RemoteTransportException[[HOST2][IP:9300][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[HOST2][IP:9300][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of processing of [199702778][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[.monitoring-beats-7-2019.04.29][0]] containing [index {[.monitoring-beats-7-2019.04.29][_doc][cA4ia2oBpquKvGWFtIto], source[{"cluster_uuid":"Ji7pE5ekSDe3tRs0reFBKg","timestamp":"2019-04-29T22:06:37.665Z","interval_ms":10000,"type":"beats_stats","source_node":{"uuid":"1IoN1a8ITsukkwI261JTmQ","host":"IP","transport_address":"IP:9300","ip":"IP","name":"HOST1","timestamp":"2019-04-29T22:06:37.666Z"},"beats_stats":{"timestamp":"2019-04-29T22:06:31.704Z","metrics":{"beat":{"memstats":{"gc_next":5906736,"rss":44802048,"memory_total":132013275744,"memory_alloc":4615248},"cpu":{"user":{"ticks":801710,"time":{"ms":801716}},"system":{"time":{"ms":446512},"ticks":446510},"total":{"value":1248220.0,"ticks":1248220,"time":{"ms":1248228}}},"handles":{"open":16,"limit":{"hard":4096,"soft":1024}},"info":{"uptime":{"ms":527520104},"ephemeral_id":"be8c2fe5-0187-4c80-bcbd-c52c48376f41"}},"registrar":{"states":{"cleanup":16,"current":8,"update":744189},"writes":{"fail":0,"success":399174,"total":399174}},"filebeat":{"harveste
Here is the readacted filebeat.yml:
filebeat.inputs:
- type: log
enabled: false
paths:- /var/log/.log
filebeat.config.modules:
path: ${path.config}/modules.d/.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
tags: ["MyTAG"]
fields:
env: ENV
setup.kibana:
output.elasticsearch:
hosts: ["HOST1:9200", "HOST2:9200"]
ilm.enabled: true
processors: - add_host_metadata: ~
- add_cloud_metadata: ~
logging.level: warning
logging.to_syslog: false
logging.to_files: true
logging.files.rotateeverybytes: 2097152
logging.files.keepfiles: 5
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch:
- /var/log/.log
Here's the /etc/rsyslog.d/50-defaults.conf:
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
kern.* -/var/log/kern.log
mail.* -/var/log/mail.log
mail.err /var/log/mail.err
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
.emerg :omusrmsg:
daemon.;mail.;
news.err;
.=debug;.=info;
.=notice;.=warn |/dev/xconsole
Can anyone help with why when it has a connection error it is logging to syslog. We only want the beats to log to their files with a certain rotation to keep the system from filling up the drives. Thanks!