# Beats (filebeat/metricbeat etc.) log to syslog with "logging.to\_syslog: false" **URL:** \ **Category:** Beats\ **Created:** [April 30, 2019, 12:56pm UTC](https://discuss.elastic.co/t/beats-filebeat-metricbeat-etc-log-to-syslog-with-logging-to-syslog-false/179085 "2019-04-30T12:56:21Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![bcsmith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bcsmith/32/45321_2.png) [@bcsmith](https://discuss.elastic.co/u/bcsmith)\ **Post date:** [April 30, 2019, 12:56pm UTC](https://discuss.elastic.co/t/beats-filebeat-metricbeat-etc-log-to-syslog-with-logging-to-syslog-false/179085/1 "2019-04-30T12:56:22Z")
Hello I am looking at a host running Ubuntu Xenial, Logging goes to the /var/log/filebeat/filebeat fine, until an index it is writing to goes read only. Then filebeat spams /var/log/syslog with messages like the following until the disk fills to 100%. This behavior is also present with the other beats we run, auditbeat, metricbeat, packetbeat etc. Apr 29 18:06:39 SYSTEM filebeat[11667]: 2019-04-29T18:06:39.367-0400#011ERROR#011pipeline/output.go:121#011Failed to publish events: 500 Internal Server Error: {"took":488,"ignored":false,"errors":true,"error":{"type":"export\_exception","reason":"Exception when closing export bulk","caused\_by":{"type":"export\_exception","reason":"failed to flush export bulks","caused\_by":{"type":"export\_exception","reason":"bulk [default\_local] reports failures when exporting documents","exceptions":[{"type":"export\_exception","reason":"RemoteTransportException[[HOST2][IP:9300][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[HOST2][IP:9300][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of processing of [199702778][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[.monitoring-beats-7-2019.04.29][0]] containing [index {[.monitoring-beats-7-2019.04.29][\_doc][cA4ia2oBpquKvGWFtIto], source[{"cluster\_uuid":"Ji7pE5ekSDe3tRs0reFBKg","timestamp":"2019-04-29T22:06:37.665Z","interval\_ms":10000,"type":"beats\_stats","source\_node":{"uuid":"1IoN1a8ITsukkwI261JTmQ","host":"IP","transport\_address":"IP:9300","ip":"IP","name":"HOST1","timestamp":"2019-04-29T22:06:37.666Z"},"beats\_stats":{"timestamp":"2019-04-29T22:06:31.704Z","metrics":{"beat":{"memstats":{"gc\_next":5906736,"rss":44802048,"memory\_total":132013275744,"memory\_alloc":4615248},"cpu":{"user":{"ticks":801710,"time":{"ms":801716}},"system":{"time":{"ms":446512},"ticks":446510},"total":{"value":1248220.0,"ticks":1248220,"time":{"ms":1248228}}},"handles":{"open":16,"limit":{"hard":4096,"soft":1024}},"info":{"uptime":{"ms":527520104},"ephemeral\_id":"be8c2fe5-0187-4c80-bcbd-c52c48376f41"}},"registrar":{"states":{"cleanup":16,"current":8,"update":744189},"writes":{"fail":0,"success":399174,"total":399174}},"filebeat":{"harveste Here is the readacted filebeat.yml: filebeat.inputs: - type: log enabled: false paths: - /var/log/_.log filebeat.config.modules: path: ${path.config}/modules.d/_.yml reload.enabled: false setup.template.settings: index.number\_of\_shards: 3 tags: ["MyTAG"] fields: env: ENV setup.kibana: output.elasticsearch: hosts: ["HOST1:9200", "HOST2:9200"] ilm.enabled: true processors: - add\_host\_metadata: ~ - add\_cloud\_metadata: ~ logging.level: warning logging.to\_syslog: false logging.to\_files: true logging.files.rotateeverybytes: 2097152 logging.files.keepfiles: 5 xpack.monitoring.enabled: true xpack.monitoring.elasticsearch: Here's the /etc/rsyslog.d/50-defaults.conf: auth,authpriv.\* /var/log/auth.log \*.\*;auth,authpriv.none -/var/log/syslog kern.\* -/var/log/kern.log mail.\* -/var/log/mail.log mail.err /var/log/mail.err news.crit /var/log/news/news.crit news.err /var/log/news/news.err news.notice -/var/log/news/news.notice _.emerg :omusrmsg:_ daemon._;mail._; news.err; _.=debug;_.=info; _.=notice;_.=warn |/dev/xconsole Can anyone help with why when it has a connection error it is logging to syslog. We only want the beats to log to their files with a certain rotation to keep the system from filling up the drives. Thanks! ---
**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\ **Post date:** [April 30, 2019, 1:22pm UTC](https://discuss.elastic.co/t/beats-filebeat-metricbeat-etc-log-to-syslog-with-logging-to-syslog-false/179085/2 "2019-04-30T13:22:52Z")
Does journald run on your system? How do you start the Beat service? ---
**Author:** ![bcsmith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bcsmith/32/45321_2.png) [@bcsmith](https://discuss.elastic.co/u/bcsmith)\ **Post date:** [April 30, 2019, 1:25pm UTC](https://discuss.elastic.co/t/beats-filebeat-metricbeat-etc-log-to-syslog-with-logging-to-syslog-false/179085/3 "2019-04-30T13:25:16Z")
I have this journald process: root 1561 1 0 2018 ? 00:13:57 /lib/systemd/systemd-journald The beats are installed from the elastic repo and I use the default system scripts to start them. I am currently on version 7.0 ---
**Author:** ![bcsmith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bcsmith/32/45321_2.png) [@bcsmith](https://discuss.elastic.co/u/bcsmith)\ **Post date:** [April 30, 2019, 1:31pm UTC](https://discuss.elastic.co/t/beats-filebeat-metricbeat-etc-log-to-syslog-with-logging-to-syslog-false/179085/4 "2019-04-30T13:31:51Z")
Another sample log from syslog Apr 29 21:39:04 HOST filebeat[11667]: ll:0xbf29d2215e5744af, ext:485438004601902, loc:(\*time.Location)(0x2576ec0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x9f346, Device:0xfc00}}}, Flags:0x1} (status=403): {"type":"cluster\_block\_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}", "input":common.MapStr{"type":"log"}, "ecs":common.MapStr{"version":"1.0.0"}, "agent":common.MapStr{"id":"1de48a7d-4522-42e5-841f-fad32abec946", "version":"7.0.0", "type":"filebeat", "ephemeral\_id":"be8c2fe5-0187-4c80-bcbd-c52c48376f41", "hostname":"HOST"}, "log":common.MapStr{"offset":4240367313, "file":common.MapStr{"path":"/var/log/syslog"}}, "tags":string{"Java", "NGINX", "web-tier"}, "event":common.MapStr{"module":"system", "dataset":"system.syslog", "timezone":"-04:00"}, "fileset":common.MapStr{"name":"syslog"}, "service":common.MapStr{"type":"system"}, "fields":common.MapStr{"env":"ENV"}, "host":common.MapStr{"name":"HOST", "id":"368271f369e754282a48113b5acbb98a", "containerized":false, "hostname":"HOST", "architecture":"x86\_64", "os":common.MapStr{"version":"16.04.4 LTS (Xenial Xerus)", "family":"debian", "name":"Ubuntu", "kernel":"4.4.0-31-generic", "codename":"xenial", "platform":"ubuntu"}}}, Private:file.State{Id:"", Finished:false, Fileinfo:(\*os.fileStat)(0xc000156b60), Source:"/var/log/syslog", Offset:4240369218, Timestamp:time.Time{wall:0xbf29d2215e5744af, ext:485438004601902, loc:(\*time.Location)(0x2576ec0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x9f346, Device:0xfc00}}}, Flags:0x1} (status=403): {"type":"cluster\_block\_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"} ---
**Author:** ![bcsmith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bcsmith/32/45321_2.png) [@bcsmith](https://discuss.elastic.co/u/bcsmith)\ **Post date:** [April 30, 2019, 5:20pm UTC](https://discuss.elastic.co/t/beats-filebeat-metricbeat-etc-log-to-syslog-with-logging-to-syslog-false/179085/5 "2019-04-30T17:20:07Z")
Ok, after looking at this, it seems -e was being set by /lib/systemd/system/filebeat.service with this config for the Service: [Service] Environment="BEAT\_LOG\_OPTS=-e" Environment="BEAT\_CONFIG\_OPTS=-c /etc/filebeat/filebeat.yml" Environment="BEAT\_PATH\_OPTS=-path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat" ExecStart=/usr/share/filebeat/bin/filebeat $BEAT\_LOG\_OPTS $BEAT\_CONFIG\_OPTS $BEAT\_PATH\_OPTS Restart=always I have commented out the Environment variable setting the -e arg. According to filebeat help: -e, --e Log to stderr and disable syslog/file output Then ran systemctl reenable filebeat Which explains the behavior of the beats sending to syslog and ignoring the setting: logging.to\_syslog: false I would recommend removing the -e by default and anyone having this issue you can remove that setting and all logging will go to /var/log/filebeat/filebeat rather than /var/log/syslog for error. ---
**Author:** ![robscott27](https://avatars.discourse-cdn.com/v4/letter/r/858c86/32.png) [@robscott27](https://discuss.elastic.co/u/robscott27)\ **Post date:** [May 1, 2019, 8:08pm UTC](https://discuss.elastic.co/t/beats-filebeat-metricbeat-etc-log-to-syslog-with-logging-to-syslog-false/179085/6 "2019-05-01T20:08:37Z")
Thank goodness I wasn't the only one with this issue - I came to specifically check whether anyone else had run into this problem. Commenting out the `Environment="BEAT_LOG_OPTS=-e"`also resolved the issue for me. I found this issue was in both the 7.0.0 and 7.0.1 releases. Thanks bcsmith! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 29, 2019, 10:08pm UTC](https://discuss.elastic.co/t/beats-filebeat-metricbeat-etc-log-to-syslog-with-logging-to-syslog-false/179085/7 "2019-05-29T22:08:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.