Beats (filebeat/metricbeat etc.) log to syslog with "logging.to_syslog: false"

[bcsmith (Brian Smith)]

Hello I am looking at a host running Ubuntu Xenial, Logging goes to the /var/log/filebeat/filebeat fine, until an index it is writing to goes read only. Then filebeat spams /var/log/syslog with messages like the following until the disk fills to 100%. This behavior is also present with the other beats we run, auditbeat, metricbeat, packetbeat etc.

Apr 29 18:06:39 SYSTEM filebeat[11667]: 2019-04-29T18:06:39.367-0400#011ERROR#011pipeline/output.go:121#011Failed to publish events: 500 Internal Server Error: {"took":488,"ignored":false,"errors":true,"error":{"type":"export_exception","reason":"Exception when closing export bulk","caused_by":{"type":"export_exception","reason":"failed to flush export bulks","caused_by":{"type":"export_exception","reason":"bulk [default_local] reports failures when exporting documents","exceptions":[{"type":"export_exception","reason":"RemoteTransportException[[HOST2][IP:9300][indices:data/write/bulk[s]]]; nested: RemoteTransportException[[HOST2][IP:9300][indices:data/write/bulk[s][p]]]; nested: EsRejectedExecutionException[rejected execution of processing of [199702778][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[.monitoring-beats-7-2019.04.29][0]] containing [index {[.monitoring-beats-7-2019.04.29][_doc][cA4ia2oBpquKvGWFtIto], source[{"cluster_uuid":"Ji7pE5ekSDe3tRs0reFBKg","timestamp":"2019-04-29T22:06:37.665Z","interval_ms":10000,"type":"beats_stats","source_node":{"uuid":"1IoN1a8ITsukkwI261JTmQ","host":"IP","transport_address":"IP:9300","ip":"IP","name":"HOST1","timestamp":"2019-04-29T22:06:37.666Z"},"beats_stats":{"timestamp":"2019-04-29T22:06:31.704Z","metrics":{"beat":{"memstats":{"gc_next":5906736,"rss":44802048,"memory_total":132013275744,"memory_alloc":4615248},"cpu":{"user":{"ticks":801710,"time":{"ms":801716}},"system":{"time":{"ms":446512},"ticks":446510},"total":{"value":1248220.0,"ticks":1248220,"time":{"ms":1248228}}},"handles":{"open":16,"limit":{"hard":4096,"soft":1024}},"info":{"uptime":{"ms":527520104},"ephemeral_id":"be8c2fe5-0187-4c80-bcbd-c52c48376f41"}},"registrar":{"states":{"cleanup":16,"current":8,"update":744189},"writes":{"fail":0,"success":399174,"total":399174}},"filebeat":{"harveste

Here is the readacted filebeat.yml:

filebeat.inputs:

• type: log
  enabled: false
  paths:
  • /var/log/.log
    filebeat.config.modules:
    path: ${path.config}/modules.d/.yml
    reload.enabled: false
    setup.template.settings:
    index.number_of_shards: 3
    tags: ["MyTAG"]
    fields:
    env: ENV
    setup.kibana:
    output.elasticsearch:
    hosts: ["HOST1:9200", "HOST2:9200"]
    ilm.enabled: true
    processors:
  • add_host_metadata: ~
  • add_cloud_metadata: ~
    logging.level: warning
    logging.to_syslog: false
    logging.to_files: true
    logging.files.rotateeverybytes: 2097152
    logging.files.keepfiles: 5
    xpack.monitoring.enabled: true
    xpack.monitoring.elasticsearch:

Here's the /etc/rsyslog.d/50-defaults.conf:
auth,authpriv.* /var/log/auth.log
*.*;auth,authpriv.none -/var/log/syslog
kern.* -/var/log/kern.log
mail.* -/var/log/mail.log
mail.err /var/log/mail.err
news.crit /var/log/news/news.crit
news.err /var/log/news/news.err
news.notice -/var/log/news/news.notice
.emerg :omusrmsg:
daemon.;mail.;
news.err;
.=debug;.=info;
.=notice;.=warn |/dev/xconsole

Can anyone help with why when it has a connection error it is logging to syslog. We only want the beats to log to their files with a certain rotation to keep the system from filling up the drives. Thanks!

[kvch (Noémi Ványi)]

Does journald run on your system? How do you start the Beat service?

[bcsmith (Brian Smith)]

I have this journald process:
root 1561 1 0 2018 ? 00:13:57 /lib/systemd/systemd-journald

The beats are installed from the elastic repo and I use the default system scripts to start them.

I am currently on version 7.0

[bcsmith (Brian Smith)]

Another sample log from syslog
Apr 29 21:39:04 HOST filebeat[11667]: ll:0xbf29d2215e5744af, ext:485438004601902, loc:(*time.Location)(0x2576ec0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x9f346, Device:0xfc00}}}, Flags:0x1} (status=403): {"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}", "input":common.MapStr{"type":"log"}, "ecs":common.MapStr{"version":"1.0.0"}, "agent":common.MapStr{"id":"1de48a7d-4522-42e5-841f-fad32abec946", "version":"7.0.0", "type":"filebeat", "ephemeral_id":"be8c2fe5-0187-4c80-bcbd-c52c48376f41", "hostname":"HOST"}, "log":common.MapStr{"offset":4240367313, "file":common.MapStr{"path":"/var/log/syslog"}}, "tags":string{"Java", "NGINX", "web-tier"}, "event":common.MapStr{"module":"system", "dataset":"system.syslog", "timezone":"-04:00"}, "fileset":common.MapStr{"name":"syslog"}, "service":common.MapStr{"type":"system"}, "fields":common.MapStr{"env":"ENV"}, "host":common.MapStr{"name":"HOST", "id":"368271f369e754282a48113b5acbb98a", "containerized":false, "hostname":"HOST", "architecture":"x86_64", "os":common.MapStr{"version":"16.04.4 LTS (Xenial Xerus)", "family":"debian", "name":"Ubuntu", "kernel":"4.4.0-31-generic", "codename":"xenial", "platform":"ubuntu"}}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc000156b60), Source:"/var/log/syslog", Offset:4240369218, Timestamp:time.Time{wall:0xbf29d2215e5744af, ext:485438004601902, loc:(*time.Location)(0x2576ec0)}, TTL:-1, Type:"log", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0x9f346, Device:0xfc00}}}, Flags:0x1} (status=403): {"type":"cluster_block_exception","reason":"blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}

[bcsmith (Brian Smith)]

Ok, after looking at this, it seems -e was being set by /lib/systemd/system/filebeat.service with this config for the Service:

[Service]
Environment="BEAT_LOG_OPTS=-e"
Environment="BEAT_CONFIG_OPTS=-c /etc/filebeat/filebeat.yml"
Environment="BEAT_PATH_OPTS=-path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat"
ExecStart=/usr/share/filebeat/bin/filebeat $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS
Restart=always

I have commented out the Environment variable setting the -e arg. According to filebeat help:
-e, --e Log to stderr and disable syslog/file output

Then ran
systemctl reenable filebeat

Which explains the behavior of the beats sending to syslog and ignoring the setting:
logging.to_syslog: false

I would recommend removing the -e by default and anyone having this issue you can remove that setting and all logging will go to /var/log/filebeat/filebeat rather than /var/log/syslog for error.

[robscott27]

Thank goodness I wasn't the only one with this issue - I came to specifically check whether anyone else had run into this problem. Commenting out the Environment="BEAT_LOG_OPTS=-e"also resolved the issue for me.

I found this issue was in both the 7.0.0 and 7.0.1 releases.

Thanks bcsmith!