Filebeat refuses to log to file (7.0.1)

Elastic Stack Beats
1

Filebeat is configured to log to /var/log/filebeat, but it only logs to syslog. /var/log/filebeat directory does exist, and filebeat logs no error or warning to syslog to indicate any issues with logging to files.

I've got elasticsearch, logstash, filebeat and kibana installed, version 7.0.1 for all components. Installation was done via yum and using the elasticsearch 7 repo. OS is CentOS 7.

yum config:

[elasticsearch-7.x]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

Logging configuration from filebeat.yml:

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: info

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
logging.selectors: ["publish"]
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644
  interval: 24h

I would really appreciate any pointers to how to get it to log to the correct place.

1 Like
2

Hi,

Thank you for including all those details and formatting them correctly, that should make it quite easy and pleasant for me to hopefully help you very quickly.

From:

The unit file included in the packages sets the -e flag by default. This flag makes Filebeat log to stderr and disables other log outputs. Systemd stores all output sent to stderr in journald.

The systemd service unit file includes environment variables that you can override to change the default options.

To override these variables, create a drop-in unit file in the /etc/systemd/system/filebeat.service.d directory.

To use [the logging] settings from the Filebeat file, empty the environment variable. For example:

[Service]
Environment="BEAT_LOG_OPTS="

To apply your changes, reload the systemd configuration and restart the service:

systemctl daemon-reload
systemctl restart filebeat

It is recommended that you use a configuration management tool to include drop-in unit files. If you need to add a drop-in manually, use systemctl edit filebeat.service .

Reference for the command line of filebeat to understand what the unit file does and what it can do:

To see the default filebeat unit file that came with the installation do:

systemctl status filebeat

and check the content of the unit file. (The path to the unit file is in the output of the above command.)

If you think this is confusing a bit because the settings you put in the config files are ignored, you now know why; "-e".
I would tend to agree this is confusing but the reasons for doing it from the POV of Elastic are valid.

From:

  • On systems with systemd, the Beats log is now written to journald by default rather than file. To revert this behaviour override BEAT_LOG_OPTS with an empty value. 8942.

See:

I do think they missed an opportunity to link to:

from:

with a warning/note that logging configurations from the config file will be completely overridden by the unit file on systemd systems and clearly inviting users to go read the "running with systemd" documentation page.

If you want to open an issue on github to suggest that documentation improvement that would be a good thing.

Let me know if this helps.

3

Great feedback, thank you!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.