Filebeat 7.5.1 logging.files.path is not working

kyle_che · March 23, 2020, 11:48pm

I have put the following in my filebeats for 7.5.1 but everything is being logged to syslog still. any idea on why?

logging.level: info
logging.to_syslog: false
logging.to_files: true
logging.files:
path: /var/log/filebeat
name: filebeat
keepfiles: 7
permissions: 0644

MarianaD · March 24, 2020, 10:32am

hi @kyle_che, can you share the command you use to start filebeat?

kyle_che · March 24, 2020, 2:30pm

i'm on ubuntu and i am just running systemctl start filebeat ... which runs this in the background

/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

kyle_che · March 24, 2020, 9:52pm

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.6 LTS
Release: 16.04
Codename: xenial

MarianaD · March 25, 2020, 9:21am

@kyle_che, the command

/usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

contains the -e flag

  -e   	Log to stderr and disable syslog/file output

if you remove this flag, I believe it should work, let us know if it does not.

kyle_che · March 25, 2020, 2:48pm

how do i remove this flag? i'm just running "systemctl start filebeat".

Magnus_Kessler · March 25, 2020, 2:58pm

Run systemctl edit filebeat to create an override.conf file that should contain

[Service]
Environment="BEAT_LOG_OPTS="

For more information see also: https://www.elastic.co/guide/en/beats/filebeat/master/running-with-systemd.html

kyle_che · March 25, 2020, 3:25pm

tried this but seems to not be overriding the defaults.

root@d-gp2-kyle77-3:/var/log/filebeat# cat /etc/systemd/system/filebeat.service.d/override.conf
[Service]
BEAT_LOG_OPTS=/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

root@d-gp2-kyle77-3:/var/log/filebeat# systemctl stop filebeat
root@d-gp2-kyle77-3:/var/log/filebeat# systemctl status filebeat

filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/filebeat.service.d
`-override.conf
Active: inactive (dead) since Wed 2020-03-25 15:23:45 UTC; 4s ago
Docs: https://www.elastic.co/products/beats/filebeat
Main PID: 4057 (code=exited, status=0/SUCCESS)

Mar 25 15:23:45 d-gp2-kyle77-3 filebeat[4057]: 2020-03-25T15:23:45.874Z INFO crawler/crawler.go:165 Crawler stopped
Mar 25 15:23:45 d-gp2-kyle77-3 filebeat[4057]: 2020-03-25T15:23:45.874Z INFO registrar/registrar.go:367 Stopping Registrar
Mar 25 15:23:45 d-gp2-kyle77-3 filebeat[4057]: 2020-03-25T15:23:45.874Z INFO registrar/registrar.go:293 Ending Registrar
Mar 25 15:23:45 d-gp2-kyle77-3 filebeat[4057]: 2020-03-25T15:23:45.878Z INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metric
Mar 25 15:23:45 d-gp2-kyle77-3 filebeat[4057]: 2020-03-25T15:23:45.878Z INFO [monitoring] log/log.go:154 Uptime: 1m38.753173504s
Mar 25 15:23:45 d-gp2-kyle77-3 filebeat[4057]: 2020-03-25T15:23:45.878Z INFO [monitoring] log/log.go:131 Stopping metrics logging.
Mar 25 15:23:45 d-gp2-kyle77-3 filebeat[4057]: 2020-03-25T15:23:45.878Z INFO [monitoring] elasticsearch/elasticsearch.go:284 Stop monitoring stats metrics sna
Mar 25 15:23:45 d-gp2-kyle77-3 filebeat[4057]: 2020-03-25T15:23:45.878Z INFO [monitoring] elasticsearch/elasticsearch.go:284 Stop monitoring state metrics sna
Mar 25 15:23:45 d-gp2-kyle77-3 filebeat[4057]: 2020-03-25T15:23:45.878Z INFO instance/beat.go:435 filebeat stopped.
Mar 25 15:23:45 d-gp2-kyle77-3 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
root@d-gp2-kyle77-3:/var/log/filebeat# systemctl start filebeat
root@d-gp2-kyle77-3:/var/log/filebeat# systemctl status filebeat

filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/filebeat.service.d
-override.conf Active: active (running) since Wed 2020-03-25 15:23:55 UTC; 2s ago Docs: https://www.elastic.co/products/beats/filebeat Main PID: 4165 (filebeat) Tasks: 10 Memory: 9.8M CPU: 46ms CGroup: /system.slice/filebeat.service -4165 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var

Mar 25 15:23:55 d-gp2-kyle77-3 filebeat[4165]: 2020-03-25T15:23:55.270Z INFO log/input.go:152 Configured paths: [/var/log/syslog]
Mar 25 15:23:55 d-gp2-kyle77-3 filebeat[4165]: 2020-03-25T15:23:55.270Z INFO input/input.go:114 Starting input of type: log; ID: 17829469489203298047
Mar 25 15:23:55 d-gp2-kyle77-3 filebeat[4165]: 2020-03-25T15:23:55.270Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
Mar 25 15:23:55 d-gp2-kyle77-3 filebeat[4165]: 2020-03-25T15:23:55.271Z INFO log/harvester.go:251 Harvester started for file: /var/log/syslog
Mar 25 15:23:55 d-gp2-kyle77-3 filebeat[4165]: 2020-03-25T15:23:55.271Z INFO cfgfile/reload.go:171 Config reloader started
Mar 25 15:23:55 d-gp2-kyle77-3 filebeat[4165]: 2020-03-25T15:23:55.271Z INFO cfgfile/reload.go:226 Loading of config files completed.
Mar 25 15:23:55 d-gp2-kyle77-3 filebeat[4165]: 2020-03-25T15:23:55.287Z INFO elasticsearch/client.go:753 Attempting to connect to Elasticsearch version 7.5.1
Mar 25 15:23:55 d-gp2-kyle77-3 filebeat[4165]: 2020-03-25T15:23:55.344Z INFO [monitoring] elasticsearch/elasticsearch.go:262 Successfully connected to X-Pack
Mar 25 15:23:55 d-gp2-kyle77-3 filebeat[4165]: 2020-03-25T15:23:55.345Z INFO [monitoring] elasticsearch/elasticsearch.go:276 Start monitoring stats metrics sn
Mar 25 15:23:55 d-gp2-kyle77-3 filebeat[4165]: 2020-03-25T15:23:55.345Z INFO [monitoring] elasticsearch/elasticsearch.go:276 Start monitoring state metrics sn

root@d-gp2-kyle77-3:/var/log/filebeat# ps -ef | grep -i filebeat
root 4165 1 0 15:23 ? 00:00:00 /usr/share/filebeat/bin/filebeat -e -c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat

kyle_che · March 25, 2020, 3:30pm

nevermind, figured it out.... thanks for your help.

cat /etc/systemd/system/filebeat.service.d/override.conf

[Service]

Environment="BEAT_LOG_OPTS=-c /etc/filebeat/filebeat.yml -path.home /usr/share/filebeat -path.config /etc/filebeat -path.data /var/lib/filebeat -path.logs /var/log/filebeat"

Magnus_Kessler · March 25, 2020, 4:38pm

There is no need to repeat all the other filebeat settings in the BEAT_LOG_OPTS environment setting. It should just be empty if you'd like to log to the standard location under /var/log/filebeat.

So your file should look exactly like this:

[Service]
Environment="BEAT_LOG_OPTS="

kyle_che · March 25, 2020, 5:46pm

thank you

system · April 22, 2020, 5:46pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.