Filebeat in Docker: logging.to_files is ignored

[bluecoffee]

Hi!
I'm running Filebeat as a Docker container. I bind-mount a config file "filebeat.yml" that defines logging to files, but the logs still end up in the console, not in a file.

The config file is being read (other config values are picked up fine, e.g. "logging.json: true"). The syntax seems to be fine (filebeat test config returns OK), and the values seem to be fine too (filebeat export config output looks fine).

When I run those two commands (filebeat test config and filebeat export config), I do this by entering into the container (docker exec -it mycont /bin/bash) and running the filebeat command from command line in there. Funny enough, those two tests write logs at the locations I specify, so the locations are picked up fine, and the permissions seems to be fine:

# No logfile yet:
bash-4.2$ ls -lpah ./logs
total 4.0K
drwxrwx---. 2 root filebeat    6 Aug 17 22:30 ./
drwxr-x---. 8 root filebeat 4.0K Aug 17 22:30 ../

# Test config:
bash-4.2$ ./filebeat test config
Config OK

# Now a logfile is there:
bash-4.2$ ls -lpah ./logs
total 8.0K
drwxrwx---. 2 root     filebeat   30 Sep  4 13:48 ./
drwxr-x---. 8 root     filebeat 4.0K Aug 17 22:30 ../
-rw-r--r--. 1 filebeat filebeat 3.6K Sep  4 13:48 this_name_please

# Export config:
bash-4.2$ ./filebeat export config
filebeat:
 [...]

# Now, a second logfile was created:
bash-4.2$ ls -lpah ./logs
total 12K
drwxrwx---. 2 root     filebeat   56 Sep  4 13:48 ./
drwxr-x---. 8 root     filebeat 4.0K Aug 17 22:30 ../
-rw-r--r--. 1 filebeat filebeat  564 Sep  4 13:48 this_name_please
-rw-r--r--. 1 filebeat filebeat 3.6K Sep  4 13:48 this_name_please.1

So the difference seems to be whether filebeat is run inside the container manually by user "filebeat", or whether it is run by being launched through docker-compose up.

Any help would be appreciated!
Thanks!

[bluecoffee]

Some additional info:

Excerpt from my filebeat.yml:

logging.level: debug
logging.to_files: true
logging.files:
  path: /usr/share/filebeat/logs/
  name: fileby
  keepfiles: 7
  permissions: 0664

logging.json: true
logging.to_syslog: false
logging.to_eventlog: false

output.console:
  enabled: false

logging.metrics.enabled: true

docker-compose:

services:

  filebeat:
    image: docker.elastic.co/beats/filebeat:6.4.0
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml

Directories inside the container:

filebeat
bash-4.2$ ls -lpah
total 33M
drwxr-x---.  8 root     filebeat 4.0K Aug 17 22:30 ./
drwxr-xr-x. 53 root     root     4.0K Aug 17 22:30 ../
-rw-r-----.  1 root     filebeat   41 Aug 17 22:20 .build_hash.txt
-rw-r-----.  1 root     filebeat  14K Aug 17 22:16 LICENSE.txt
-rw-r-----.  1 root     filebeat 146K Aug 17 22:16 NOTICE.txt
-rw-r-----.  1 root     filebeat  802 Aug 17 22:20 README.md
drwxrwx---.  2 root     filebeat   39 Sep  4 13:47 data/
-rw-r-----.  1 root     filebeat  67K Aug 17 22:19 fields.yml
-rwxr-x---.  1 root     filebeat  33M Aug 17 22:20 filebeat
-rw-r-----.  1 root     filebeat  64K Aug 17 22:19 filebeat.reference.yml
-rw-r--r--.  1 filebeat filebeat  845 Sep  4 13:46 filebeat.yml
drwxr-x---.  4 root     filebeat   24 Aug 17 22:19 kibana/
drwxrwx---.  2 root     filebeat    6 Aug 17 22:30 logs/
drwxr-x---. 18 root     filebeat 4.0K Aug 17 22:19 module/
drwxrwx---.  2 root     filebeat 4.0K Aug 17 22:19 modules.d/
drwxr-x---.  2 root     filebeat   25 Aug 17 22:29 prospectors.d/
bash-4.2$ ls -lpah ./logs
total 4.0K
drwxrwx---. 2 root filebeat    6 Aug 17 22:30 ./
drwxr-x---. 8 root filebeat 4.0K Aug 17 22:30 ../
bash-4.2$ 

The filebeat.yml, exported by filebeat:

bash-4.2$ ./filebeat export config
filebeat:
  prospectors:
  - enabled: true
[...]
logging:
  files:
    keepfiles: 7
    name: this_name_please
    path: /usr/share/filebeat/logs/
    permissions: 436
  json: true
  level: debug
  metrics:
    enabled: true
  to_eventlog: false
  to_files: true
  to_syslog: false
output:
  console:
    enabled: false
  logstash:
    enabled: true
    hosts:
    - xyz:5044
    timeout: 240s
path:
  config: /usr/share/filebeat
  data: /usr/share/filebeat/data
  home: /usr/share/filebeat
  logs: /usr/share/filebeat/logs
bash-4.2$ 

[kvch (Noémi Ványi)]

Are you sure you mounted correctly the config file? logging.files.permissions is different in your filebeat.yml and in the exported config.

[bluecoffee]

Hi,
good point. I checked it again, and you are right, it is "permissions: 436" (that should be r---wxrw-, right?) in the exported, and "0664" (rw-rw-r--) in the .yml file.

But the other values (e.g. logging.level, logging.metrics.enabled, logging.files.path) are picked up fine, so I do not have an explanation.

I think I have copied this from some examples, I have not really given those permissions much thought as of yet. But as they apply to the permissions of the resulting log files, I did not think that they'd cause my problem.

(Funny enough, the created log file in the container has rw-r--r-- (0644), which is neither the given value, nor the exported value, nor the default value 0600 (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html). But I have played a bit with the config since I asked the question, but I can start up a clean example if this is relevant.)

[kvch (Noémi Ványi)]

So you are sure now that Filebeat has the proper permissions to write to the log file you specified in the config?

[bluecoffee]

Good morning!
I have set the permissions to default now. The volume where the logs should be written in is not mounted from the host, but the default directory inside the container. So it should work, as I have not touched anything.

The docs say about the default: "0600: give read and write access to the file owner, and no access to all others." (https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html)

The directory (inside the container) is by default:

 [root@1398e1d608d1 filebeat]# ls -lpah logs
total 4.0K
drwxrwx---. 2 root filebeat    6 Aug 17 22:30 ./
drwxr-x---. 8 root filebeat 4.0K Aug 17 22:30 ../

So root and filebeat may both write into it. I don't see any reason why permissions should be keeping the logs from being written.

When I run ./filebeat export config it does write a log, and it has 0600 permissions, so permissions is not the issue.

Any help is very welcome

(Next week I'll be travelling and will not be able to try out things very quickly, but I'll try my best)

[CpuID (Nathan Sullivan)]

I can reproduce this locally myself (on 6.4.1), also seems to be ignoring logging.level which is weird. I have the log level set to warning below, but I still get info logs. In addition, logs go to stdout instead of the to_files destination.

[nathan@ns-desktop-ub filebeat (master)]$ docker logs filebeat_filebeat_1 2>&1 | head -n 5
{"level":"info","timestamp":"2018-09-25T21:28:08.626Z","caller":"instance/beat.go:544","message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]"}
{"level":"info","timestamp":"2018-09-25T21:28:08.636Z","caller":"instance/beat.go:551","message":"Beat UUID: c5bcb721-9140-4e7f-91c5-aedbba4fd8ec"}
{"level":"info","timestamp":"2018-09-25T21:28:08.636Z","logger":"seccomp","caller":"seccomp/seccomp.go:116","message":"Syscall filter successfully installed"}
{"level":"info","timestamp":"2018-09-25T21:28:08.636Z","logger":"beat","caller":"instance/beat.go:768","message":"Beat info","system_info":{"beat":{"path":{"config":"/usr/share/filebeat","data":"/usr/share/filebeat/data","home":"/usr/share/filebeat","logs":"/usr/share/filebeat/logs"},"type":"filebeat","uuid":"c5bcb721-9140-4e7f-91c5-aedbba4fd8ec"}}}
{"level":"info","timestamp":"2018-09-25T21:28:08.636Z","logger":"beat","caller":"instance/beat.go:777","message":"Build info","system_info":{"build":{"commit":"37b5f2d2a20f2734b2373a454b4b4cbb2627e841","libbeat":"6.4.1","time":"2018-09-13T21:25:47.000Z","version":"6.4.1"}}}

sh-4.2# cat /usr/share/filebeat/filebeat.yml
filebeat.shutdown_timeout: 5s


filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            regexp:
              docker.container.name: ".*"
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              processors:
                - add_docker_metadata: ~
                - decode_json_fields:
                    fields: ["message"]
                    target: "message_json"

              tags: ["filebeat-ecs"]

output:
  redis:
    hosts: ["redis:6379"]
    key: "logstash-dev"
    datatype: list
    # We used 5 logstash-agent workers in the past at the edge
    worker: 3
    # We used 500 on our logstash-agent batches
    bulk_max_size: 2048

logging.level: warning
logging.json: true
logging.to_files: true
logging.files:
    path: /var/log/filebeat
    name: filebeat

sh-4.2# ls -la /var/log/filebeat/
total 8
drwxr-xr-x 2 root root 4096 Sep 25 21:28 .
drwxr-xr-x 1 root root 4096 Sep 25 21:28 ..

[bluecoffee]

Hi! Thanks for replying. Interesting to hear that you can reproduce it, so it is (probably) not a problem with my settings.

Has anyone been successful in solving this yet? I'm still looking for a solution!

Best,
Merret

[ruflin]

I haven't followed the full thread but I think the reason is that filebeat inside the container is run with the -e flag and if I remember correctly that overwrites the logging flag. Changing the CMD part should change this behaviour: https://github.com/elastic/beats-docker/blob/master/templates/Dockerfile.j2#L62

[bluecoffee]

Hi! Thanks for the suggestion! I'll try to see if that does the trick.

But would there be a way to switch off that flag withouth having to rebuild the image? I like using the original images and not build custom ones, especially if it is for such a small change.

[ruflin]

I think you can take the existing docker image and just overwrite CMD with what you need in your own. It will take the last one defined.

[bluecoffee]

Hi!
I realized I had not replied to this yet - sorry for this!

Just to tell that your advice worked perfectly, thanks a lot .

I overwrote the CMD of the filebeat image by just putting an empty string into my docker compose:

version: "3.3"
services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:6.4.0
    volumes:
      - ./logs/:/usr/share/filebeat/logs/
     # [...]
    command: "" # to overwrite the -e that disables logging to file!

And it logged into file like a charm.

(Here are the possible flags / options to pass in that CMD / command: https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html#global-flags)