# Filebeat in Docker: logging.to\_files is ignored

**URL:** <https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 4, 2018, 1:59pm UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224 "2018-09-04T13:59:04Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![bluecoffee](https://avatars.discourse-cdn.com/v4/letter/b/2acd7d/32.png) [@bluecoffee](https://discuss.elastic.co/u/bluecoffee)\
**Post date:** [September 4, 2018, 1:59pm UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/1 "2018-09-04T13:59:04Z")

</div>

Hi!  
I'm running Filebeat as a Docker container. I bind-mount a config file "filebeat.yml" that defines logging to files, but the logs still end up in the console, not in a file.

The config file is being read (other config values are picked up fine, e.g. "logging.json: true"). The syntax seems to be fine (filebeat test config returns OK), and the values seem to be fine too (filebeat export config output looks fine).

When I run those two commands (filebeat test config and filebeat export config), I do this by entering into the container (docker exec -it mycont /bin/bash) and running the filebeat command from command line in there. Funny enough, those two tests write logs at the locations I specify, so the locations are picked up fine, and the permissions seems to be fine:

```auto
# No logfile yet:
bash-4.2$ ls -lpah ./logs
total 4.0K
drwxrwx---. 2 root filebeat 6 Aug 17 22:30 ./
drwxr-x---. 8 root filebeat 4.0K Aug 17 22:30 ../

# Test config:
bash-4.2$ ./filebeat test config
Config OK

# Now a logfile is there:
bash-4.2$ ls -lpah ./logs
total 8.0K
drwxrwx---. 2 root filebeat 30 Sep 4 13:48 ./
drwxr-x---. 8 root filebeat 4.0K Aug 17 22:30 ../
-rw-r--r--. 1 filebeat filebeat 3.6K Sep 4 13:48 this_name_please

# Export config:
bash-4.2$ ./filebeat export config
filebeat:
 [...]

# Now, a second logfile was created:
bash-4.2$ ls -lpah ./logs
total 12K
drwxrwx---. 2 root filebeat 56 Sep 4 13:48 ./
drwxr-x---. 8 root filebeat 4.0K Aug 17 22:30 ../
-rw-r--r--. 1 filebeat filebeat 564 Sep 4 13:48 this_name_please
-rw-r--r--. 1 filebeat filebeat 3.6K Sep 4 13:48 this_name_please.1

```

So the difference seems to be whether filebeat is run inside the container manually by user "filebeat", or whether it is run by being launched through docker-compose up.

Any help would be appreciated!  
Thanks!

---

<div class="post-metadata">

**Author:** ![bluecoffee](https://avatars.discourse-cdn.com/v4/letter/b/2acd7d/32.png) [@bluecoffee](https://discuss.elastic.co/u/bluecoffee)\
**Post date:** [September 4, 2018, 2:00pm UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/2 "2018-09-04T14:00:14Z")

</div>

Some additional info:

Excerpt from my filebeat.yml:

```auto
logging.level: debug
logging.to_files: true
logging.files:
  path: /usr/share/filebeat/logs/
  name: fileby
  keepfiles: 7
  permissions: 0664

logging.json: true
logging.to_syslog: false
logging.to_eventlog: false

output.console:
  enabled: false

logging.metrics.enabled: true

```

docker-compose:

```auto
services:

  filebeat:
    image: docker.elastic.co/beats/filebeat:6.4.0
    volumes:
      - ./filebeat.yml:/usr/share/filebeat/filebeat.yml

```

Directories inside the container:

```auto
filebeat
bash-4.2$ ls -lpah
total 33M
drwxr-x---. 8 root filebeat 4.0K Aug 17 22:30 ./
drwxr-xr-x. 53 root root 4.0K Aug 17 22:30 ../
-rw-r-----. 1 root filebeat 41 Aug 17 22:20 .build_hash.txt
-rw-r-----. 1 root filebeat 14K Aug 17 22:16 LICENSE.txt
-rw-r-----. 1 root filebeat 146K Aug 17 22:16 NOTICE.txt
-rw-r-----. 1 root filebeat 802 Aug 17 22:20 README.md
drwxrwx---. 2 root filebeat 39 Sep 4 13:47 data/
-rw-r-----. 1 root filebeat 67K Aug 17 22:19 fields.yml
-rwxr-x---. 1 root filebeat 33M Aug 17 22:20 filebeat
-rw-r-----. 1 root filebeat 64K Aug 17 22:19 filebeat.reference.yml
-rw-r--r--. 1 filebeat filebeat 845 Sep 4 13:46 filebeat.yml
drwxr-x---. 4 root filebeat 24 Aug 17 22:19 kibana/
drwxrwx---. 2 root filebeat 6 Aug 17 22:30 logs/
drwxr-x---. 18 root filebeat 4.0K Aug 17 22:19 module/
drwxrwx---. 2 root filebeat 4.0K Aug 17 22:19 modules.d/
drwxr-x---. 2 root filebeat 25 Aug 17 22:29 prospectors.d/
bash-4.2$ ls -lpah ./logs
total 4.0K
drwxrwx---. 2 root filebeat 6 Aug 17 22:30 ./
drwxr-x---. 8 root filebeat 4.0K Aug 17 22:30 ../
bash-4.2$ 

```

The filebeat.yml, exported by filebeat:

```auto
bash-4.2$ ./filebeat export config
filebeat:
  prospectors:
  - enabled: true
[...]
logging:
  files:
    keepfiles: 7
    name: this_name_please
    path: /usr/share/filebeat/logs/
    permissions: 436
  json: true
  level: debug
  metrics:
    enabled: true
  to_eventlog: false
  to_files: true
  to_syslog: false
output:
  console:
    enabled: false
  logstash:
    enabled: true
    hosts:
    - xyz:5044
    timeout: 240s
path:
  config: /usr/share/filebeat
  data: /usr/share/filebeat/data
  home: /usr/share/filebeat
  logs: /usr/share/filebeat/logs
bash-4.2$ 

```

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 5, 2018, 2:34pm UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/3 "2018-09-05T14:34:39Z")

</div>

Are you sure you mounted correctly the config file? `logging.files.permissions` is different in your filebeat.yml and in the exported config.

---

<div class="post-metadata">

**Author:** ![bluecoffee](https://avatars.discourse-cdn.com/v4/letter/b/2acd7d/32.png) [@bluecoffee](https://discuss.elastic.co/u/bluecoffee)\
**Post date:** [September 6, 2018, 9:46am UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/4 "2018-09-06T09:46:25Z")

</div>

Hi,  
good point. I checked it again, and you are right, it is "permissions: 436" (that should be r---wxrw-, right?) in the exported, and "0664" (rw-rw-r--) in the .yml file.

But the other values (e.g. logging.level, logging.metrics.enabled, logging.files.path) are picked up fine, so I do not have an explanation.

I think I have copied this from some examples, I have not really given those permissions much thought as of yet. But as they apply to the permissions of the resulting log files, I did not think that they'd cause my problem.

(Funny enough, the created log file in the container has rw-r--r-- (0644), which is neither the given value, nor the exported value, nor the default value 0600 ([https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html)). But I have played a bit with the config since I asked the question, but I can start up a clean example if this is relevant.)

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 6, 2018, 3:18pm UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/5 "2018-09-06T15:18:18Z")

</div>

So you are sure now that Filebeat has the proper permissions to write to the log file you specified in the config?

---

<div class="post-metadata">

**Author:** ![bluecoffee](https://avatars.discourse-cdn.com/v4/letter/b/2acd7d/32.png) [@bluecoffee](https://discuss.elastic.co/u/bluecoffee)\
**Post date:** [September 7, 2018, 9:14am UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/6 "2018-09-07T09:14:37Z")

</div>

Good morning!  
I have set the permissions to default now. The volume where the logs should be written in is not mounted from the host, but the default directory inside the container. So it should work, as I have not touched anything.

The docs say about the default: _"0600: give read and write access to the file owner, and no access to all others."_ ([https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html))

The directory (inside the container) is by default:

```auto
 [root@1398e1d608d1 filebeat]# ls -lpah logs
total 4.0K
drwxrwx---. 2 root filebeat 6 Aug 17 22:30 ./
drwxr-x---. 8 root filebeat 4.0K Aug 17 22:30 ../

```

So root and filebeat may both write into it. I don't see any reason why permissions should be keeping the logs from being written.

When I run `./filebeat export config` it does write a log, and it has 0600 permissions, so permissions is not the issue.

Any help is very welcome 🙂

(Next week I'll be travelling and will not be able to try out things very quickly, but I'll try my best)

---

<div class="post-metadata">

**Author:** ![CpuID](https://avatars.discourse-cdn.com/v4/letter/c/8baadc/32.png) [@CpuID](https://discuss.elastic.co/u/CpuID)\
**Post date:** [September 25, 2018, 9:30pm UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/7 "2018-09-25T21:30:03Z")

</div>

I can reproduce this locally myself (on 6.4.1), also seems to be ignoring `logging.level` which is weird. I have the log level set to warning below, but I still get info logs. In addition, logs go to stdout instead of the to\_files destination.

```auto
[nathan@ns-desktop-ub filebeat (master)]$ docker logs filebeat_filebeat_1 2>&1 | head -n 5
{"level":"info","timestamp":"2018-09-25T21:28:08.626Z","caller":"instance/beat.go:544","message":"Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]"}
{"level":"info","timestamp":"2018-09-25T21:28:08.636Z","caller":"instance/beat.go:551","message":"Beat UUID: c5bcb721-9140-4e7f-91c5-aedbba4fd8ec"}
{"level":"info","timestamp":"2018-09-25T21:28:08.636Z","logger":"seccomp","caller":"seccomp/seccomp.go:116","message":"Syscall filter successfully installed"}
{"level":"info","timestamp":"2018-09-25T21:28:08.636Z","logger":"beat","caller":"instance/beat.go:768","message":"Beat info","system_info":{"beat":{"path":{"config":"/usr/share/filebeat","data":"/usr/share/filebeat/data","home":"/usr/share/filebeat","logs":"/usr/share/filebeat/logs"},"type":"filebeat","uuid":"c5bcb721-9140-4e7f-91c5-aedbba4fd8ec"}}}
{"level":"info","timestamp":"2018-09-25T21:28:08.636Z","logger":"beat","caller":"instance/beat.go:777","message":"Build info","system_info":{"build":{"commit":"37b5f2d2a20f2734b2373a454b4b4cbb2627e841","libbeat":"6.4.1","time":"2018-09-13T21:25:47.000Z","version":"6.4.1"}}}

```

```auto
sh-4.2# cat /usr/share/filebeat/filebeat.yml
filebeat.shutdown_timeout: 5s

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            regexp:
              docker.container.name: ".*"
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              processors:
                - add_docker_metadata: ~
                - decode_json_fields:
                    fields: ["message"]
                    target: "message_json"

              tags: ["filebeat-ecs"]

output:
  redis:
    hosts: ["redis:6379"]
    key: "logstash-dev"
    datatype: list
    # We used 5 logstash-agent workers in the past at the edge
    worker: 3
    # We used 500 on our logstash-agent batches
    bulk_max_size: 2048

logging.level: warning
logging.json: true
logging.to_files: true
logging.files:
    path: /var/log/filebeat
    name: filebeat

```

```auto
sh-4.2# ls -la /var/log/filebeat/
total 8
drwxr-xr-x 2 root root 4096 Sep 25 21:28 .
drwxr-xr-x 1 root root 4096 Sep 25 21:28 ..

```

---

<div class="post-metadata">

**Author:** ![bluecoffee](https://avatars.discourse-cdn.com/v4/letter/b/2acd7d/32.png) [@bluecoffee](https://discuss.elastic.co/u/bluecoffee)\
**Post date:** [September 26, 2018, 3:51pm UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/8 "2018-09-26T15:51:56Z")

</div>

Hi! Thanks for replying. Interesting to hear that you can reproduce it, so it is (probably) not a problem with my settings.

Has anyone been successful in solving this yet? I'm still looking for a solution!

Best,  
Merret

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [September 26, 2018, 9:25pm UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/9 "2018-09-26T21:25:54Z")

</div>

I haven't followed the full thread but I think the reason is that filebeat inside the container is run with the `-e` flag and if I remember correctly that overwrites the logging flag. Changing the `CMD` part should change this behaviour: [https://github.com/elastic/beats-docker/blob/master/templates/Dockerfile.j2#L62](https://github.com/elastic/beats-docker/blob/master/templates/Dockerfile.j2#L62)

---

<div class="post-metadata">

**Author:** ![bluecoffee](https://avatars.discourse-cdn.com/v4/letter/b/2acd7d/32.png) [@bluecoffee](https://discuss.elastic.co/u/bluecoffee)\
**Post date:** [September 27, 2018, 12:54pm UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/10 "2018-09-27T12:54:58Z")

</div>

Hi! Thanks for the suggestion! I'll try to see if that does the trick.

But would there be a way to switch off that flag withouth having to rebuild the image? I like using the original images and not build custom ones, especially if it is for such a small change.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [October 1, 2018, 6:56am UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/11 "2018-10-01T06:56:11Z")

</div>

I think you can take the existing docker image and just overwrite `CMD` with what you need in your own. It will take the last one defined.

---

<div class="post-metadata">

**Author:** ![bluecoffee](https://avatars.discourse-cdn.com/v4/letter/b/2acd7d/32.png) [@bluecoffee](https://discuss.elastic.co/u/bluecoffee)\
**Post date:** [October 19, 2018, 2:22pm UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/12 "2018-10-19T14:22:55Z")

</div>

Hi!  
I realized I had not replied to this yet - sorry for this!

Just to tell that your advice worked perfectly, thanks a lot 👍👏.

I overwrote the CMD of the filebeat image by just putting an empty string into my docker compose:

```auto
version: "3.3"
services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:6.4.0
    volumes:
      - ./logs/:/usr/share/filebeat/logs/
     # [...]
    command: "" # to overwrite the -e that disables logging to file!

```

And it logged into file like a charm.

(Here are the possible flags / options to pass in that CMD / command: [https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html#global-flags](https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html#global-flags))

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 16, 2018, 2:22pm UTC](https://discuss.elastic.co/t/filebeat-in-docker-logging-to-files-is-ignored/147224/13 "2018-11-16T14:22:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.