No output to stdout/stderr from Docker image after upgrading to v8.15.x

Elastic Stack Beats
,
1

Filebeat has a command line option named --environment (in filebeat --help) or -environment (in the online command reference):

-environment
For logging purposes, specifies the environment that Filebeat is running in. This setting is used to select a default log output when no log output is configured. Supported values are: systemd, container, macos_service, and windows_service. If systemd or container is specified, Filebeat will log to stdout and stderr by default.

The official Dockerimage automatically runs filebeat with -environment container (through CMD ["-environment" "container"]) to enable output to stdout/stderr, and up until v8.14.3 this has worked flawlessly.

After upgrading to v8.15.0 or v8.15.1, this configuration option seems to no longer have any effect, and filebeat no longer prints any information at all unless -e is specified in addition.

This can be reproduced by running the official Docker images and comparing v8.14.3 with v8.15.1:

$ docker run docker.elastic.co/beats/filebeat:8.15.1 filebeat version
filebeat version 8.15.1 (amd64), libbeat 8.15.1 [88cc526a2d3e52dcbfa52c9dd25eb09ed95470e4 built 2024-09-02 08:36:21 +0000 UTC]

Version 8.15.1 never prints anything at all, but instead writes logs to logs/filebeat-<currentdate>.ndjson inside the Docker container:

$ time timeout 10 docker run docker.elastic.co/beats/filebeat:8.15.1

real    0m11.101s
user    0m0.010s
sys     0m0.005s

In v8.14.3 usually ~36 log lines will be printed to stderr in the same amount of time:

$ docker run docker.elastic.co/beats/filebeat:8.14.3 filebeat version
filebeat version 8.14.3 (amd64), libbeat 8.14.3 [71819961045386b23edc18455f1b54764292816c built 2024-07-08 22:05:44 +0000 UTC]

$ time timeout 10 docker run docker.elastic.co/beats/filebeat:8.14.3
{"log.level":"info","@timestamp":"2024-09-19T14:30:35.160Z","log.origin":"<trimmed-for-clarity>",service.name":"filebeat","ecs.version":"1.6.0"}
<removed 34 lines of standard filebeat output>
{"log.level":"info","@timestamp":"2024-09-19T14:30:45.000Z","log.origin":"<trimmed-for-clarity>","message":"filebeat stopped.","service.name":"filebeat","ecs.version":"1.6.0"}

real    0m10.087s
user    0m0.007s
sys     0m0.009s

It looks like running the Docker image with filebeat -e in addition to any other flags is a possible workaround, but the -environment/--environment seems to be broken at the moment. This causes issues for anyone running filebeat in Kubernetes or similar environments, especially for anyone using these logs for monitoring, debugging or auditing purposes.

(As a side note and a possible caution to anyone reading this, there's a known issue with connection failure recovery affecting all beats in v8.15.1, but the problem with -environment/--environment is present in both 8.15.0 and 8.15.1)

1 Like
2

Thanks for reporting this, I'm able to reproduce this and am investigating. If I do not reply in the next day or so please post an issue in the beats repository here GitHub · Where software is built

3

Issue created here:

1 Like
4

Thanks for confirming and reporting back with the issue number - much appreciated!

For anyone else who might be following this, seems like a fix is already on its way in logp: don't write to files by default if running in a container environment by mauri870 · Pull Request #236 · elastic/elastic-agent-libs · GitHub