Beats in docker: logging so filebeat picks up logs via "docker logs" API

sgreszcz · October 30, 2019, 1:32pm

I'm running several containers in docker and using filebeat and metricbeat to poll the Docker APIs to get metrics and logfiles.

However I can't see any of the filebeat or metricbeat logs (also running in containers) using "docker logs filebeat" or "docker logs metricbeat".

By default the beats logs in docker go to:
/var/log/<beatname>
and I tried to send the docker logs to syslog but it seems like it is not enabled in the docker container that Elastic uses to ship the beats. I see this error:

docker logs metricbeat

Exiting: error initializing logging: failed to build log output: failed to get a syslog writer: Unix syslog delivery error

Exiting: error initializing logging: failed to build log output: failed to get a syslog writer: Unix syslog delivery error

sgreszcz · November 12, 2019, 6:06pm

Seems also not to work with "logging.to_stderr" either:

Exiting: error loading config file: yaml: line 47: mapping values are not allowed in this context

Any ideas how to do proper logging for any of the beats inside the docker implementation to harvest the logs properly via the "docker logs" API?

philippkahr · November 12, 2019, 6:13pm

Hi,

can you show us your configuration? If you just start filebeat / metricbeat/ any-beat you see it with docker container logs ...

e.g.

$ docker run -d --name=filebeat docker.elastic.co/beats/filebeat:7.4.1                                                                                   
3d0c9f1a7325cdd905c508a1030703dc55c0e86ddc6cb2ad31b9e4ec5af0b4ca
$ docker container logs filebeat                                                                                                                            
2019-11-12T18:12:37.894Z        INFO    instance/beat.go:607    Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2019-11-12T18:12:37.897Z        INFO    instance/beat.go:615    Beat ID: 336fd8da-136e-4a21-b09c-ef60cfef44dc
2019-11-12T18:12:37.898Z        INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2019-11-12T18:12:37.899Z        INFO    [beat]  instance/beat.go:903    Beat info       {"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "336fd8da-136e-4a21-b09c-ef60cfef44dc"}}}
2019-11-12T18:12:37.899Z        INFO    [beat]  instance/beat.go:912    Build info      {"system_info": {"build": {"commit": "12ee6cd05c1bfdc69721ddab1f473417b1514576", "libbeat": "7.4.1", "time": "2019-10-22T16:22:37.000Z", "version": "7.4.1"}}}
2019-11-12T18:12:37.899Z        INFO    [beat]  instance/beat.go:915    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.12.9"}}}
2019-11-12T18:12:37.900Z        INFO    add_cloud_metadata/add_cloud_metadata.go:87     add_cloud_metadata: hosting provider type not detected.
2019-11-12T18:12:37.900Z        INFO    [beat]  instance/beat.go:919    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-11-12T18:10:58Z","containerized":true,"name":"3d0c9f1a7325","ip":["127.0.0.1/8","172.17.0.2/16"],"kernel_version":"4.9.184-linuxkit","mac":["02:42:ac:11:00:02"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2019-11-12T18:12:37.901Z        INFO    [beat]  instance/beat.go:948    Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2019-11-12T18:12:36.630Z"}}}
2019-11-12T18:12:37.901Z        INFO    instance/beat.go:292    Setup Beat: filebeat; Version: 7.4.1
2019-11-12T18:12:37.901Z        INFO    [index-management]      idxmgmt/std.go:178      Set output.elasticsearch.index to 'filebeat-7.4.1' as ILM is enabled.
2019-11-12T18:12:37.901Z        INFO    elasticsearch/client.go:170     Elasticsearch url: http://elasticsearch:9200
2019-11-12T18:12:37.902Z        INFO    [publisher]     pipeline/module.go:97   Beat name: 3d0c9f1a7325
2019-11-12T18:12:37.903Z        INFO    [monitoring]    log/log.go:118  Starting metrics logging every 30s
2019-11-12T18:12:37.903Z        INFO    instance/beat.go:422    filebeat start running.
2019-11-12T18:12:37.903Z        INFO    registrar/migrate.go:104        No registry home found. Create: /usr/share/filebeat/data/registry/filebeat
2019-11-12T18:12:37.903Z        INFO    registrar/migrate.go:112        Initialize registry meta file
2019-11-12T18:12:37.905Z        INFO    registrar/registrar.go:108      No registry file found under: /usr/share/filebeat/data/registry/filebeat/data.json. Creating a new registry file.
2019-11-12T18:12:37.907Z        INFO    registrar/registrar.go:145      Loading registrar data from /usr/share/filebeat/data/registry/filebeat/data.json
2019-11-12T18:12:37.907Z        INFO    registrar/registrar.go:152      States Loaded from registrar: 0
2019-11-12T18:12:37.907Z        INFO    crawler/crawler.go:72   Loading Inputs: 0
2019-11-12T18:12:37.907Z        INFO    crawler/crawler.go:106  Loading and starting Inputs completed. Enabled inputs: 0
2019-11-12T18:12:37.907Z        INFO    cfgfile/reload.go:171   Config reloader started
2019-11-12T18:12:37.908Z        INFO    cfgfile/reload.go:226   Loading of config files completed.

sgreszcz · November 13, 2019, 3:07pm

root@nfl-aer-001:~# docker ps
CONTAINER ID        IMAGE                                    COMMAND                  CREATED             STATUS              PORTS               NAMES
4c6f34af078d        docker.elastic.co/beats/filebeat:7.4.2   "/usr/local/bin/dock…"   21 hours ago        Up 21 hours                             filebeat-netflow
c17fe240abbb        docker.elastic.co/beats/filebeat:7.4.2   "/usr/local/bin/dock…"   2 days ago          Up 2 days                               filebeat
root@nfl-aer-001:~# docker container logs filebeat-netflow
root@nfl-aer-001:~# docker logs filebeat
root@nfl-aer-001:~# docker logs filebeat-netflow 
root@nfl-aer-001:~#

As I said, the only options that I could get working was this config which maps the logs to a file which is exposed to the docker host using volumes:

#==============================   Logging  =====================================
logging.level: info
#  logging.to_stderr: true
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7

When I tried the other options in the docs: https://www.elastic.co/guide/en/beats/filebeat/current/configuration-logging.html

I couldn't get the options "logging.to_stderr" and "logging.to_syslog" to work. Both options threw errors.

system · December 11, 2019, 5:11pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Docker+filebeat cannot get logs Beats filebeat	5	2244	July 18, 2018
Metricbeat in container - missing docker logs Beats metricbeat	7	1553	October 18, 2018
Filebeat not writing docker containers logs into Elasticsearch Beats filebeat	8	2668	March 12, 2019
ELK no read logs from docker containers Beats filebeat	2	405	June 25, 2019
Docker Logs not sent to ES Beats filebeat	5	531	February 25, 2020

Beats in docker: logging so filebeat picks up logs via "docker logs" API

Related topics