Filebeat not writing docker containers logs into Elasticsearch


(Alejandro Kapit) #1

Hi,
I have several docker containers (using docker-compose) and I'm trying to use Filebeat (from docker) to push my containers logs to elasticsearch with no success.

This is my filebeat.yml:

filebeat.inputs:
  - type: docker
containers.ids: '*'

output.elasticsearch:
  hosts: ["outflink-elastic:9200"]
  protocol: "http"

This is its config block in docker-compose.yml:

filebeat:
  image: akapit/outflink_platform:outflink-filebeat
  container_name: outflink-filebeat
  user: root
  volumes:
    - $HOME/filebeat_data:/usr/share/filebeat/data
    - /var/lib/docker/containers:/usr/share/filebeat/dockerlogs:ro
    - /var/run/docker.sock:/var/run/docker.sock
  depends_on:
    - elasticsearch

This is my Filebeat Dockerfile:

FROM docker.elastic.co/beats/filebeat:6.5.4
COPY filebeat.yml /usr/share/filebeat/filebeat.yml
USER root
RUN chown root:filebeat /usr/share/filebeat/filebeat.yml
RUN mkdir /usr/share/filebeat/dockerlogs

USER filebeat

This is the filebeat console output:

2019-01-24T10:09:42.348Z	INFO	instance/beat.go:592	Home path: [/usr/share/filebeat] Config     path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2019-01-24T10:09:42.354Z	INFO	instance/beat.go:599	Beat UUID: 7e44573c-940f-4ea6-94ab-2673f9962e6c
2019-01-24T10:09:42.354Z	INFO	[seccomp]	seccomp/seccomp.go:116	Syscall filter successfully installed
2019-01-24T10:09:42.354Z	INFO	[beat]	instance/beat.go:825	Beat info	{"system_info": {"beat": {"path": {"config": "/usr/share/filebeat", "data": "/usr/share/filebeat/data", "home": "/usr/share/filebeat", "logs": "/usr/share/filebeat/logs"}, "type": "filebeat", "uuid": "7e44573c-940f-4ea6-94ab-2673f9962e6c"}}}
2019-01-24T10:09:42.356Z	INFO	[beat]	instance/beat.go:834	Build info	{"system_info": {"build": {"commit": "bd8922f1c7e93d12b07e0b3f7d349e17107f7826", "libbeat": "6.5.4", "time": "2018-12-17T20:22:29.000Z", "version": "6.5.4"}}}
2019-01-24T10:09:42.356Z	INFO	[beat]	instance/beat.go:837	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.10.6"}}}
2019-01-24T10:09:42.363Z	INFO	[beat]	instance/beat.go:841	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-01-24T04:34:57Z","containerized":true,"name":"92a6c0af3b08","ip":["127.0.0.1/8","192.168.96.7/20"],"kernel_version":"4.9.125-linuxkit","mac":["02:42:c0:a8:60:07"],"os":{"family":"redhat","platform":"centos","name":"CentOS Linux","version":"7 (Core)","major":7,"minor":6,"patch":1810,"codename":"Core"},"timezone":"UTC","timezone_offset_sec":0}}}
2019-01-24T10:09:42.364Z	INFO	[beat]	instance/beat.go:870	Process info	{"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/usr/share/filebeat", "exe": "/usr/share/filebeat/filebeat", "name": "filebeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter"}, "start_time": "2019-01-24T10:09:39.920Z"}}}
2019-01-24T10:09:42.365Z	INFO	instance/beat.go:278	Setup Beat: filebeat; Version: 6.5.4
2019-01-24T10:09:42.366Z	INFO	elasticsearch/client.go:163	Elasticsearch url: http://outflink-elastic:9200
2019-01-24T10:09:42.367Z	INFO	[publisher]	pipeline/module.go:110	Beat name: 92a6c0af3b08
2019-01-24T10:09:42.442Z	INFO	instance/beat.go:400	filebeat start running.
2019-01-24T10:09:42.443Z	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2019-01-24T10:09:42.448Z	INFO	registrar/registrar.go:134	Loading registrar data from /usr/share/filebeat/data/registry
2019-01-24T10:09:42.451Z	INFO	registrar/registrar.go:141	States Loaded from registrar: 36
2019-01-24T10:09:42.452Z	INFO	crawler/crawler.go:72	Loading Inputs: 1
2019-01-24T10:09:42.630Z	INFO	log/input.go:138	Configured paths: [/var/lib/docker/containers/*/*.log]
2019-01-24T10:09:42.630Z	INFO	input/input.go:114	Starting input of type: docker; ID: 8783912846562670036
2019-01-24T10:09:42.630Z	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2019-01-24T10:10:12.486Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":260,"time":{"ms":260}},"total":{"ticks":330,"time":{"ms":330},"value":330},"user":{"ticks":70,"time":{"ms":70}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":5},"info":{"ephemeral_id":"cab0928f-44f4-4838-ab5c-d0ab24c8818d","uptime":{"ms":30310}},"memstats":{"gc_next":4194304,"memory_alloc":1821080,"memory_total":4872672,"rss":9252864}},"filebeat":{"events":{"added":56,"done":56},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0,"filtered":56,"total":56}}},"registrar":{"states":{"cleanup":28,"current":8,"update":56},"writes":{"success":56,"total":56}},"system":{"cpu":{"cores":4},"load":{"1":3.25,"15":1.42,"5":1.42,"norm":{"1":0.8125,"15":0.355,"5":0.355}}}}}}

As you can see, it doesn't look that there is an error, however nothing gets written in elastic.

I'd appreciate any help,

Thanks!


(Toby McLaughlin) #2

Hi,

Have you declared a network in docker-compose.yml? Could you post the whole file?

Thanks.


(Carolina Baldino) #3

Hi,

I think you are indenting wrong the yml. It should be:

filebeat.inputs:
  - type: docker
    containers.ids: '*'
...

Regards,


(Alejandro Kapit) #4

I've tried that also,... and I don't see any new index.
Also no errors in filebeat log.

This is my config:

filebeat.inputs:


  - type: docker
    containers.ids: '*'

    enabled: true

    json.message_key: log
    json.keys_under_root: true


  filebeat.config.modules:
  # Glob pattern for configuration loading
    path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
    reload.enabled: false

  # Period on which files under path should be checked for changes
    #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false


#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]

This is the log:

2019-02-12T10:35:40.643Z	INFO	[seccomp]	seccomp/seccomp.go:116	Syscall filter successfully installed
2019-02-12T10:35:40.643Z	INFO	[beat]	instance/beat.go:936	Beat info	{"system_info": {"beat": {"path": {"config": "/home/outflink/filebeat-6.6.0-linux-x86_64", "data": "/home/outflink/filebeat-6.6.0-linux-x86_64/data", "home": "/home/outflink/filebeat-6.6.0-linux-x86_64", "logs": "/home/outflink/filebeat-6.6.0-linux-x86_64/logs"}, "type": "filebeat", "uuid": "fb60548b-3b65-40c8-8798-0e22acb3fae3"}}}
2019-02-12T10:35:40.643Z	INFO	[beat]	instance/beat.go:945	Build info	{"system_info": {"build": {"commit": "2c385a0764bdc537b6dc078a1d9bf11bb6d7bd95", "libbeat": "6.6.0", "time": "2019-01-24T10:29:55.000Z", "version": "6.6.0"}}}
2019-02-12T10:35:40.643Z	INFO	[beat]	instance/beat.go:948	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.10.8"}}}
2019-02-12T10:35:40.645Z	INFO	[beat]	instance/beat.go:952	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-01-10T10:35:39Z","containerized":false,"name":"localhost","ip":["127.0.0.1/8","139.162.245.198/24","172.17.0.1/16","192.168.144.1/20"],"kernel_version":"4.18.16-x86_64-linode118","mac":["ee:6f:7e:60:16:75","f2:3c:91:5a:0f:e0","02:42:27:0d:79:89","02:42:c4:22:87:f4","6a:59:23:ab:b6:87","82:3e:ea:9a:1b:f2","72:27:43:2b:5d:76","f2:7e:58:bc:68:af","e6:64:b2:99:24:c3","16:0e:49:76:1d:1f","96:c7:89:c6:69:2a","d2:49:67:1b:a0:b2","22:41:d8:dc:f2:f7"],"os":{"family":"debian","platform":"ubuntu","name":"Ubuntu","version":"16.04.5 LTS (Xenial Xerus)","major":16,"minor":4,"patch":5,"codename":"xenial"},"timezone":"UTC","timezone_offset_sec":0,"id":"07c3b878e09b682e46a477335b1ac472"}}}
2019-02-12T10:35:40.645Z	INFO	[beat]	instance/beat.go:981	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/home/outflink/filebeat-6.6.0-linux-x86_64", "exe": "/home/outflink/filebeat-6.6.0-linux-x86_64/filebeat", "name": "filebeat", "pid": 5984, "ppid": 2545, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2019-02-12T10:35:40.570Z"}}}
2019-02-12T10:35:40.646Z	INFO	instance/beat.go:281	Setup Beat: filebeat; Version: 6.6.0
2019-02-12T10:35:43.656Z	INFO	add_cloud_metadata/add_cloud_metadata.go:319	add_cloud_metadata: hosting provider type not detected.
2019-02-12T10:35:43.656Z	INFO	elasticsearch/client.go:165	Elasticsearch url: http://localhost:9200
2019-02-12T10:35:43.656Z	INFO	[publisher]	pipeline/module.go:110	Beat name: localhost
2019-02-12T10:35:43.657Z	INFO	instance/beat.go:403	filebeat start running.
2019-02-12T10:35:43.657Z	INFO	registrar/registrar.go:134	Loading registrar data from /home/outflink/filebeat-6.6.0-linux-x86_64/data/registry
2019-02-12T10:35:43.657Z	INFO	registrar/registrar.go:141	States Loaded from registrar: 0
2019-02-12T10:35:43.657Z	INFO	crawler/crawler.go:72	Loading Inputs: 1
2019-02-12T10:35:43.657Z	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2019-02-12T10:35:43.658Z	INFO	log/input.go:138	Configured paths: [/var/lib/docker/containers/*/*.log]
2019-02-12T10:35:43.658Z	INFO	input/input.go:114	Starting input of type: docker; ID: 3192729855305101819
2019-02-12T10:35:43.658Z	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2019-02-12T10:35:43.658Z	INFO	cfgfile/reload.go:150	Config reloader started
2019-02-12T10:35:43.658Z	INFO	cfgfile/reload.go:205	Loading of config files completed.

Any ideas?


(Alejandro Kapit) #5

No, I have volumes as follows:

volumes:
      - $HOME/filebeat_data:/usr/share/filebeat/data
      - /var/lib/docker/containers:/usr/share/filebeat/dockerlogs:ro
      - /var/run/docker.sock:/var/run/docker.sock

But anyways I'm currently running filebeat manually from the host machine, to get rid of possible docker configs errors.


(Alejandro Kapit) #6

I also tried with the config as exactly described here at the elastic blog:

With no success..


(Alejandro Kapit) #7

I've realised that filebeat wasn't taking the right config file, I'm running it like this:

sudo ./filebeat -e -c filebeat.yml

and it tries to read the docker logs. Now there is another issue:

2019-02-12T13:51:10.432Z ERROR log/harvester.go:282 Read line error: invalid CRI log format; File: /var/lib/docker/containers/7e8d82f610ae381298e0f5cd8686dbd5d52545cac03f04813d943ae24a1d4773/7e8d82f610ae381298e0f5cd8686dbd5d52545cac03f04813d943ae24a1d4773-json.log


(Carolina Baldino) #8

Hi! So if now containers.ids: '*' is correct, I think that these other keys are making filebeat stop:

    json.message_key: log
    json.keys_under_root: true

They need to be indented at the same level as ids. Try the following:

filebeat.inputs:
  - type: docker
    containers:
        ids:
             - '*'
        json.message_key: log
        json.keys_under_root: true

It happend to me some days ago and that solved my problem. (Pay attention to the indentation)

Cheers!