Docker Logs not sent to ES

[rorixrebel (Miguel Valdes)]

So i got a home cluster running on Docker which is working fine and i can push data to it from filebeat and metricbeat, but im having issues understanding why my docker logs are not being sent to the cluster.

i have the path setup in my filebeat.yml and i can see in the log that it does pick up that path.

config

filebeat.inputs:

- type: log
enabled: true
paths:
- /var/log/*.log
- /var/lib/docker/containers/*/*.log
fields:
source_system: system
fields_under_root: true

logs:

2020-01-22T14:43:51.624-0600    INFO    log/input.go:152        Configured paths: [/var/log/*.log /var/lib/docker/containers/*/*.log]

But for some reason the logs are not read, not even added to the registry.

2020-01-22T14:44:01.625-0600    DEBUG   [input] input/input.go:152      Run input
2020-01-22T14:44:01.625-0600    DEBUG   [input] log/input.go:191        Start next scan
2020-01-22T14:44:01.625-0600    DEBUG   [input] log/input.go:421        Check file for harvesting: /var/log/Xorg.0.log
2020-01-22T14:44:01.625-0600    DEBUG   [input] log/input.go:511        Update existing file for harvesting: /var/log/Xorg.0.log, offset: 39306
2020-01-22T14:44:01.625-0600    DEBUG   [input] log/input.go:565        File didn't change: /var/log/Xorg.0.log
2020-01-22T14:44:01.625-0600    DEBUG   [input] log/input.go:421        Check file for harvesting: /var/log/Xorg.1.log
2020-01-22T14:44:01.625-0600    DEBUG   [input] log/input.go:511        Update existing file for harvesting: /var/log/Xorg.1.log, offset: 40622
2020-01-22T14:44:01.625-0600    DEBUG   [input] log/input.go:565        File didn't change: /var/log/Xorg.1.log
2020-01-22T14:44:01.625-0600    DEBUG   [input] log/input.go:421        Check file for harvesting: /var/log/pacman.log
2020-01-22T14:44:01.625-0600    DEBUG   [input] log/input.go:511        Update existing file for harvesting: /var/log/pacman.log, offset: 640504
2020-01-22T14:44:01.625-0600    DEBUG   [input] log/input.go:565        File didn't change: /var/log/pacman.log
2020-01-22T14:44:01.625-0600    DEBUG   [input] log/input.go:212        input states cleaned up. Before: 3, After: 3, Pending: 0

Got my beat running in debug mode hoping i would see an error related to permissions or similar but no luck so far.

anything i could be missing?

[Mario_Castro]

Hi @rorixrebel

The indentation in your config doesn't seem correct. Maybe it is just a copy-paste error but take a closer look if not.

You can also use the add_docker_metadata processor https://www.elastic.co/guide/en/beats/filebeat/master/add-docker-metadata.html

[rorixrebel (Miguel Valdes)]

Yeah thats a copy paste error, the beat is running and working correctly for my other logs. Will add the metadata_processor but that shouldn't prevent it from collecting my docker logs.

Any other clue on what i'm missing?

[Mario_Castro]

I forgot to mention: try to run filebeat with -e -d "*", maybe it will show something.

Also, I don't think the problem is this but it's a cheap shot... Filebeat stores the current position of the files it is reading. If you are testing in a home server, maybe you are trying to re-read the same file many times (for testing). If you do this, you must delete the data folder where Filebeat is storing the current line read every time you execute Filebeat (to re-read the same file)

[rorixrebel (Miguel Valdes)]

Yeah tried that before, no errors or anything related.
Will try to make a script to move the logs someplace else or use syslog to push the data to elasticsearch.
Open to more suggestions.