Docker swarm container log monitoring - require help

[sl1729 (Siva)]

Hello All,

We got filebeat monitoring setup for the VM hosted apps and now trying to monitor logs of containers of docker swarm setup which uses json-file as a logging driver.

I have a filebeat (docker worker node) --> logstash --> elasticsearch configured,

Got the raw event passed to elasticsearch but it has only the container id of the docker as metadata, I am looking for more attributes like image, service and labels to be sent along with the event to make the dashboard more meaningful/useful.

My file beat config looks like,

- type: log
  paths:
    - '/mnt/docker/logs/containers/*/*.log'
  json.message_key: log
  json.keys_under_root: true
  processors:
    - add_docker_metadata: ~
  fields:
    app_id: docker
    env: dev
  fields_under_root: true

Any pointer or help is appreciated.

Note: Elastic version : 6.3.2

[Mario_Castro]

Hi

Please, can you provide us with a couple of lines of actual input and the output JSON which is going to Elasticsearch?

I don't see anything wrong in your config. I'm just wondering if the logs that you're fetching are the containers stderr / stdout and that maybe it's not mixed with logs of Swarm itself.

[Mario_Castro]

Ah, no wait, I think that your configuration isn't correct in the key add_docker_metadata. Refer here to double check https://www.elastic.co/guide/en/beats/filebeat/6.0/add-docker-metadata.html

[sl1729 (Siva)]

Thanks it is working now, and I am able to get the metadata added to the event.

- add_docker_metadata:
    host: "unix:///var/run/docker.sock"
    match_source_index: 4

And now trying to monitor how to filter(drop events) based on image names as we want to monitor specific application. Any pointers.

[Mario_Castro]

Well, you should filter your queries in Elasticsearch and this is the preferred method (you never know if you'll need that "filtered" data later).

If you really don't want that data to arrive to Elasticsearch, then you can filter it using Logstash https://www.elastic.co/guide/en/logstash/current/filter-plugins.html