Filebeat Autodiscovery for coredns in docker, what labels to use?

[Masta_Boombastic]

I'm running elk and coredns in docker.

I am having difficulty getting any log data from coredns docker container using the autodiscovery.

The sample apache auto discovery works a treat for me, I get data in ES and nice apache fields, https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html

For coredns I've tried -

--label co.elastic.logs/module=coredns \

and using labels from the apache example -

--label co.elastic.logs/module=coredns \
--label co.elastic.logs/fileset.stdout=access \
--label co.elastic.logs/fileset.stderr=error \
--label co.elastic.metrics/module=coredns \
--label co.elastic.metrics/metricsets=status \
--label co.elastic.metrics/hosts='${data.host}:${data.port}' \

Is there a list of what labels are needed for coredns module?

How do I go about debugging this? Does filebeat output an error somewhere?

Thanks!

[MangledDeutz]

Same problem here.

After much exploration, it seems that the coredns ingester will not work by default, because it is expecting a leading timestamp that is not there with the default coredns / docker logging.

Here is how I solved it:

sed -i'' -e "s,%{timestamp} ,,g" module/coredns/log/ingest/pipeline-plaintext.json
sed -i'' -e "s,%{timestamp} ,,g" module/coredns/log/ingest/pipeline-json.json
sed -i'' -e 's,"ignore_failure" : true,"if": "ctx.timestamp != null",g' module/coredns/log/ingest/pipeline-entry.json

This removes the timestamp entry from the ingester (must be applied in your logbeat config / image).

Also, your labels seem wrong. Coredns does not have "access" and "error", just "log".

Here are working labels for me (assuming the above patch as well):

    co.elastic.logs/module=coredns
    co.elastic.logs/fileset=log

[MangledDeutz]

@Masta_Boombastic Here is a docker image for filebeat that contains the fixes above (and does work as expected for the coredns module).

https://hub.docker.com/r/dubodubonduponey/filebeat

Source:

dubo-dubon-duponey/docker-elastic-filebeat

!!!USE AT YOUR OWN RISK!!! as I don't intend on maintaining this, beyond my own needs.

Hope that helps.

Note that the dashboard still doesn't work (getting exceptions).
I wonder if this module is even maintained.

Latest version of the image also includes fixes for the dashboard & viz.

[Masta_Boombastic]

Excellent! Worked a treat.

Thanks for this. You should get a double bonus for a reply using sed. I hadn't seen sed used with commas before, so that's another bonus point.

While investigating this I found a comment on coredns forum that removed the timestamp due to it being displayed doublely when using some logging mechanism. I guess this is all a moving target and difficult to keep everthing up to date.

In the end I copied the coredns module config from the current filebeat image docker container, made your changes to my local config, and then mount those files again into filebeat container.

For those in similar situation, useful commands to run. (where coredns is your container name, or use its id) :

docker logs coredns
docker logs -t coredns

Thanks Again