Filebeat and Docker EE

Elastic Stack Beats
1

Hi There,
We have just installed a Docker EE cluster on 10 nodes (3 UCP , 3 DTR and 6 workers) .
Filebeat and Metricbeat are installed on each nodes with this configuration :

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            regexp:
              docker.container.name: ".*"
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              processors:
               - add_docker_metadata: ~

As you seen docker logs are caught by Filebeat , parsed and send to Elasticsearch.
Everything works fine but now we would like to activate Kubernetes as Orchestrator (embedded in Docker EE).

Do you ever experienced this configuration with Filebeat ? What is the conf to address Kubernetes api for the autodiscovering ?

Thank you for your help.
Eric

2

Have you considered the kubernetes autodiscovery provider?

3

Yes, but Filebeat or Metricbeats need to have some informations about kubernetes (like the hostname and the port). I don't find in the documentation a way to declare them inside my Beats configuration.
does it need the kubelet in workers or just the Kubernetes API in Masters nodes ?

4

Filebeat/Metricbeat auto discovery does connects to the local kubelet and listens to the local event stream.

Also see:

5

Fyi , Filebeat does not run in a container , it has been installed by a linux package.
It's like a unix agent.
Where can I specify the host and the port of k8s in Filebeat to enable the autodiscover ?

6

See kubernetes provider docs. The kubernetes providers has the settings in_cluster, host, and kube_config.

Normally host is not configured. Beat tries to discover/select the kubernetes host by these rules:

  • If host is provided in the config use it directly.
  • If the Beat is deployed in the k8s cluster, it uses the hostname of the pod. It uses the pod name to query the pod meta in order to read the node name.
  • If the Beat is deployed outside the k8s cluster, it use the machine-id to match against k8s nodes for the current node name.

That is, if Filebeat is installed on the same host as as the kubelet, it should still be able to find the correct node. If not you can use the host configuration:

filebeat.autodiscover:
  providers:
  - type: kubernetes
    host: "<node endpoint>"
    templates:
      - ...
1 Like
7

Thank you for the example.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.