K8 logging with ECK

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi,

Im interested in pushing logs from kubernetes containers to my elastic cluster are there any guides on how to implement logstash and/or filebeat with ECK?

2

ECK 1.2. now supports Beats. Take a look here https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-beat-quickstart.html

3

Cool, however i managed to setup filebeats before ECK 1.2 was released as stand alone. I setup filbeats using the following recipies you have on github which have been super helpful https://github.com/elastic/cloud-on-k8s/blob/1.1/config/recipes/beats/2_filebeat-kubernetes.yaml

  1. Im curious should i move to the ECK version since im using ECK api general elasticsearch and kibana.

  2. Looking at the new version https://github.com/elastic/cloud-on-k8s/blob/1.2/config/recipes/beats/filebeat_autodiscover.yaml im not sure where i can define my templates and modules.

for example i have this section in my current filebeat.yml it allows me to use filebeat haproxy module. How would i set this in the ECK version. Is this feature missing to configure filebeat templates/modules?

data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          host: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
          templates:
            - condition.or:
                - equals.kubernetes.labels.app: "cgg-haproxy"
                - equals.kubernetes.labels.app: "haproxy-test"
              config:
                - module: haproxy
                  log:
                    enabled: true
                    input:
                      type: container
                      paths:
                        - /var/log/containers/*-${data.kubernetes.container.id}.log
    processors:
      - add_cloud_metadata:
      - add_host_metadata:

    setup.dashboards.enabled: true

    setup.kibana:
      host: "https://${KIBANA_HOST:kibana}:${KIBANA_PORT:5601}"
      ssl.enabled: true
      ssl.certificate_authorities:
      - /mnt/kibana/ca.crt


    output.elasticsearch:
      hosts: ['https://${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      ssl.certificate_authorities:
      - /mnt/elastic/tls.crt
---
4

Regarding your first question the main benefits about managing Beats with ECK is that all the Beats output configuration is managed for you: the certificate trust relationship between Beats, Elasticsearch and Kibana is automatically established and the output configuration (url, user, password) is also automatically set.

This way you can focus on the most meaningful Beats configuration settings.

Regarding your second question you can set this configuration right below the config element of the example you mentioned: https://github.com/elastic/cloud-on-k8s/blob/1.2/config/recipes/beats/filebeat_autodiscover.yaml#L14