Filebeat trouble with separate indexes per namespace

Elastic Stack Beats
1

Hello,

I currently try to split data into multiple indexes to simplify managing user access only for data from specific namespaces. I know this can be a problem to have too many small indexes. So normally I want to group some namespaces into one index.

I just want to see one index for each namespace working. But for sure I accept solutions with my grouping namespaces too :wink:

Right now there is not even one filebeat* index created and filling with data.

As far as I know, this is not possible with ILM enabled.

Elastic Cloud on Kubernetes 1.8 - Openshift 4.6.x

My filebeat config example:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
spec:
  type: filebeat
  version: 7.15.0
  elasticsearchRef:
    name: elasticsearch
  kibanaRef:
    name: kibana
  config:
    output.elasticsearch:
      index: 'filebeat-%{[agent.version]}-%{[kubernetes.namespace]}-%{+yyyy.MM.dd}'
      indices:
        - index: 'filebeat-%{[agent.version]}-%{[kubernetes.namespace]}-%{+yyyy.MM.dd}'
          when:
            or:
              - equals:
                   kubernetes.namespace: kubernetes.namespace-01
              - equals:
                   kubernetes.namespace: kubernetes.namespace-02
              - equals:
                   kubernetes.namespace: kubernetes.namespace-03
              - equals:
                   kubernetes.namespace: kubernetes.namespace-04
    setup.template.name: "filebeat-%{[agent.version]}-%{[kubernetes.namespace]}"
    setup.template.pattern: "filebeat-%{[agent.version]}-%{[kubernetes.namespace]}-*"
    setup.ilm.enabled: false
    filebeat.autodiscover.providers:
    - node: ${NODE_NAME}
      type: kubernetes
      hints.default_config.enabled: "false"
      templates:
      - condition.equals.kubernetes.namespace: kubernetes.namespace-01
        config:
        - paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
          type: container
          processors:
          - decode_json_fields:
              fields: "message"
              process_array: false
              max_depth: 1
              target: "logMessage"
              overwrite_keys: false
              add_error_key: true
              expand_keys: true
      - condition.equals.kubernetes.namespace: kubernetes.namespace-02
        config:
        - paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
          type: container
          processors:
          - decode_json_fields:
              fields: "message"
              process_array: false
              max_depth: 1
              target: "logMessage"
              overwrite_keys: false
              add_error_key: true
              expand_keys: true
      - condition.equals.kubernetes.namespace: kubernetes.namespace-03
        config:
        - paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
          type: container
      - condition.equals.kubernetes.namespace: kubernetes.namespace-04
        config:
        - paths: ["/var/log/containers/*${data.kubernetes.container.id}.log"]
          type: container

Regards,
Marius

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.