Filebeat is not creating indices for each pod name

Elastic Stack Beats
,
1

Please help on this and stuck in this from last 5 days and not able to create index for each pod in Elasticsearch using filebeat.

Elastic search version: 7.10.1 AWS
Filebeat version: 7.12.1

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: filebeat
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.inputs:
    - type: container
      paths:
        - /var/lib/kubelet/pods/*_communication-service_*
      tags: ["communication-service"]
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/lib/kubelet/pods/"
                resource_type: "pod"
            default_indexers.enabled: false
            default_matchers.enabled: false
            indexers:
              - pod_name:
    - type: container
      paths:
        - /var/lib/kubelet/pods/*_bgv-dl-verification_*
      tags: ["bgv-dl-verification"]
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/lib/kubelet/pods/"
                resource_type: "pod"
            default_indexers.enabled: false
            default_matchers.enabled: false
            indexers:
              - pod_name:
  
 output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:443}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      protocol: "https"
      indices:
      - index: "logs-%{[kubernetes.pod.name]}-%{+yyyy.MM.dd}"
        when.contains:
          tags: "communication-service"
      - index: "logs-%{[kubernetes.pod.name]}-%{+yyyy.MM.dd}"
        when.contains:
          tags: "bgv-dl-verification"
      - index: "logs-%{[kubernetes.pod.name]}-%{+yyyy.MM.dd}"
        when.contains:
          tags: "reports-generator"
      - index: "logs-%{[kubernetes.pod.name]}-%{+yyyy.MM.dd}"
        when.contains:

nd the file beat logs says index is created but dont see any index with name of the pod

    DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: logging/fluentd-8cvhz {"libbeat.processor": "add_kubernetes_metadata"}
    2024-07-05T13:45:46.712Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:296 Created index logging/fluentd-8cvhz for pod logging/fluentd-8cvhz {"libbeat.processor": "add_kubernetes_metadata"}
    2024-07-05T13:45:46.712Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: kube-system/ebs-csi-node-fkj9g {"libbeat.processor": "add_kubernetes_metadata"}
    2024-07-05T13:45:46.712Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:296 Created index kube-system/ebs-csi-node-fkj9g for pod kube-system/ebs-csi-node-fkj9g {"libbeat.processor": "add_kubernetes_metadata"}
    2024-07-05T13:45:46.712Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:206 Adding kubernetes pod: dev/insurance-service-dev-deployment-649c79c86d-nbrvw {"libbeat.processor": "add_kubernetes_metadata"}
    2024-07-05T13:45:46.718Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:296 Created index dev/
2

Hi @nandeesh_b,

Welcome! To confirm, by Elasticsearch version 7.10.1 AWS, are you referring to the Amazon Elasticsearch Service (now OpenSearch) or a 7.10.1 Elasticsearch instance deployed to AWS?

It's an old version that is past EOL so I want to check.

3

Hi @carly.richmond ,

Yes..Its AWS Opensearch --Elastic search 7.10 ..

4

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance. See What is OpenSearch and the OpenSearch Dashboard? | Elastic for more details.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

5

You can't mix major versions. 7.12 with 7.10.

The best option is to at least use 7.17 which contains all the bug fixes and more importantly the security patches.

Better is to go to 8.14.2. It's also available from the AWS marketplace.