Issue with Filebeat Custom cluster_id Field and Routing Logs to Specific Kibana Spaces via Ingest Pipeline

Elastic Stack Beats
1

I'm configuring Filebeat to add a custom field (cluster_id) to logs and route these logs to specific indices based on the cluster_id. Additionally, I want to ensure that these logs are only visible in specific Kibana spaces. Despite multiple attempts, I'm facing issues with bulk insert failures and "string index out of bounds" exceptions, and I'm unsure how to correctly route logs to specific spaces.

Environment:

  • Filebeat Version: 8.14.3
  • Elasticsearch Version: 8.14.3
  • Kibana Version: 8.14.3
  • Deployment: Kubernetes on AWS EKS
  • Elasticsearch and Filebeat Pods Running in: Elastic namespace

Steps Taken

Step : Adding a Custom cluster_id Field

Objective: Add a custom cluster_id field to logs to distinguish logs from different Kubernetes clusters.

Configuration:

yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: elasticsearch
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*-${data.kubernetes.container.id}.log

    output.elasticsearch:
      hosts: ["http://localhost:9200"]
      username: ""
      password: ""
      pipeline: "route-by-cluster"

    processors:
      - add_cloud_metadata: ~
      - add_host_metadata: ~
      - add_kubernetes_metadata:
          in_cluster: true
      - add_fields:
          target: ''
          fields:
            cluster_id: 'eks-cluster-1'

    logging:
      level: debug
      to_files: true
      files:
        path: /var/log/filebeat
        name: filebeat
        keepfiles: 7
        permissions: 0644

Results: After adding the cluster_id field, I encountered bulk insert failures and "string index out of bounds" exceptions in Filebeat logs.

Step 3: Implementing an Ingest Pipeline to Route Logs by cluster_id

Objective: Use the cluster_id field to route logs to specific indices, with the intention of linking these indices to specific Kibana spaces.

Ingest Pipeline Configuration:

json

PUT _ingest/pipeline/route-by-cluster
{
  "description": "Pipeline to route logs based on cluster_id",
  "processors": [
    {
      "set": {
        "field": "_index",
        "value": "{{#ctx.cluster_id}}{% if ctx.cluster_id == 'eks-cluster-1' %}eks-cluster-1-logs{% else %}eks-cluster-2-logs{% endif %}{{/ctx.cluster_id}}"
      }
    }
  ]
}

Results: Logs failed to be routed as expected. Filebeat continued to show errors, and the logs did not appear in the specified indices.

Step 3: Attempting to Send Logs to Specific Kibana Spaces

Objective: Configure Filebeat and Elasticsearch so that logs are directed to specific Kibana spaces based on their cluster_id.

Steps Considered:

  • Linking specific indices (e.g., eks-cluster-1-logs, eks-cluster-2-logs) to specific Kibana spaces.
  • Using the ingest pipeline to ensure the correct routing of logs.

Results: Uncertainty on how to correctly configure this setup. Not sure if the approach of using an ingest pipeline to set the index pattern and then linking it to a Kibana space is correct.

Request for Assistance:

I need help with the following:

  1. Custom cluster_id Field:
  • Is the method of adding a cluster_id field via the add_fields processor in Filebeat correct?
  • Could this method be causing the bulk insert failures and "string index out of bounds" exceptions?
  1. Routing Logs to Specific Indices:
  • Is the ingest pipeline configuration appropriate for routing logs based on the cluster_id?
  • Are there better ways to achieve this?
  1. Directing Logs to Specific Kibana Spaces:
  • How can I ensure that logs indexed based on the cluster_id are only visible in specific Kibana spaces?
  • What are the best practices for linking specific indices to Kibana spaces?

Any guidance or solutions for these issues would be greatly appreciated. Thank you!