# Issue with Filebeat Custom cluster\_id Field and Routing Logs to Specific Kibana Spaces via Ingest Pipeline

**URL:** <https://discuss.elastic.co/t/issue-with-filebeat-custom-cluster-id-field-and-routing-logs-to-specific-kibana-spaces-via-ingest-pipeline/364645>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 9, 2024, 6:35am UTC](https://discuss.elastic.co/t/issue-with-filebeat-custom-cluster-id-field-and-routing-logs-to-specific-kibana-spaces-via-ingest-pipeline/364645 "2024-08-09T06:35:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anudeep](https://avatars.discourse-cdn.com/v4/letter/a/c67d28/32.png) [@Anudeep](https://discuss.elastic.co/u/Anudeep)\
**Post date:** [August 9, 2024, 6:35am UTC](https://discuss.elastic.co/t/issue-with-filebeat-custom-cluster-id-field-and-routing-logs-to-specific-kibana-spaces-via-ingest-pipeline/364645/1 "2024-08-09T06:35:01Z")

</div>

I'm configuring Filebeat to add a custom field (`cluster_id`) to logs and route these logs to specific indices based on the `cluster_id`. Additionally, I want to ensure that these logs are only visible in specific Kibana spaces. Despite multiple attempts, I'm facing issues with bulk insert failures and "string index out of bounds" exceptions, and I'm unsure how to correctly route logs to specific spaces.

### **Environment** :

- **Filebeat Version** : 8.14.3
- **Elasticsearch Version** : 8.14.3
- **Kibana Version** : 8.14.3
- **Deployment** : Kubernetes on AWS EKS
- **Elasticsearch and Filebeat Pods Running in** : Elastic namespace

* * *

Steps Taken

Step : Adding a Custom `cluster_id` Field

**Objective** : Add a custom `cluster_id` field to logs to distinguish logs from different Kubernetes clusters.

**Configuration** :

yaml

```auto
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: elasticsearch
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*-${data.kubernetes.container.id}.log

    output.elasticsearch:
      hosts: ["http://localhost:9200"]
      username: ""
      password: ""
      pipeline: "route-by-cluster"

    processors:
      - add_cloud_metadata: ~
      - add_host_metadata: ~
      - add_kubernetes_metadata:
          in_cluster: true
      - add_fields:
          target: ''
          fields:
            cluster_id: 'eks-cluster-1'

    logging:
      level: debug
      to_files: true
      files:
        path: /var/log/filebeat
        name: filebeat
        keepfiles: 7
        permissions: 0644

```

**Results** : After adding the `cluster_id` field, I encountered bulk insert failures and "string index out of bounds" exceptions in Filebeat logs.

* * *

Step 3: Implementing an Ingest Pipeline to Route Logs by `cluster_id`

**Objective** : Use the `cluster_id` field to route logs to specific indices, with the intention of linking these indices to specific Kibana spaces.

**Ingest Pipeline Configuration** :

json

```auto
PUT _ingest/pipeline/route-by-cluster
{
  "description": "Pipeline to route logs based on cluster_id",
  "processors": [
    {
      "set": {
        "field": "_index",
        "value": "{{#ctx.cluster_id}}{% if ctx.cluster_id == 'eks-cluster-1' %}eks-cluster-1-logs{% else %}eks-cluster-2-logs{% endif %}{{/ctx.cluster_id}}"
      }
    }
  ]
}

```

**Results** : Logs failed to be routed as expected. Filebeat continued to show errors, and the logs did not appear in the specified indices.

* * *

Step 3: Attempting to Send Logs to Specific Kibana Spaces

**Objective** : Configure Filebeat and Elasticsearch so that logs are directed to specific Kibana spaces based on their `cluster_id`.

**Steps Considered** :

- Linking specific indices (e.g., `eks-cluster-1-logs`, `eks-cluster-2-logs`) to specific Kibana spaces.
- Using the ingest pipeline to ensure the correct routing of logs.

**Results** : Uncertainty on how to correctly configure this setup. Not sure if the approach of using an ingest pipeline to set the index pattern and then linking it to a Kibana space is correct.

* * *

Request for Assistance:

I need help with the following:

1. Custom `cluster_id` Field:

- Is the method of adding a `cluster_id` field via the `add_fields` processor in Filebeat correct?
- Could this method be causing the bulk insert failures and "string index out of bounds" exceptions?

1. **Routing Logs to Specific Indices** :

- Is the ingest pipeline configuration appropriate for routing logs based on the `cluster_id`?
- Are there better ways to achieve this?

1. **Directing Logs to Specific Kibana Spaces** :

- How can I ensure that logs indexed based on the `cluster_id` are only visible in specific Kibana spaces?
- What are the best practices for linking specific indices to Kibana spaces?

Any guidance or solutions for these issues would be greatly appreciated. Thank you!

---

<div class="post-metadata">

**Author:** ![Trevor\_Blackford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/trevor_blackford/32/120087_2.png) [@Trevor\_Blackford](https://discuss.elastic.co/u/Trevor_Blackford)\
**Post date:** [February 19, 2025, 9:38pm UTC](https://discuss.elastic.co/t/issue-with-filebeat-custom-cluster-id-field-and-routing-logs-to-specific-kibana-spaces-via-ingest-pipeline/364645/2 "2025-02-19T21:38:23Z")

</div>

Hello, I think you may want to change the value of target in your filebeat configuration. You may be creating an empty subdictionary. I think @metadata would be the best target for this kind of field [Add fields | Filebeat Reference [8.17] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/add-fields.html)