# Filebeat reporting stops after index errors

**URL:** https://discuss.elastic.co/t/filebeat-reporting-stops-after-index-errors/255485
**Category:** Beats
**Tags:** docker, filebeat
**Created:** [November 16, 2020, 9:44am UTC](https://discuss.elastic.co/t/filebeat-reporting-stops-after-index-errors/255485 "2020-11-16T09:44:29Z")
**Posts on this page:** 2
**Page:** 1

<div class="post-metadata">

### Author: ![CatalinM](https://avatars.discourse-cdn.com/v4/letter/c/cab0a1/32.png) [@CatalinM](https://discuss.elastic.co/u/CatalinM)
#### Post date: [November 16, 2020, 9:44am UTC](https://discuss.elastic.co/t/filebeat-reporting-stops-after-index-errors/255485/1 "2020-11-16T09:44:29Z")

</div>

I've noticed Filebeat stopped sending logs to ES after awhile (seems like some pods are restarting - might be the Jaeger agent with the cleanup jobs start around midnight). It's always caused by an empty index name due to variable replacement failure, but I can't seem to figure out why... I've kept adding filters (to prevent picking up logs from pods that don't have the metadata I'm using) and a drop\_event processor to prevent picking up events where no kubernetes metadata is available.

Errors are like

```auto
2020-11-16T09:16:02.819Z	DEBUG	[elasticsearch]	elasticsearch/client.go:390	Bulk item insert failed (i=0, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}
2020-11-16T09:16:02.819Z	DEBUG	[elasticsearch]	elasticsearch/client.go:390	Bulk item insert failed (i=1, status=500): {"type":"string_index_out_of_bounds_exception","reason":"String index out of range: 0"}

```

```auto
Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":249200,"time":{"ms":16}},"total":{"ticks":3447000,"time":{"ms":688},"value":3447000},"user":{"ticks":3197800,"time":{"ms":672}}},"handles":{"limit":{"hard":65536,"soft":65536},"open":10},"info":{"ephemeral_id":"1e130f4a-c465-4398-844a-8437c48be4ab","uptime":{"ms":238260052}},"memstats":{"gc_next":32620960,"memory_alloc":21225256,"memory_total":366173647424},"runtime":{"goroutines":144}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"batches":4,"failed":8,"total":8}},"pipeline":{"clients":13,"events":{"active":2389,"retry":18}}},"registrar":{"states":{"current":68}},"system":{"load":{"1":5.4,"15":7.36,"5":7.24,"norm":{"1":2.7,"15":3.68,"5":3.62}}}}}}

```

Here's my config:

```auto
filebeat.yml: |-
    setup:
      ilm.enabled: false
      template:
        name: "mylogs"
        pattern: "mylogs-*"
        overwrite: true

    filebeat.autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}
          templates:
            - condition.and:
              - equals: 
                  kubernetes.namespace: sandbox
              - not.contains:
                  kubernetes.pod.name: jaeger-collector
              - not.contains:
                  kubernetes.pod.name: jaeger-spark
              config:
                - type: container
                  tail_files: true
                  symlinks: true
                  paths:
                    - /var/log/containers/*-${data.kubernetes.container.id}.log

    processors:
      - add_cloud_metadata:
      - add_host_metadata:
      
      - drop_event:
          when:
            not:
              regexp:
                kubernetes.namespace: ".*"

      - decode_json_fields:
          fields: ["message"]
          max_depth: 8
          target: ""
          overwrite_keys: true

    output.elasticsearch:  
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      index: "kytelogs-%{[kubernetes.labels.app]}-%{[kubernetes.labels.domain]}-%{[agent.version]}-%{+yyyy.MM.dd}" 

```

Usually, this failed every night. After adding the `drop_event` processor, it lasted the weekend, but then it started failing again.

A quick fix is to just delete the filebeat pod, but it's a lot of manual work and it still results in lost logs. I would appreciate any suggestions on how to debug this further.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 14, 2020, 11:44am UTC](https://discuss.elastic.co/t/filebeat-reporting-stops-after-index-errors/255485/2 "2020-12-14T11:44:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.