# Filebeat 6.2.3 index is empty

**URL:** <https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 21, 2018, 8:04pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070 "2018-03-21T20:04:44Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![Wanderer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wanderer/32/55140_2.png) [@Wanderer](https://discuss.elastic.co/u/Wanderer)\
**Post date:** [March 21, 2018, 8:04pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/1 "2018-03-21T20:04:44Z")

</div>

Since upgrading to 6.2.3 the filebeat index is empty.

Any idea where to start looking?

Filebeat debug output:

2018-03-21T16:11:28.994-0400 DEBUG [publish] pipeline/processor.go:275 Publish event: {  
"@timestamp": "2018-03-21T20:11:28.994Z",  
"@metadata": {  
"beat": "filebeat",  
"type": "doc",  
"version": "6.2.3"  
},  
"source": "/usr/local/apache/logs/error\_log",  
"offset": 222572523,  
"message": "2018-03-21 16:11:27.897285 [INFO] [APVH\_ocpsa\_Suphp56:] PID: 14576, add child process pid: 16067, procinfo: 0x8972610",  
"prospector": {  
"type": "log"  
},  
"beat": {  
"name": "servername-filebeat",  
"hostname": "servername",  
"version": "6.2.3"

Discover output:  
{  
"\_index": "filebeat-6.2.3-2018.03.21",  
"\_type": "doc",  
"\_id": "GVckSmIB41zG7qDLXuZ8",  
"\_version": 1,  
"\_score": null,  
"fields": {  
"@timestamp": [  
"2018-03-21T19:58:39.000Z"  
]  
},  
"sort": [  
1521662319000  
]  
}

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 22, 2018, 11:45am UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/2 "2018-03-22T11:45:14Z")

</div>

> [@Wanderer](#):
>
> Any idea where to start looking?

Check your Filebeat and Elasticsearch logs for errors.

Please share your Filebeat config.

---

<div class="post-metadata">

**Author:** ![Wanderer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wanderer/32/55140_2.png) [@Wanderer](https://discuss.elastic.co/u/Wanderer)\
**Post date:** [March 22, 2018, 1:26pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/3 "2018-03-22T13:26:46Z")

</div>

No errors in filebeat log.

One recurring error in the elasticsearch log. Is this the proper way to enable logging?

```auto
PUT /_cluster/settings
{
  "transient": {
    "logger._root": "debug"
  }
}

```

`Elasticsearch error: [DEBUG][o.e.g.MetaStateService] [es-01] [OiByffRoRqWXA1pJ7tdPiA] failed to find metadata for existing index location`

filebeat.yml

```auto
filebeat.prospectors:
- type: log

  paths:
    - /var/log/*.log
    - /usr/local/cpanel/logs/*.log
    - /usr/local/apache/logs/*.log
    - /usr/local/apache/logs/*_log
    - /usr/local/lsws/logs/*.log
    - /usr/local/cpanel/logs/*_log
    - /var/lib/mysql/rssd5382.webaccountserver.err*

#============================= Filebeat modules ===============================

filebeat.config.modules:
  path: /etc/filebeat/modules.d/*.yml
  reload.enabled: true
  reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  index.codec: best_compression
  _source.enabled: false

#================================ Outputs =====================================

#-------------------------- Elasticsearch output -------------------------------

output.elasticsearch:
  enabled: true
  hosts: ["https://XXXXXXXXXXXXXXXXX:443/"]
  protocol: "https"
  username: "XXXXXXXXXX"
  password: "XXXXXXXXXXXXXX"
  worker: 4
  timeout: 90
  ssl.enabled: true
  ssl.verification_mode: none
  ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]
  ssl.certificate_authorities: ["XXXXXXXXXXXXXXXXXX"]
  ssl.certificate: "XXXXXXXXXXXXXXXXXX"
  ssl.key: "XXXXXXXXXXXXXXXXXX"

```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 22, 2018, 1:54pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/4 "2018-03-22T13:54:14Z")

</div>

Your filebeat config looks pretty standard. No issues there.

What does `GET _cat/indices/filebeat-6.2.3*` return?

And in your Filebeat logs can you post some of the lines containing "monitoring"? These contain metrics that get logged every 30s and will show if events are being written to ES.

> [@Wanderer](#):
>
> Is this the proper way to enable logging?

That looks right to me.

---

<div class="post-metadata">

**Author:** ![Wanderer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wanderer/32/55140_2.png) [@Wanderer](https://discuss.elastic.co/u/Wanderer)\
**Post date:** [March 22, 2018, 1:59pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/5 "2018-03-22T13:59:33Z")

</div>

yellow open filebeat-6.2.3-2018.03.22 UqJCmvBARQmTZYN4L\_RfgA 3 1 1756515 0 164.4mb 164.4mb  
yellow open filebeat-6.2.3-2018.03.21 o1oQLEH6QYaVXTi\_TUgm\_g 3 1 1785767 0 167.8mb 167.8mb  
yellow open filebeat-6.2.3-2018.03.20 OU5a1GkPRz2cYxixrf9qlw 3 1 59354 0 5.1mb 5.1mb

filebeat log output:  
2018-03-22T09:57:25.112-0400 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":129400,"time":129400},"total":{"ticks":682240,"time":682243,"value":682240},"user":{"ticks":552840,"time":552843}},"info":{"ephemeral\_id":"275a6546-e45a-4f3a-b5c4-56c62efc88f7","uptime":{"ms":63990014}},"memstats":{"gc\_next":8065776,"memory\_alloc":6928800,"memory\_total":47135657920}},"filebeat":{"events":{"active":7,"added":148,"done":141},"harvester":{"open\_files":10,"running":10,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"events":{"acked":132,"batches":28,"total":132},"read":{"bytes":22393},"write":{"bytes":62758}},"pipeline":{"clients":1,"events":{"active":10,"filtered":9,"published":139,"total":148},"queue":{"acked":132}}},"registrar":{"states":{"current":813,"update":141},"writes":28},"system":{"load":{"1":4.58,"15":4.63,"5":4.12,"norm":{"1":0.1908,"15":0.1929,"5":0.1717}}}}}}  
2018-03-22T09:57:55.113-0400 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":129470,"time":129476},"total":{"ticks":682620,"time":682627,"value":682620},"user":{"ticks":553150,"time":553151}},"info":{"ephemeral\_id":"275a6546-e45a-4f3a-b5c4-56c62efc88f7","uptime":{"ms":64020015}},"memstats":{"gc\_next":7479104,"memory\_alloc":5016448,"memory\_total":47160140872}},"filebeat":{"events":{"active":-6,"added":97,"done":103},"harvester":{"open\_files":10,"running":10}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"events":{"acked":103,"batches":28,"total":103},"read":{"bytes":22204},"write":{"bytes":52646}},"pipeline":{"clients":1,"events":{"active":4,"published":97,"total":97},"queue":{"acked":103}}},"registrar":{"states":{"current":813,"update":103},"writes":28},"system":{"load":{"1":4.77,"15":4.64,"5":4.22,"norm":{"1":0.1987,"15":0.1933,"5":0.1758}}}}}}

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 22, 2018, 2:48pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/6 "2018-03-22T14:48:09Z")

</div>

You have data in your indices (docs.count is 1756515, 1785767, and 59354 in the indicies). And Filebeat indicates that it's sending successfully.

```auto
        "output": {
          "events": {
            "acked": 132,
            "batches": 28,
            "total": 132
          },

```

Do you see events when you query `GET filebeat-6.2.3-2018.03.22/_search`?

---

<div class="post-metadata">

**Author:** ![Wanderer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wanderer/32/55140_2.png) [@Wanderer](https://discuss.elastic.co/u/Wanderer)\
**Post date:** [March 22, 2018, 2:49pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/7 "2018-03-22T14:49:51Z")

</div>

It appears the message field and others are missing. If I do the same query on a 6.2.2 index they are populated.

Output:

```
  {
    "_index": "filebeat-6.2.3-2018.03.22",
    "_type": "doc",
    "_id": "J5bdS2IB41zG7qDLINXB",
    "_score": 1
  },
  {
    "_index": "filebeat-6.2.3-2018.03.22",
    "_type": "doc",
    "_id": "KJbdS2IB41zG7qDLINXB",
    "_score": 1
  },
  {
    "_index": "filebeat-6.2.3-2018.03.22",
    "_type": "doc",
    "_id": "N5bdS2IB41zG7qDLINXi",
    "_score": 1
  },
  {
    "_index": "filebeat-6.2.3-2018.03.22",
    "_type": "doc",
    "_id": "OZbdS2IB41zG7qDLINXi",
    "_score": 1
```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 22, 2018, 3:29pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/8 "2018-03-22T15:29:40Z")

</div>

> [@Wanderer](#):
>
> \_source.enabled: false

You have disabled `_source`. I think this is a consequence. See [\_source field | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html)

---

<div class="post-metadata">

**Author:** ![Wanderer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wanderer/32/55140_2.png) [@Wanderer](https://discuss.elastic.co/u/Wanderer)\
**Post date:** [March 22, 2018, 3:43pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/9 "2018-03-22T15:43:25Z")

</div>

Interesting. We have about 4 servers running filebeat and on 6.2.2 they were all working. 6.2.3 they all stopped. I have to confirm with one other person - but we would have had no reason to modify this.

In looking at our old config this was disabled before as well. Is it enabled out of the box?

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 22, 2018, 3:45pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/10 "2018-03-22T15:45:53Z")

</div>

`_source` is enabled by default. You could check your index template and mappings used by 6.2.2 to see if \_source was enabled. Those are stored in Elasticsearch.

---

<div class="post-metadata">

**Author:** ![Wanderer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wanderer/32/55140_2.png) [@Wanderer](https://discuss.elastic.co/u/Wanderer)\
**Post date:** [March 22, 2018, 4:35pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/11 "2018-03-22T16:35:44Z")

</div>

I've reinstalled filebeat and still the same issue. I am guessing the problem is on the elastic-side since the filebeat log output shows all the correct fields with message.

Is there a way to tell if elastic is stripping out the fields? Is there a processor or something to look at in elastic?

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 22, 2018, 4:49pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/12 "2018-03-22T16:49:20Z")

</div>

So you re-installed. Did you set `_source.enabled: true` or remove that option so that the default is used?

If you do make a change to that value then you need to overwrite the installed index template. ([docs](https://www.elastic.co/guide/en/beats/filebeat/6.2/configuration-template.html))

`filebeat setup -e --template -E setup.template.overwrite=true`

And even after updating the template the change won't take effect until the next daily index is created.

---

<div class="post-metadata">

**Author:** ![Wanderer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wanderer/32/55140_2.png) [@Wanderer](https://discuss.elastic.co/u/Wanderer)\
**Post date:** [March 22, 2018, 4:53pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/13 "2018-03-22T16:53:45Z")

</div>

I left \_source.enabled to #\_source.enabled: false so it is commented out as in the default yml.

> [@andrewkroh](#):
>
> If you do make a change to that value then you need to overwrite the installed index template. (docs)
> 
> filebeat setup -e --template -E setup.template.overwrite=true
> 
> And even after updating the template the change won't take effect until the next daily index is created.

BAM! That did it.

So the moral of the story when updating filebeat is to run filebeat setup -e --template -E setup.template.overwrite=true after the update?

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [March 22, 2018, 4:57pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/14 "2018-03-22T16:57:04Z")

</div>

Well, filebeat installs versioned templates (e.g. \_template/filebeat-6.2.3) so it's not necessary to forcibly update the template on a version update because it will be installing a whole new template. It's only necessary to force an update if you modify config options affecting the template.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 19, 2018, 4:59pm UTC](https://discuss.elastic.co/t/filebeat-6-2-3-index-is-empty/125070/15 "2018-04-19T16:59:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.