# Filebeat: Wrong index structure

**URL:** <https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700>\
**Category:** Logstash\
**Created:** [January 8, 2016, 7:59am UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700 "2016-01-08T07:59:40Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![pbmsys](https://avatars.discourse-cdn.com/v4/letter/p/57b2e6/32.png) [@pbmsys](https://discuss.elastic.co/u/pbmsys)\
**Post date:** [January 8, 2016, 7:59am UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/1 "2016-01-08T07:59:40Z")

</div>

Hi.

I am having some difficulties with the indexing of filebeat instead of using the "filebeat-YYYY.MM.dd" it sporadically indexes as %{[@metadata][beat]}-%{+YYYY.MM.dd}.. like it does not handle the metadata:

\*\* curl 'localhost:9200/\_cat/indices?v'\*\*

> health status index pri rep docs.count docs.deleted store.size pri.store.size  
> green open %{[@metadata][beat]}-2016.01.06 5 1 15 0 132.3kb 66.1kb  
> green open filebeat-2016.01.07 5 1 2257 0 1.9mb 978.6kb  
> green open .kibana 1 1 4 2 49.8kb 24.9kb  
> green open filebeat-2016.01.06 5 1 772300 0 567.5mb 284mb  
> green open %{[@metadata][beat]}-2016.01.07 5 1 2 0 18.3kb 9.1kb

**Logstash output conf:**  
\> output {  
\> elasticsearch {  
\> hosts =\> [somehosts]  
\> manage\_template =\> false  
\> index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
\> document\_type =\> "%{[@metadata][type]}"  
\> sniffing =\> true  
\> }  
\> }

**filebeat conf:**

```
> ################### Filebeat Configuration Example #########################

> ############################# Filebeat ######################################
> filebeat:
> # List of prospectors to fetch data.
> prospectors:
> -
> paths:
> # - /var/log/*.log
> - C:/tools/wildfly-8.2.0.Final/standalone/log/server.log
> - C:/tools/wildfly-8.2.0.Final/standalone/log/systemevent.log
> - C:/tools/wildfly-8.2.0.Final/standalone/log/access.log
> encoding: latin1

> input_type: log

> force_close_files: true

> registry_file: "C:/ProgramData/filebeat/registry"

> ############################# Output ##########################################
> # Configure what outputs to use when sending the data collected by the beat.
> # Multiple outputs may be used.
> output:

> ### Logstash as output
> logstash:
> # The Logstash hosts
> hosts: ["mylogstashhostsip:andport"]

> tls:
> # List of root certificates for HTTPS server verifications
> certificate_authorities: ["C:/Tools/filebeat-1.0.1/cicerologs_ca_systematic_com.crt","C:/Tools/filebeat-1.0.1/somecert.crt"]

> ############################# Logging #########################################

> logging:

> to_files: true

> files:

> path: C:/logs/filebeat

> name: filebeat.log

> rotateeverybytes: 10485760 # = 10MB

```

Any insight??

Best Regards  
Peter

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [January 8, 2016, 9:28am UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/2 "2016-01-08T09:28:27Z")

</div>

you directly have filebeat -\> logstash -\> elasticsearch?

can you share your full logstash config (hope it's not too much). Filebeat always outputs `[@metadata][beat]` and `[@metadata][type]`. In Logstash, if fields are missing, the 'original pattern' is used. This makes me wonder if you've non filebeat inputs or if some filter is overwriting the @metadata. Somewhere in logstash `@metadata` get's lost.

---

<div class="post-metadata">

**Author:** ![pbmsys](https://avatars.discourse-cdn.com/v4/letter/p/57b2e6/32.png) [@pbmsys](https://discuss.elastic.co/u/pbmsys)\
**Post date:** [January 8, 2016, 9:36am UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/3 "2016-01-08T09:36:26Z")

</div>

@Steffens.

Yeah directly filebeat --\> Logstash --\> El

i just merged the configuration to this: (i will just add that it made the indexes before i added the filters as well)

```
> input {
> beats{
> codec => multiline {
> pattern => "(^[a-zA-Z.]+(?:Error|Exception): .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
> what => "previous"
> }
> port => 3515
> ssl => true
> ssl_certificate => "/etc/pki/tls/certs/somecert.crt"
> ssl_key => "/etc/pki/tls/private/somepriv.key"
> }
> }

> filter {
> mutate {
> add_field => { "logstash_host" => "the logstash hostname for debug purposes" }
> }
> }

> filter {
> if [source]{
> grok {
> match => { "source" => '(?<log_type>[^\\/:+?"]+$)' }
> }
> }
> }

> output {
> elasticsearch {
> hosts => ["Somehosts"]
> manage_template => false
> index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
> document_type => "%{[@metadata][type]}"
> sniffing => true
> }
> }
```

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [January 8, 2016, 10:39am UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/4 "2016-01-08T10:39:36Z")

</div>

Makes me wonder if multiline drops the `@metadata`.

Will move to topic logstash forum.

---

<div class="post-metadata">

**Author:** ![pbmsys](https://avatars.discourse-cdn.com/v4/letter/p/57b2e6/32.png) [@pbmsys](https://discuss.elastic.co/u/pbmsys)\
**Post date:** [January 8, 2016, 11:20am UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/5 "2016-01-08T11:20:48Z")

</div>

It believe it did this before we added the multiline as well.

\*edit: My colleague confirms that the indexes was seen before the multiline filter was added to the configuration as well.

---

<div class="post-metadata">

**Author:** ![Steiniche](https://avatars.discourse-cdn.com/v4/letter/s/96bed5/32.png) [@Steiniche](https://discuss.elastic.co/u/Steiniche)\
**Post date:** [January 13, 2016, 2:04pm UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/8 "2016-01-13T14:04:18Z")

</div>

I have the exact same issue.  
It seems like it is only some messeges send from filebeat that does not contain the metadata information.  
It is, as far as I know, only an issue when this tag is present: beats\_input\_flushed\_by\_end\_of\_connection

---

<div class="post-metadata">

**Author:** ![byoung0589](https://avatars.discourse-cdn.com/v4/letter/b/59ef9b/32.png) [@byoung0589](https://discuss.elastic.co/u/byoung0589)\
**Post date:** [June 2, 2016, 12:49am UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/9 "2016-06-02T00:49:44Z")

</div>

Did anyone ever get an answer to this?

---

<div class="post-metadata">

**Author:** ![Steiniche](https://avatars.discourse-cdn.com/v4/letter/s/96bed5/32.png) [@Steiniche](https://discuss.elastic.co/u/Steiniche)\
**Post date:** [June 4, 2016, 9:09am UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/10 "2016-06-04T09:09:08Z")

</div>

I did not get any answer, however we are currently updated to Filebeat 1.2 and hopefully this will fix it.

---

<div class="post-metadata">

**Author:** ![pbmsys](https://avatars.discourse-cdn.com/v4/letter/p/57b2e6/32.png) [@pbmsys](https://discuss.elastic.co/u/pbmsys)\
**Post date:** [June 20, 2016, 12:58pm UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/11 "2016-06-20T12:58:09Z")

</div>

Hi @byoung0589

After running with logstash and elasticsearch for a while we have seen a pattern, that every time we restart one of our logstash servers it creates this index, when the logstash host is up to speed it allocates the data correct.

---

<div class="post-metadata">

**Author:** ![Rory\_Savage](https://avatars.discourse-cdn.com/v4/letter/r/65b543/32.png) [@Rory\_Savage](https://discuss.elastic.co/u/Rory_Savage)\
**Post date:** [August 2, 2016, 5:23pm UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/12 "2016-08-02T17:23:55Z")

</div>

> [@pbmsys](#):
>
> ogstash servers it creates t

So, I am not actually experiencing the "index" problem, but what I am seeing is that the tags on certain log messages are being set with "beats\_input\_flushed\_by\_end\_of\_connection". This is completely overwriting the tags for some of the logs I am sending and makes searching by tags a complete PITA. I have googled the crap out of this, and can not find a clear indication as to why this is happening.

I do have a Logstash-\>Elasticsearch setup, but what's funny is most of my logs are coming in fine. But! I have not found a pattern as to why some of my logs are coming in with this new tag "beats\_input\_flushed\_by\_end\_of\_connection"

Does anyone know why this is happening?

---

<div class="post-metadata">

**Author:** ![Rory\_Savage](https://avatars.discourse-cdn.com/v4/letter/r/65b543/32.png) [@Rory\_Savage](https://discuss.elastic.co/u/Rory_Savage)\
**Post date:** [August 2, 2016, 5:27pm UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/13 "2016-08-02T17:27:32Z")

</div>

Hmmm.... I am also seeing this "multiline\_codec\_max\_lines\_reached"

---

<div class="post-metadata">

**Author:** ![Rory\_Savage](https://avatars.discourse-cdn.com/v4/letter/r/65b543/32.png) [@Rory\_Savage](https://discuss.elastic.co/u/Rory_Savage)\
**Post date:** [August 2, 2016, 6:42pm UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/14 "2016-08-02T18:42:39Z")

</div>

> [@Steiniche](#):
>
> beats\_input\_flushed\_by\_end\_of\_connection

So I tracked the error down to being inside the logstash-input-beats ruby gem. It seems to be called when there is a new thread connection, but I am not sure why anyone would want to overwrite the exiting tags with this message on a new thread? Kinda lame.

---

<div class="post-metadata">

**Author:** ![dkkriste](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkkriste/32/13685_2.png) [@dkkriste](https://discuss.elastic.co/u/dkkriste)\
**Post date:** [December 7, 2016, 11:07am UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/15 "2016-12-07T11:07:22Z")

</div>

I am having the same issue with filebeat 5.0.1 and logstash 5.0.1  
Most messages are fine, but some are missing all metadata

---

<div class="post-metadata">

**Author:** ![Maher.Glenza](https://avatars.discourse-cdn.com/v4/letter/m/e5b9ba/32.png) [@Maher.Glenza](https://discuss.elastic.co/u/Maher.Glenza)\
**Post date:** [January 20, 2017, 7:24am UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/16 "2017-01-20T07:24:30Z")

</div>

> [@pbmsys](#):
>
> codec =\> multiline {  
> \> pattern =\> "(\[1\]+(?:Error|Exception): .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s\*Caused by:.+)"  
> \> what =\>

can i ask plz ?  
what is (codec =\> multiline {

> ```
> pattern => "(^[a-zA-Z.]+(?:Error|Exception): .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)"
> what => "previous"
> })
> 
> ```

and ( mutate {

> ```
> add_field => { "logstash_host" => "the logstash hostname for debug purposes" }
> })
> 
> ```

what do they do i want to understand how can i configure a filter with them .  
thank you

* * *

1. a-zA-Z.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:29am UTC](https://discuss.elastic.co/t/filebeat-wrong-index-structure/38700/17 "2017-07-06T04:29:12Z")

</div>