# Indexing on the basis of fields in filebeat.yml

**URL:** <https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 28, 2016, 1:58pm UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684 "2016-04-28T13:58:03Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\
**Post date:** [April 28, 2016, 1:58pm UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/1 "2016-04-28T13:58:03Z")

</div>

Hi,  
I am using FB to send logs to LS-Shipper.  
Event processing pipeline is FB-LS-Shipper-\>Redis-\>LS-Indexer-\>nGinx--\>ES  
Below is FB configuration.  
Note: I have created one field Application with 'A' in uppercase.

```
filebeat:
  # List of prospectors to fetch data.
  prospectors:
    - 
      paths:
        - /var/log/messages
    
      input_type: syslogs
   
      fields:
 
        Application: tf

     
      fields_under_root: true
     
      document_type: syslogs
   
  registry_file: /var/lib/filebeat/registry

output:
  ### Logstash as output
  logstash:
    # The Logstash hosts
    hosts: ["shipper:5000"]
 
    # Optional TLS. By default is off.
    #tls:
      # List Iof root certificates for HTTPS server verifications
      #certificate_authorities: ["/opt/logstash-forwarder/SSL/abc-issuing.cer.pem"]

logging:

   to_files: true

  # To enable logging to files, to_files option has to be set to true
   files:
    # The directory where the log files will written to.
     path: /var/log/filebeat

    # The name of the files where the logs are written to.
     name: filebeat

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
     rotateeverybytes: 10485760 # = 10MB
 
   level: debug

```

I want to create Index %{Application}-YYYY-MM-DD so that each application logs will go in its own index.  
For this, I have done LS-Indexer configuration as below.

```
input {
  redis {
    host => "localhost"
    data_type => "list"
    key => "logstash"
   }

}
output {
elasticsearch {
   hosts => ["xxx.xx.xx.xx:8008"]
   index => "%{Application}-%{+YYYY.MM.dd}"
   document_type => "%{[@metadata][type]}"
   }
}

```

it shows me exception in ES.  
`org.elasticsearch.indices.InvalidIndexNameException: [%{Application}-2016.04.28] Invalid index name [%{Application}-2016.04.28], must be lowercase`

Event processing pipeline is FB-LS-Shipper-\>Redis-\>LS-Indexer-\>nGinx--\>ES  
When I connect FB directly to Indexer, then it works fine.

Please tell me whats wrong with above configuration.

br,  
Sunil

---

<div class="post-metadata">

**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\
**Post date:** [April 29, 2016, 5:51am UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/2 "2016-04-29T05:51:24Z")

</div>

Hi,  
I have also tried below code in indexer, still its says index name must be lower case.

output {  
elasticsearch {  
hosts =\> ["xxx.xx.xx.xx:8008"]  
index =\> "%{[fields][Application]}-%{+YYYY.MM.dd}"  
document\_type =\> "%{[@metadata][type]}"  
}

Why does it works well when I bypass LS-SHieppr and nginx. and it works when I directly send logs to LS-indexer.  
Is there anything extra happens with medata and fields when LS-Shiper sends the data to redis?

br,  
Sunil

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 29, 2016, 6:07am UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/3 "2016-04-29T06:07:09Z")

</div>

Replace the elasticsearch output with a `stdout { codec => rubydebug { metadata => true } }` output so that you can see exactly what the events look like. Perhaps you somewhere use a plain codec instead of a json codec.

---

<div class="post-metadata">

**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\
**Post date:** [April 29, 2016, 6:08am UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/4 "2016-04-29T06:08:38Z")

</div>

Hi Mgnus,  
I will do that change and let you know.  
Can you please explain more below statement?

> [@magnusbaeck](#):
>
> Perhaps you somewhere use a plain codec instead of a json codec

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 29, 2016, 6:14am UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/5 "2016-04-29T06:14:12Z")

</div>

If the codec settings of inputs and outputs don't match you can get unwanted results. For example, if the output sends the JSON string

```
{"message": "this is a message", "@timestamp": "2016-04-29T08:10:00.000Z"}

```

but the receiving input uses a plain string this won't get deserialized into an event with `message` and `@timestamp` fields. You'll get an event with a `message` field that contains '{"message": "this is a message", "@timestamp": "2016-04-29T08:10:00.000Z"}'.

---

<div class="post-metadata">

**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\
**Post date:** [April 29, 2016, 6:17am UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/6 "2016-04-29T06:17:34Z")

</div>

I just saw output at stout

{  
"@timestamp" =\> "2016-04-29T06:12:41.710Z",  
"message" =\> "{"message":"Apr 28 13:04:20 [hostname.fi](http://hostname.fi) SYSLOG 20727 - - INFO;2016-04-28T16:04:20.128+0300;AM-Process;-;Transactions processing has been initialized;-;-;-;-;transactionProcessorFactory.js;-;-","@version":"1","@timestamp":"2016-04-29T06:09:55.990Z","application":"tf","beat":{"hostname":"[hostname.lij.fi](http://hostname.lij.fi)","name":"[hostname.lij.fi](http://hostname.lij.fi)"},"count":1,"input\_type":"log","offset":604095170,"source":"/var/log/messages","type":"syslogs","host":"[hostname.lij.fi](http://hostname.lij.fi)","tags":["\> beats\_input\_codec\_plain\_applied"]}",  
"@version" =\> "1",  
"eventLogTime" =\> "2016-04-28T13:04:20.128Z"  
}

It says "beats\_input\_codec\_plain\_applied"  
Where does it come from?  
br,  
Sunil

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 29, 2016, 6:18am UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/7 "2016-04-29T06:18:55Z")

</div>

This confirms my hypothesis. One of your inputs (maybe the redis one?) should use a json input but currently doesn't.

---

<div class="post-metadata">

**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\
**Post date:** [April 29, 2016, 6:23am UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/8 "2016-04-29T06:23:51Z")

</div>

Hi,  
Thanks.  
OK I will try to find out this setting.  
If you have any hint, please share.

is there any possibility of something wrong at shipper side.

**Note:** I had old LS-1.5.5. Now I have upgraded to LS 2.2.0 using rpm -Uvh LS-2.2x.rpm .

br,  
Sunil

---

<div class="post-metadata">

**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\
**Post date:** [April 29, 2016, 9:41am UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/9 "2016-04-29T09:41:04Z")

</div>

Hello,  
I tried codec =\> "json" and json\_lines in both shiper and indexer , but it didn't resolve my problem. ☹

br,  
Sunil

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 30, 2016, 2:51pm UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/10 "2016-04-30T14:51:49Z")

</div>

Exactly where did you set the `codec` option? Exactly what do the messages stored in Redis look like?

---

<div class="post-metadata">

**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\
**Post date:** [May 2, 2016, 5:21am UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/11 "2016-05-02T05:21:52Z")

</div>

Hi MAgnus,

I have 3 places where I tried.

In LS-Shipper:  
Input{  
beats {  
....

}  
}  
And output redis {

}

In LS-Indexer:

Input {  
redis {

}  
}  
Anything wrong in this?

br,  
Sunil

---

<div class="post-metadata">

**Author:** ![sunilmchaudhari](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunilmchaudhari/32/9475_2.png) [@sunilmchaudhari](https://discuss.elastic.co/u/sunilmchaudhari)\
**Post date:** [May 3, 2016, 10:37am UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/12 "2016-05-03T10:37:20Z")

</div>

Hi Magnus,

there was multiline codec in input redis in LS-Shipper. When I removed that it started working.

But now, I have no way to use multiline codec.  
Multiline filter is deprecated in LS2.2  
Multliline error stacktraces are splitting in different rows on KIbana. ☹

br,  
Sunil

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [May 3, 2016, 11:59am UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/13 "2016-05-03T11:59:10Z")

</div>

why not use multiline support in filebeat?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:52pm UTC](https://discuss.elastic.co/t/indexing-on-the-basis-of-fields-in-filebeat-yml/48684/14 "2017-07-05T21:52:28Z")

</div>