# Mutiple mutate filters and thier output to different elasticsearch indexs

**URL:** https://discuss.elastic.co/t/mutiple-mutate-filters-and-thier-output-to-different-elasticsearch-indexs/59876
**Category:** Logstash
**Created:** [September 6, 2016, 11:20am UTC](https://discuss.elastic.co/t/mutiple-mutate-filters-and-thier-output-to-different-elasticsearch-indexs/59876 "2016-09-06T11:20:23Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![Ayush\_Rastogi](https://avatars.discourse-cdn.com/v4/letter/a/5daacb/32.png) [@Ayush\_Rastogi](https://discuss.elastic.co/u/Ayush_Rastogi)
#### Post date: [September 6, 2016, 11:20am UTC](https://discuss.elastic.co/t/mutiple-mutate-filters-and-thier-output-to-different-elasticsearch-indexs/59876/1 "2016-09-06T11:20:23Z")

</div>

HI All,

I have a log file like the following which I use logstash to process.

```
FilterPerfLogging input_count:2 ,TimeTaken:0.272 ,ConfigRate:7.35294 ,output_count:2
SubFilterLogging ContraintSize:5, ,TimeTaken:0.501

```

Now these two keywords FilterPerfLogging , SubFilterLogging are two different events with different parameters which I use KV filter to extract and store in elasticsearch.

My problem is that I am not able to store them separately in different Indexes,

This is my conf file

> input {  
> file {  
> path =\> "C:/Temp1/\*.syslog"  
> start\_position =\> beginning  
> ignore\_older =\> 0  
> }  
> }

> filter {  
> if [message] !~ /FilterPerfLogging / and [message] !~ /SubFilterLogging / {  
> drop { }  
> }  
> if [message] == /FilterPerfLogging / {  
> kv {  
> value\_split =\> ":"  
> field\_split =\> " ,"  
> }  
> mutate {  
> convert =\> {  
> "input\_count" =\> "integer"  
> "output\_count" =\> "integer"  
> "TimeTaken" =\> "float"  
> "ConfigRate" =\> "float"  
> }  
> }  
> }  
> if [message] == /SubFilterLogging / {  
> kv {  
> value\_split =\> ":"  
> field\_split =\> " ,"  
> }  
> mutate {  
> convert =\> {  
> "ContraintSize" =\> "integer"  
> }  
> add\_tag =\> ["subfilter"]  
> }  
> }  
> }

> output {  
> if "subfilter" in [tags] {  
> elasticsearch {  
> index =\> "subFilterIndex"  
> }  
> }  
> else{  
> elasticsearch {  
> }  
> }  
> }

My problem is the index subFilterIndex is never created and every entry goes to the default index.  
And the fields which were earlier created by the KV filter is also gone and all I see is the message string in kibana.  
Can somebody suggest some way or figure out what I have been missing.

Thanks,  
Ayush

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 6, 2016, 11:36am UTC](https://discuss.elastic.co/t/mutiple-mutate-filters-and-thier-output-to-different-elasticsearch-indexs/59876/2 "2016-09-06T11:36:42Z")

</div>

I think you'll find that your second kv filter doesn't produce a field named `[ConstraintSize]`. As a consequence the subsequent mutate filter fails and then the "subfilter" tag is never added. You should (temporarily) replace your elasticsearch output(s) with `stdout { codec => rubydebug }` output(s) to verify what the resulting events actually look like.

---

<div class="post-metadata">

### Author: ![Ayush\_Rastogi](https://avatars.discourse-cdn.com/v4/letter/a/5daacb/32.png) [@Ayush\_Rastogi](https://discuss.elastic.co/u/Ayush_Rastogi)
#### Post date: [September 6, 2016, 12:03pm UTC](https://discuss.elastic.co/t/mutiple-mutate-filters-and-thier-output-to-different-elasticsearch-indexs/59876/3 "2016-09-06T12:03:39Z")

</div>

It looks like

> {  
> "message" =\> "FilterPerfLogging input\_count:6 ,TimeTaken:0.444 ,ConfigRate:13.5135 ,output\_count:6\r",  
> "@version" =\> "1",  
> "@timestamp" =\> "2016-09-06T11:52:40.862Z",  
> "path" =\> "C:/Temp1/\*.exe22902044.syslog",  
> "host" =\> "XXXX"  
> }

> {  
> "message" =\> "SubFilterLogging ContraintSize:5\r",  
> "@version" =\> "1",  
> "@timestamp" =\> "2016-09-06T11:53:00.080Z",  
> "path" =\> "C:/Temp1/\*.exe22902044.syslog",  
> "host" =\> "XXXX"  
> }

What do you think could be this issue with configuration/ any workarounds ?

---

<div class="post-metadata">

### Author: ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [September 6, 2016, 12:15pm UTC](https://discuss.elastic.co/t/mutiple-mutate-filters-and-thier-output-to-different-elasticsearch-indexs/59876/4 "2016-09-06T12:15:50Z")

</div>

> ```
> if [message] == /FilterPerfLogging / {
> 
> ```

Use `=~`, not `==`.

---

<div class="post-metadata">

### Author: ![Ayush\_Rastogi](https://avatars.discourse-cdn.com/v4/letter/a/5daacb/32.png) [@Ayush\_Rastogi](https://discuss.elastic.co/u/Ayush_Rastogi)
#### Post date: [September 7, 2016, 5:10am UTC](https://discuss.elastic.co/t/mutiple-mutate-filters-and-thier-output-to-different-elasticsearch-indexs/59876/5 "2016-09-07T05:10:37Z")

</div>

Thanks Magnus, Working as expected.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:39am UTC](https://discuss.elastic.co/t/mutiple-mutate-filters-and-thier-output-to-different-elasticsearch-indexs/59876/6 "2017-07-06T04:39:36Z")

</div>


