# Output data in multiple indices based on fields

**URL:** <https://discuss.elastic.co/t/output-data-in-multiple-indices-based-on-fields/112303>\
**Category:** Logstash\
**Created:** [December 18, 2017, 8:25pm UTC](https://discuss.elastic.co/t/output-data-in-multiple-indices-based-on-fields/112303 "2017-12-18T20:25:38Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Harsh\_Verma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harsh_verma/32/26136_2.png) [@Harsh\_Verma](https://discuss.elastic.co/u/Harsh_Verma)\
**Post date:** [December 18, 2017, 8:25pm UTC](https://discuss.elastic.co/t/output-data-in-multiple-indices-based-on-fields/112303/1 "2017-12-18T20:25:39Z")

</div>

I have a logfile with some special logs and some general logs.I want the general logs to go in a general index and the special logs(logs which match any of my grok patterns) to go in a separate index and in the general index as well.  
My pipeline conf looks like this:

> input {  
> beats {  
> port =\> "5043"  
> }  
> }  
> filter {  
> grok {  
> match =\> [ "message", "PATTERN1",  
> "message", "PATTERN2"  
> ]  
> }  
> if "\_grokparsefailure" in [tags] {  
> mutate {  
> add\_field =\> {"[@metadata][index]" =\> "generallogs"}  
> }  
> }  
> else{  
> mutate {  
> add\_field =\> {"[@metadata][index]" =\> "speciallogs"}  
> }  
> }  
> }  
> output {  
> elasticsearch {  
> hosts =\> ["localhost:9200"]  
> index =\> "%{[@metadata][index]}"  
> }  
> }

The only problem with this is that special logs go only in the special log index and not in the general log index. Is there a solution ? Thanks !

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 18, 2017, 9:12pm UTC](https://discuss.elastic.co/t/output-data-in-multiple-indices-based-on-fields/112303/2 "2017-12-18T21:12:30Z")

</div>

Make the output to the other index conditional using something like:

```
output {

elasticsearch {
hosts => ["localhost:9200"]
index => "generallogs"
}

if "_grokparsefailure" not in [tags] {
elasticsearch {
hosts => ["localhost:9200"]
index => "speciallogs"
}
}

}
```

---

<div class="post-metadata">

**Author:** ![Harsh\_Verma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/harsh_verma/32/26136_2.png) [@Harsh\_Verma](https://discuss.elastic.co/u/Harsh_Verma)\
**Post date:** [December 18, 2017, 9:37pm UTC](https://discuss.elastic.co/t/output-data-in-multiple-indices-based-on-fields/112303/3 "2017-12-18T21:37:18Z")

</div>

Makes perfect sense. Don't know why I was not able to think about this !

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 15, 2018, 9:37pm UTC](https://discuss.elastic.co/t/output-data-in-multiple-indices-based-on-fields/112303/4 "2018-01-15T21:37:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.