# Multiple Elasticsearch Indices in Logstash output **URL:** \ **Category:** Elasticsearch\ **Created:** [July 31, 2015, 2:01am UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607 "2015-07-31T02:01:49Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![shadowman13](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@shadowman13](https://discuss.elastic.co/u/shadowman13)\ **Post date:** [July 31, 2015, 2:01am UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/1 "2015-07-31T02:01:49Z")
Hi all, Currently I'm using Elasticsearch auto clear index script which find all the indexes with format logstash-%{+YYYY.MM.dd} and clear all the indices older than 10 days, but for some reason I need to exclude a specific log type. So I tried to change the elasticsearch's \_index value in logstash, I have found a way to do it, but it made elasticsearch to generate new node in cluster which decrease the performance, so I don't want to use it. output { if [Type] == "IndexType1" { elasticsearch { host =\> localhost cluster =\> elasticsearch index =\> "IndexType1-%{+YYYY.MM.dd}" } } else { elasticsearch { host =\> localhost cluster =\> elasticsearch } } So I have tried another way on Goolge like below to filter the new field on output but so far not success: filter { if [Type] == "IndexType1" { grok { match =\> { "message" =\> "%{Pattern}" } mutate { add\_field =\> { "IndexType" =\> "IndexType1" } } } } output { elasticsearch { host =\> localhost cluster =\> elasticsearch index =\> "logstash%{IndexType}-%{+YYYY.MM.dd}" } But it's not working the \_index value displayed on Kibana still not changed. Even the field IndexType was created. ![](https://us1.discourse-cdn.com/elastic/original/2X/b/b67e95056de6b08adebec3480dc03fa70bc23225.png) I don't know if I did it correctly. I'm new to logstash, elasticsearch and kibana.So would you please help me to correct the code or advice me if there's any other solutions. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 31, 2015, 7:50am UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/2 "2015-07-31T07:50:44Z")
> So I tried to change the elasticsearch's \_index value in logstash, I have found a way to do it, but it made elasticsearch to generate new node in cluster which decrease the performance, so I don't want to use it. What do you mean by "generate new node in cluster"? Creating an additional index each day will impact performance but I doubt it makes a very big impact. Any method of having different retention policies for different documents is going to impact performance in some way. ---
**Author:** ![shadowman13](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@shadowman13](https://discuss.elastic.co/u/shadowman13)\ **Post date:** [August 1, 2015, 10:27am UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/3 "2015-08-01T10:27:37Z")
Hi magnusbaeck, It's mean currenly my elasticsearch cluster has 2 hosts and 4 nodes, when I add the config: if [Type] == "IndexType1" { elasticsearch { host =\> localhost cluster =\> elasticsearch index =\> "IndexType1-%{+YYYY.MM.dd}" } Two more nodes appear in the cluster. I don't want that. Is there any other way to use dynamic indices instead? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 1, 2015, 6:40pm UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/4 "2015-08-01T18:40:36Z")
Indexes and cluster nodes are completely unrelated concepts. I don't know why you're talking about indexes here. If you don't want Logstash to show up as a cluster node, don't use the node protocol but instead the transport protocol or HTTP. Use the [`protocol`](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-protocol) option to change this. See also _[Transport Client Versus Node Client](https://www.elastic.co/guide/en/elasticsearch/guide/current/_transport_client_versus_node_client.html)_ in the Definitive Guide. ---
**Author:** ![shadowman13](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@shadowman13](https://discuss.elastic.co/u/shadowman13)\ **Post date:** [August 2, 2015, 1:19pm UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/5 "2015-08-02T13:19:38Z")
Thank so much for clarifying! Actually my main question is simple: Is there any other way to change the \_index default value ![](https://us1.discourse-cdn.com/elastic/original/2X/b/b47a8f0cee5672c1104c10bb45b65df8e1f195bf.png) Instead of using "index =\> "logstash%{IndexType}-%{+YYYY.MM.dd}" in the output sector? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 2, 2015, 4:11pm UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/6 "2015-08-02T16:11:52Z")
Not that I'm aware of. Why would you want to do that? ---
**Author:** ![shadowman13](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@shadowman13](https://discuss.elastic.co/u/shadowman13)\ **Post date:** [August 3, 2015, 1:37am UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/7 "2015-08-03T01:37:08Z")
Hi magnusbaeck, As describe from begin: Currently I'm using Elasticsearch auto clear index script which find all the indexes base on format logstash-%{+YYYY.MM.dd} and clear all older than 10 days, but I want to exclude a specific log type. Ex: I have apache log, nginx log, postfix log. Now I want to change the \_indexe logstash-%{+YYYY.MM.dd} value of postfix log so it wont be clean by the script. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 3, 2015, 6:12am UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/8 "2015-08-03T06:12:12Z")
Yes, but what's the problem with setting the `index` option? If it's not working it's because you're not setting the `IndexType` field correctly. Looking at your previously posted configuration, ``` filter { if [Type] == "IndexType1" { grok { match => { "message" => "%{Pattern}" } mutate { add_field => { "IndexType" => "IndexType1" } } } } output { elasticsearch { host => localhost cluster => elasticsearch index => "logstash%{IndexType}-%{+YYYY.MM.dd}" } } ``` I'd start by changing `[Type]` to `[type]`. I'd also change the index name pattern to `logstash-%{IndexType}-%{+YYYY.MM.dd}` (note hyphen after "logstash") to make sure that the index template still applies (I think it by default applies to indexes whose names match logstash-\*). ---
**Author:** ![shadowman13](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@shadowman13](https://discuss.elastic.co/u/shadowman13)\ **Post date:** [August 4, 2015, 2:46am UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/9 "2015-08-04T02:46:52Z")
Sorry for slow reply! Thank you so much for your advice! I will try your method with logstash-%{IndexType}-%{+YYYY.MM.dd}, hope it should affect on the IndexType1 only. ---
**Author:** ![shadowman13](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@shadowman13](https://discuss.elastic.co/u/shadowman13)\ **Post date:** [August 10, 2015, 4:45am UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/10 "2015-08-10T04:45:22Z")
Hi magnus, Changing to logstash-%{IndexType}-%{+YYYY.MM.dd} didn't work also. Seems like there's no other way to change the \_index. Thank for all your helps ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 10, 2015, 5:31am UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/11 "2015-08-10T05:31:52Z")
Please show the configuration you're using. Also, please prove that the `IndexType` field is getting the correct value, e.g. by using the stdout output to dump the contents of a message at the end of the pipeline. ---
**Author:** ![shadowman13](https://avatars.discourse-cdn.com/v4/letter/s/b38774/32.png) [@shadowman13](https://discuss.elastic.co/u/shadowman13)\ **Post date:** [April 22, 2016, 8:55am UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/12 "2016-04-22T08:55:13Z")
Thank you magnusbaeck! The issue solved when I upgrade to new logstash. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:57pm UTC](https://discuss.elastic.co/t/multiple-elasticsearch-indices-in-logstash-output/26607/13 "2017-07-05T22:57:04Z")