# Create Dynamic Index in Elasticsearch **URL:** \ **Category:** Logstash\ **Created:** [November 6, 2017, 7:22pm UTC](https://discuss.elastic.co/t/create-dynamic-index-in-elasticsearch/106579 "2017-11-06T19:22:06Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\ **Post date:** [November 6, 2017, 7:22pm UTC](https://discuss.elastic.co/t/create-dynamic-index-in-elasticsearch/106579/1 "2017-11-06T19:22:06Z")
I am trying to configure logstash to create dynamic indexes in Elasticsearch. Here are my configuration ``` input{ kafka { bootstrap_servers => ["192.168.56.2:9092,192.168.56.23:9092,192.168.56.24:9092"] topics => ["logstash", "qafilebeat", "prfilebeat", "app-error-logs"] auto_offset_reset => latest decorate_events => true group_id => "logstash_indexer" } } filter { json { source => "message" } } output { elasticsearch { hosts => ["192.168.56.5"] } } ``` Here are sample logs ``` {"log":"Nov 06, 2017 6:50:12 PM org.glassfish.jersey.internal.Errors logErrors\n","stream":"stderr","attrs":{"APPLICATION":"app1","DATACENTER":"ny","ENVIRONMENT":"qa12","com.cluster":"qa12-app1","com.container-name":"app1-2_33_1","time":"2017-11-06T18:50:12.578546934Z"} {"log":"WARNING: The following warnings have been detected: WARNING: The (sub)resource method postTaxeneConfig in com.app1.server.taxene.config.connector.jersey.TaxeneConfigResource contains empty path annotation.\n","stream":"stderr","attrs":{"APPLICATION":"app1","DATACENTER":"NY","ENVIRONMENT":"qa12","com.cluster":"qa12-app1"},"time":"2017-11-06T18:50:12.57858747Z"} ``` ​It is creating only logstash-\* index in Elasticsearch. But I want to create -\* where Environment could be prod or qa, Application be app1, app2 etc and cluster be a name. Any help would be highly appreciated​. Cheers Ferdous Shibly ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 6, 2017, 8:42pm UTC](https://discuss.elastic.co/t/create-dynamic-index-in-elasticsearch/106579/2 "2017-11-06T20:42:46Z")
Example: ```nohighlight elasticsearch { ... index => "%{[attrs][ENVIRONMENT]}-%{[attrs][APPLICATION]}-%{+YYYY.MM.dd}" } ``` ---
**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\ **Post date:** [November 6, 2017, 9:45pm UTC](https://discuss.elastic.co/t/create-dynamic-index-in-elasticsearch/106579/3 "2017-11-06T21:45:41Z")
Thanks Magnus Bäck, it is working. I have one more quire. How can I send logs to different Elasticsearch cluster based upon fields? I want to send qa application logs to qa Elasticsearch cluster and prod logs to prod Elasticsearch cluster. like ``` output { if [attrs][ENVIRONMENT] =~ /(qa|dv|ee|pe)\d+/ { elasticsearch { hosts => ["192.168.56.5"] index => "%{[attrs][ENVIRONMENT]}%{[attrs][APPLICATION]}-%{+YYYY.MM.dd}" } }else { elasticsearch { hosts => ["192.168.56.10"] index => "%{[attrs][ENVIRONMENT]}%{[attrs][APPLICATION]}-%{+YYYY.MM.dd}" } } } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 6, 2017, 9:49pm UTC](https://discuss.elastic.co/t/create-dynamic-index-in-elasticsearch/106579/4 "2017-11-06T21:49:52Z")
Yeah, that's what it would look like. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 6, 2017, 10:53pm UTC](https://discuss.elastic.co/t/create-dynamic-index-in-elasticsearch/106579/5 "2017-11-06T22:53:03Z")
> [@Ferdous\_Shibly](#): > > index =\> "%{[attrs][ENVIRONMENT]}%{[attrs][APPLICATION]}-%{+YYYY.MM.dd}" If you are doing this then it probably makes sense to change your indices to monthly and/or reduce the shard count from the default 5. ---
**Author:** ![Ferdous\_Shibly](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ferdous_shibly/32/23470_2.png) [@Ferdous\_Shibly](https://discuss.elastic.co/u/Ferdous_Shibly)\ **Post date:** [November 7, 2017, 10:01pm UTC](https://discuss.elastic.co/t/create-dynamic-index-in-elasticsearch/106579/6 "2017-11-07T22:01:59Z")
Magnus, How can I check whether the field is emply or not? like in this case, if [attrs][APPLICATION] is empty or not present or if [attrs][APPLICATION] is generic, how can I send the logs to qageneric index other with it will create appropriate indexes? ``` output { if [attrs][ENVIRONMENT] =~ /(qa*|dv*|ee*|pe*)\d+/ { if [attrs][APPLICATION] =~ /generic/ or ![attrs][APPLICATION] { elasticsearch { hosts => ["192.168.56.5"] index => "%{[attrs][ENVIRONMENT]}-generic-%{+YYYY.MM.dd}" } } } } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 7, 2017, 10:20pm UTC](https://discuss.elastic.co/t/create-dynamic-index-in-elasticsearch/106579/7 "2017-11-07T22:20:39Z")
> ``` > if [attrs][APPLICATION] =~ /generic/ or ![attrs][APPLICATION] { > > ``` This should work except that the negation operator is `not` rather than `!`. > **[Accessing event data and fields | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals)** ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 5, 2017, 10:21pm UTC](https://discuss.elastic.co/t/create-dynamic-index-in-elasticsearch/106579/8 "2017-12-05T22:21:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.