# Custom index name results in temporary bulk send failure **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 31, 2018, 11:51am UTC](https://discuss.elastic.co/t/custom-index-name-results-in-temporary-bulk-send-failure/142355 "2018-07-31T11:51:52Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![roland.otta](https://avatars.discourse-cdn.com/v4/letter/r/c77e96/32.png) [@roland.otta](https://discuss.elastic.co/u/roland.otta)\ **Post date:** [July 31, 2018, 11:51am UTC](https://discuss.elastic.co/t/custom-index-name-results-in-temporary-bulk-send-failure/142355/1 "2018-07-31T11:51:52Z")
Hi, we would like to use filebeats for our kubernetes installation to indexing our container logfiles. as there are many containers running per host we would like to have sepereate indexes for every application. so we setup our filebeat with the option output.elasticsearch: index: k8s-%{[kubernetes.namespace]}-%{[kubernetes.container.name]}-%{+yyyy.MM.dd} without that setting everything works fine, but after setting the custom index nothing is indexed and we are seeing the following error in our logs ``` 2018-07-31T11:37:11.901Z ERROR pipeline/output.go:92 Failed to publish events: temporary bulk send failure ``` no errors on the elasticsearch side. any ideas what could have gone wrong here? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 31, 2018, 11:54am UTC](https://discuss.elastic.co/t/custom-index-name-results-in-temporary-bulk-send-failure/142355/2 "2018-07-31T11:54:13Z")
Elasticsearch requires index names to be lower case. Could that be the problem? ---
**Author:** ![roland.otta](https://avatars.discourse-cdn.com/v4/letter/r/c77e96/32.png) [@roland.otta](https://discuss.elastic.co/u/roland.otta)\ **Post date:** [July 31, 2018, 11:58am UTC](https://discuss.elastic.co/t/custom-index-name-results-in-temporary-bulk-send-failure/142355/3 "2018-07-31T11:58:39Z")
unfortunately not .... our pod names + namespaces are lower case ase well i checked the indexes at es and the indexes got created ... there are also some documents in those indizes .... but at some point it seems that filebeat stops indexing ... there are far to less documents indexed and no new documents are getting indexed ---
**Author:** ![roland.otta](https://avatars.discourse-cdn.com/v4/letter/r/c77e96/32.png) [@roland.otta](https://discuss.elastic.co/u/roland.otta)\ **Post date:** [July 31, 2018, 2:38pm UTC](https://discuss.elastic.co/t/custom-index-name-results-in-temporary-bulk-send-failure/142355/4 "2018-07-31T14:38:06Z")
i did an additional tests with a hardcoded index ``` index: k8s-ttt-%{+yyyy.MM.dd} ``` that works without any problems maybe it is an issue indexing documents in batch that are stored in different es indexes? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 31, 2018, 2:43pm UTC](https://discuss.elastic.co/t/custom-index-name-results-in-temporary-bulk-send-failure/142355/5 "2018-07-31T14:43:02Z")
Well, I am not sure field references work where you are trying to use them. Creating daily indices per namespace and container risk generating a lot of very small shards, which as outlined in [this blog post](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) is very inefficient and is likely to cause you problems down the line. Even if it worked I would therefore recommend against doing that. ---
**Author:** ![roland.otta](https://avatars.discourse-cdn.com/v4/letter/r/c77e96/32.png) [@roland.otta](https://discuss.elastic.co/u/roland.otta)\ **Post date:** [July 31, 2018, 2:53pm UTC](https://discuss.elastic.co/t/custom-index-name-results-in-temporary-bulk-send-failure/142355/6 "2018-07-31T14:53:20Z")
its not an index per container + namespace ... its an index per container-name + namespace - which is basically an index per application (not per running container instance) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 31, 2018, 3:00pm UTC](https://discuss.elastic.co/t/custom-index-name-results-in-temporary-bulk-send-failure/142355/7 "2018-07-31T15:00:24Z")
See how many shards this will generate and consider reducing the number of primary shards and/or the length of time each index covers to avoid getting too many small shards in your cluster. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 28, 2018, 3:00pm UTC](https://discuss.elastic.co/t/custom-index-name-results-in-temporary-bulk-send-failure/142355/8 "2018-08-28T15:00:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.