# Logstash behaivour when elasticsearch fail to index
**URL:** https://discuss.elastic.co/t/logstash-behaivour-when-elasticsearch-fail-to-index/209904
**Category:** Logstash
**Created:** [November 28, 2019, 6:45pm UTC](https://discuss.elastic.co/t/logstash-behaivour-when-elasticsearch-fail-to-index/209904 "2019-11-28T18:45:40Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@mtudisco](https://discuss.elastic.co/u/mtudisco)
#### Post date: [November 28, 2019, 6:45pm UTC](https://discuss.elastic.co/t/logstash-behaivour-when-elasticsearch-fail-to-index/209904/1 "2019-11-28T18:45:40Z")
Hi,
I'm running a test environment with one node only with elasticsearch and generating indexes with the patern prefix-[type]-yyyy.mm.dd.
Information is generated in logfiles, that are sent with filebeat to logstash to process.
There are different values for [type] so each day i get several indexes.
Todavy i realized that i did not have any information in elasticsearch since yerterday, and looking in logstash logfiles i realized when it send the information to elasticsearch, the later had found max number of shards and where not able to create a new shard.
Then i increased the value of `cluster.max_shards_per_node` and new information coming from filebeat got indexed, however the logs that failed to be indexed during the night where not indexed.
Isnt it supposed that logstash retry if it cannot reach the output? or as in this case the output was reachable but elasticsearch returned error then logstash does not try to index the information again?
thanks
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 28, 2019, 10:11pm UTC](https://discuss.elastic.co/t/logstash-behaivour-when-elasticsearch-fail-to-index/209904/2 "2019-11-28T22:11:37Z")
If elasticsearch was non-responsive I would expect logstash to queue. If elasticsearch returned an error then you would lose events (unless you have a DLQ configured and it is a retryable error).
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [November 29, 2019, 7:02am UTC](https://discuss.elastic.co/t/logstash-behaivour-when-elasticsearch-fail-to-index/209904/3 "2019-11-29T07:02:00Z")
> [@mtudisco](#):
>
> Then i increased the value of `cluster.max_shards_per_node` and new information coming from filebeat got indexed, however the logs that failed to be indexed during the night where not indexed.
Having large number of small shards is inefficient and can cause performance and stability problems. I would recommend reading [this blog post](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) and look to reduce the number of shards in the cluster, e.g. by switching from daily to weekly or monthly indices. The limit is there for a reason and set quite high, so just increasing it is not a great solution.
---
### Author:  [@mtudisco](https://discuss.elastic.co/u/mtudisco)
#### Post date: [November 29, 2019, 12:13pm UTC](https://discuss.elastic.co/t/logstash-behaivour-when-elasticsearch-fail-to-index/209904/4 "2019-11-29T12:13:56Z")
Thanks Badger, what is a DLQ and where do you configure it?
---
### Author:  [@mtudisco](https://discuss.elastic.co/u/mtudisco)
#### Post date: [November 29, 2019, 12:16pm UTC](https://discuss.elastic.co/t/logstash-behaivour-when-elasticsearch-fail-to-index/209904/5 "2019-11-29T12:16:01Z")
Thanks Christian, I have read that document while researching on the original problem and i'm changing the way indexes are being generated to have them monthly, after i removed some indexes i returned the parameter back to 1000.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 29, 2019, 2:29pm UTC](https://discuss.elastic.co/t/logstash-behaivour-when-elasticsearch-fail-to-index/209904/6 "2019-11-29T14:29:43Z")
DLQs are documented [here](https://www.elastic.co/guide/en/logstash/current/dead-letter-queues.html).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 27, 2019, 2:29pm UTC](https://discuss.elastic.co/t/logstash-behaivour-when-elasticsearch-fail-to-index/209904/7 "2019-12-27T14:29:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.