# My logstash output is creating hundreds of shards per index **URL:** \ **Category:** Elasticsearch\ **Created:** [March 18, 2016, 7:46am UTC](https://discuss.elastic.co/t/my-logstash-output-is-creating-hundreds-of-shards-per-index/44799 "2016-03-18T07:46:27Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![dave\_IT\_guy](https://avatars.discourse-cdn.com/v4/letter/d/a8b319/32.png) [@dave\_IT\_guy](https://discuss.elastic.co/u/dave_IT_guy)\ **Post date:** [March 18, 2016, 7:46am UTC](https://discuss.elastic.co/t/my-logstash-output-is-creating-hundreds-of-shards-per-index/44799/1 "2016-03-18T07:46:27Z")
Hi all, I am setting up a new ELK system to centralize Windows Event logs and in the future syslogs etc. The shipping of the logs using winlogbeat is working fine however when I look at my index for logstash-ddmmyyy I find I have hundreds of shards being created and the performance of the system makes it unusable. I would be expecting 5 shards per index and a new index each day. I have cleared all of my indexes this morning to start fresh and currently have about 30 Windows Servers sending logs into the system. It has currently been running for 15 minutes and my cluster health is: { "cluster\_name" : "elasticsearch", "status" : "yellow", "timed\_out" : false, "number\_of\_nodes" : 1, "number\_of\_data\_nodes" : 1, "active\_primary\_shards" : 1111, "active\_shards" : 1111, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 1111, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 50.0 } My config file for logstash to Elastic Search is: input { beats { port =\> 5044 ssl =\> true ssl\_certificate =\> "/etc/pki/tls/certs/logstash-forwarder.crt" ssl\_key =\> "/etc/pki/tls/private/logstash-forwarder.key" } } # Filter syslog events from filebeat filter { if [type] == "syslog" { grok { match =\> { "message" =\> "%{SYSLOGTIMESTAMP:syslog\_timestamp} %{SYSLOGHOST:syslog\_hostname} %{DATA:syslog\_program}(?:[%{POSINT:syslog\_pid}])?: %{GREEDYDATA:syslog\_message}" } add\_field =\> ["received\_at", "%{@timestamp}"] add\_field =\> ["received\_from", "%{host}"] } syslog\_pri { } date { match =\> ["syslog\_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"] } } } output { elasticsearch { hosts =\> ["localhost:9200"] sniffing =\> true manage\_template =\> false index =\> "logstash-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" } } I am clearly making a mistake and would be grateful if someone could help me out. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 19, 2016, 3:32pm UTC](https://discuss.elastic.co/t/my-logstash-output-is-creating-hundreds-of-shards-per-index/44799/2 "2016-03-19T15:32:25Z")
Please show the output of `curl localhost:9200/_cat/indices`. ---
**Author:** ![dave\_IT\_guy](https://avatars.discourse-cdn.com/v4/letter/d/a8b319/32.png) [@dave\_IT\_guy](https://discuss.elastic.co/u/dave_IT_guy)\ **Post date:** [March 21, 2016, 10:55am UTC](https://discuss.elastic.co/t/my-logstash-output-is-creating-hundreds-of-shards-per-index/44799/3 "2016-03-21T10:55:11Z")
Hi Magnus, I started everything fresh just now and this is what has happened. It looks like logstash is receiving data from a long time ago instead of the past couple of days. Would you agree? Output: green open logstash-2016.03.18 5 1 9203 0 12.5mb 5.5mb green open logstash-2015.05.28 5 1 26 0 119.1kb 59.5kb green open logstash-2016.03.19 5 1 17090 0 14.6mb 6.6mb green open logstash-2015.11.26 5 1 87 0 207.3kb 98.8kb green open logstash-2015.11.27 5 1 126 0 228kb 99.4kb green open logstash-2015.11.28 5 1 13 0 105.4kb 52.7kb green open logstash-2015.11.29 5 1 40 0 165kb 68.2kb green open logstash-2015.11.30 5 1 87 0 198.3kb 89.8kb green open logstash-2015.09.30 5 1 940 0 930.2kb 414.7kb green open logstash-2015.12.01 5 1 68 0 194.5kb 82.9kb green open logstash-2015.12.02 5 1 82 0 186.7kb 93.3kb green open logstash-2016.03.21 5 1 8899 0 15.4mb 8mb green open logstash-2016.03.20 5 1 18150 0 15.1mb 6.3mb green open logstash-2016.01.01 5 1 40 0 131.2kb 65.6kb green open logstash-2016.01.04 5 1 57 0 136.7kb 68.3kb green open logstash-2016.01.05 5 1 43 0 137.3kb 63.9kb green open logstash-2016.01.02 5 1 19 0 110.5kb 55.2kb green open logstash-2016.01.03 5 1 17 0 109kb 54.5kb green open logstash-2016.01.08 5 1 103 0 217.9kb 108.9kb green open logstash-2016.01.09 5 1 63 0 138.8kb 69.4kb green open logstash-2016.01.06 5 1 99 0 193.7kb 96.8kb green open logstash-2016.01.07 5 1 116 0 218.9kb 95kb yellow open logstash-2015.12.27 5 1 71 0 136.9kb 126.5kb green open logstash-2015.05.30 5 1 193 0 95.2kb 23.3kb green open logstash-2015.10.03 5 1 667 0 661.8kb 315kb green open logstash-2015.12.28 5 1 23 0 113.1kb 56.5kb green open logstash-2015.10.04 5 1 655 0 607.9kb 304.3kb yellow open logstash-2015.10.01 5 1 766 0 635.2kb 347.7kb yellow open logstash-2015.11.13 5 1 0 0 780b 260b green open logstash-2015.12.25 5 1 37 0 141.5kb 84kb green open logstash-2015.10.02 5 1 661 0 655kb 310.4kb green open logstash-2015.12.26 5 1 68 0 198kb 140.7kb green open logstash-2015.10.05 5 1 490 0 91.9kb 40.8kb green open logstash-2015.12.29 5 1 25 0 115kb 57.5kb green open logstash-2016.12.22 5 1 1 0 28.9kb 14.4kb green open logstash-2015.12.30 5 1 28 0 117.3kb 58.6kb green open logstash-2015.12.31 5 1 52 0 155.6kb 77.8kb green open logstash-2015.09.20 5 1 0 0 34.2kb 260b green open logstash-2016.12.15 5 1 1 0 29.1kb 14.5kb yellow open logstash-2015.09.21 5 1 0 0 10.3kb 260b green open logstash-2015.09.25 5 1 694 0 856.9kb 404.5kb green open logstash-2015.09.26 5 1 858 0 844.7kb 394kb green open logstash-2015.09.27 5 1 851 0 823kb 397.2kb green open logstash-2015.09.28 5 1 881 0 686.2kb 343.1kb green open logstash-2015.09.29 5 1 860 0 835.3kb 358.8kb green open logstash-2016.01.11 5 1 97 0 197.5kb 84kb green open logstash-2016.01.12 5 1 88 0 175.5kb 78.1kb green open logstash-2015.05.29 5 1 481 0 649.7kb 324.8kb green open logstash-2016.01.10 5 1 61 0 137.5kb 68.7kb green open logstash-2016.01.15 5 1 104 0 45.4kb 17.7kb green open logstash-2016.02.27 5 1 1 0 29.2kb 14.6kb green open logstash-2016.01.16 5 1 74 0 59.7kb 14.1kb green open logstash-2016.01.13 5 1 125 0 71.5kb 27.6kb green open logstash-2016.01.14 5 1 94 0 80.4kb 15.7kb green open logstash-2016.01.19 5 1 0 0 1kb 260b green open logstash-2016.01.17 5 1 0 0 12.2kb 260b green open logstash-2016.01.18 5 1 0 0 24.2kb 260b yellow open logstash-2015.12.16 5 1 422 0 650.8kb 400.1kb yellow open logstash-2015.12.17 5 1 706 0 656.6kb 416kb yellow open logstash-2015.12.14 5 1 60 0 231.8kb 180.3kb green open logstash-2015.12.15 5 1 124 0 339.6kb 169.8kb green open logstash-2015.11.09 5 1 23 0 120.3kb 60.1kb green open logstash-2015.12.18 5 1 82 0 318.8kb 148.4kb green open logstash-2015.12.19 5 1 60 0 271.2kb 135.6kb green open logstash-2015.12.20 5 1 56 0 233.6kb 116.8kb yellow open logstash-2015.12.23 5 1 208 0 251.7kb 208.7kb green open .kibana 1 1 2 0 21.6kb 10.8kb green open logstash-2015.11.11 5 1 416 0 552.6kb 275.1kb yellow open logstash-2015.12.24 5 1 71 0 176.6kb 125.6kb green open logstash-2015.09.01 5 1 236 0 294.2kb 122.8kb green open logstash-2015.12.10 5 1 103 0 304.7kb 152.3kb green open logstash-2016.11.29 5 1 1 0 29kb 14.5kb green open logstash-2015.12.11 5 1 85 0 378.4kb 186.7kb green open logstash-2015.09.02 5 1 578 0 611.9kb 276.6kb green open logstash-2015.09.03 5 1 335 0 83.7kb 260b ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 21, 2016, 11:43am UTC](https://discuss.elastic.co/t/my-logstash-output-is-creating-hundreds-of-shards-per-index/44799/4 "2016-03-21T11:43:03Z")
Yes. Indexes are created based on the `@timestamp` field, so either you are indeed reading old log data and then the number of indexes is to be expected, or your date filter is parsing timestamps incorrectly. Either way you should probably rethink having five shards per index. Unless you're logging 100 GB/day you don't need five shards. ---
**Author:** ![dave\_IT\_guy](https://avatars.discourse-cdn.com/v4/letter/d/a8b319/32.png) [@dave\_IT\_guy](https://discuss.elastic.co/u/dave_IT_guy)\ **Post date:** [March 21, 2016, 12:53pm UTC](https://discuss.elastic.co/t/my-logstash-output-is-creating-hundreds-of-shards-per-index/44799/5 "2016-03-21T12:53:32Z")
Thanks Magnus, Nowhere near 100GB/day so I will drop the 5 shards down to... 2? I will have to understand that better before I make a decision. Happy to listen to any suggestion you may have. Thanks for all your help. ---
**Author:** ![seidler2547](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/seidler2547/32/8570_2.png) [@seidler2547](https://discuss.elastic.co/u/seidler2547)\ **Post date:** [March 21, 2016, 1:02pm UTC](https://discuss.elastic.co/t/my-logstash-output-is-creating-hundreds-of-shards-per-index/44799/6 "2016-03-21T13:02:59Z")
In my experience, winlogbeats is reading the whole Windows event log, which reaches back as far as some month. I had this issue as well, and I tried setting, the parameter to ignore old data, but it wasn't working for me. I eventually accepted the fact and then consolidated the old events (they get more sparse as they get older) into month-based indices using the Python helper library (using reindex). ---
**Author:** ![dave\_IT\_guy](https://avatars.discourse-cdn.com/v4/letter/d/a8b319/32.png) [@dave\_IT\_guy](https://discuss.elastic.co/u/dave_IT_guy)\ **Post date:** [March 21, 2016, 1:29pm UTC](https://discuss.elastic.co/t/my-logstash-output-is-creating-hundreds-of-shards-per-index/44799/7 "2016-03-21T13:29:48Z")
Hi Stefan, I'm seeing exactly what you described. I need to sort out the index as suggested by Magnus and then possibly curator to deal with old information that's coming in. Like you I tried the parameter in the winlogbeat.yml file however it had no affect. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 21, 2016, 2:40pm UTC](https://discuss.elastic.co/t/my-logstash-output-is-creating-hundreds-of-shards-per-index/44799/8 "2016-03-21T14:40:19Z")
> Nowhere near 100GB/day so I will drop the 5 shards down to... 2? I will have to understand that better before I make a decision. Happy to listen to any suggestion you may have. There are some good chapters in _Elasticsearch: The Definitive Guide_ that talk about scaling, shard sizees, etc. It regularly comes up here too. I don't have the whole picture but I suspect a single shard will be fine in your case. ---
**Author:** ![dave\_IT\_guy](https://avatars.discourse-cdn.com/v4/letter/d/a8b319/32.png) [@dave\_IT\_guy](https://discuss.elastic.co/u/dave_IT_guy)\ **Post date:** [March 21, 2016, 2:53pm UTC](https://discuss.elastic.co/t/my-logstash-output-is-creating-hundreds-of-shards-per-index/44799/9 "2016-03-21T14:53:47Z")
Thanks Magnus. I am working through each chapter so will certainly learn all this in the time to come. In the meantime I really appreciate your assistance. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:06pm UTC](https://discuss.elastic.co/t/my-logstash-output-is-creating-hundreds-of-shards-per-index/44799/10 "2017-07-05T23:06:33Z")