# Testing Elastic Stack and winlogbeat / query exceeds 1000 shards **URL:** \ **Category:** Elasticsearch\ **Created:** [March 9, 2017, 1:46am UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923 "2017-03-09T01:46:17Z")\ **Posts on this page:** 13\ **Page:** 2
**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\ **Post date:** [March 10, 2017, 3:41pm UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/21 "2017-03-10T15:41:01Z")
It is fine, you got it right. The doc uses localhost because their ES resides on the same machine as Kibana. If your output was incorrect, you would not have been able to create those indices. We need to take a look at "ignore\_older". What is the date time on your ES, Kibana, and Windows event machine? ---
**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\ **Post date:** [March 10, 2017, 3:56pm UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/22 "2017-03-10T15:56:19Z")
@kernelpanic I figured out why: The ignore\_older is only set for the Application log in the config you provided. Use the following if you would like a 72h set for each event log. ```auto event_logs: - name: Application ignore_older: 72h - name: Security ignore_older: 72h - name: System ignore_older: 72h ``` ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [March 10, 2017, 4:31pm UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/23 "2017-03-10T16:31:16Z")
Thanks, I didn't choose that setting its just the default in the config file. What I can't get my head around is that I've hit a limit in Elasticsearch just by uploading data from one laptop - what happens when I want to collect the event logs from hundreds of machines? There must be something I'm doing wrong. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 10, 2017, 4:56pm UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/24 "2017-03-10T16:56:34Z")
Each shard comes with some overhead in terms of file handles and heap usage, so having a very large number of small shards is very inefficient and can cause problems, as described in [this blog post](https://www.elastic.co/blog/found-crash-elasticsearch). If you have small data volumes coming in and want to keep it for a long time, as the data you accidentally uploaded, it makes sense to reduce the number of primary shards per index and often also switch from daily to e.g. monthly indices. ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [March 10, 2017, 5:07pm UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/25 "2017-03-10T17:07:30Z")
Thanks for the input; I'm still very new to all this so I'm not quite sure what you mean by daily and monthly indices - does winlogbeat / Elastic stack create an index for each day and therefore are you saying that a monthly index is better as there would be fewer shards? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 10, 2017, 5:15pm UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/26 "2017-03-10T17:15:09Z")
Yes, exactly. To get the most out of your cluster, you want to ensure that your shards are not too small. ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [March 10, 2017, 5:18pm UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/27 "2017-03-10T17:18:09Z")
Ah.... thankyou, I'm getting it now 🙂 The problem was me not understanding indexes properly and simply using the examples in the documentation for a laptop containing 3 years of event log data - you've opened my eyes 🙂 Just out of curiosity, is it possible to use a yearly index? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 10, 2017, 5:20pm UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/28 "2017-03-10T17:20:02Z")
You can certainly use a yearly index. As you probably have relatively little data, that would make perfect sense. ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [March 10, 2017, 5:24pm UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/29 "2017-03-10T17:24:32Z")
Excellent, thankyou both for your time. Just one last request, can you point me to any resources that advise on what index to use for a particular scenario? ---
**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\ **Post date:** [March 11, 2017, 12:19am UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/30 "2017-03-11T00:19:04Z")
I haven't seen such doc. This will probably require ES consulting. ---
**Author:** ![kernelpanic](https://avatars.discourse-cdn.com/v4/letter/k/c89c15/32.png) [@kernelpanic](https://discuss.elastic.co/u/kernelpanic)\ **Post date:** [March 11, 2017, 12:47am UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/31 "2017-03-11T00:47:44Z")
No problem Jimmy, thanks again. ---
**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\ **Post date:** [March 11, 2017, 3:35am UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/32 "2017-03-11T03:35:47Z")
You're welcome. 😃 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 8, 2017, 3:36am UTC](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923/33 "2017-04-08T03:36:21Z")
Closed on April 8, 2017, 3:36am UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/testing-elastic-stack-and-winlogbeat-query-exceeds-1000-shards/77923.md?page=1)