Testing Elastic Stack and winlogbeat / query exceeds 1000 shards

Elastic Stack Elasticsearch
21

It is fine, you got it right. The doc uses localhost because their ES resides on the same machine as Kibana. If your output was incorrect, you would not have been able to create those indices. We need to take a look at "ignore_older".

What is the date time on your ES, Kibana, and Windows event machine?

22

@kernelpanic

I figured out why:

The ignore_older is only set for the Application log in the config you provided. Use the following if you would like a 72h set for each event log.

  event_logs:
    - name: Application
      ignore_older: 72h 
    - name: Security
      ignore_older: 72h
    - name: System
      ignore_older: 72h
23

Thanks, I didn't choose that setting its just the default in the config file. What I can't get my head around is that I've hit a limit in Elasticsearch just by uploading data from one laptop - what happens when I want to collect the event logs from hundreds of machines? There must be something I'm doing wrong.

24

Each shard comes with some overhead in terms of file handles and heap usage, so having a very large number of small shards is very inefficient and can cause problems, as described in this blog post. If you have small data volumes coming in and want to keep it for a long time, as the data you accidentally uploaded, it makes sense to reduce the number of primary shards per index and often also switch from daily to e.g. monthly indices.

25

Thanks for the input; I'm still very new to all this so I'm not quite sure what you mean by daily and monthly indices - does winlogbeat / Elastic stack create an index for each day and therefore are you saying that a monthly index is better as there would be fewer shards?

26

Yes, exactly. To get the most out of your cluster, you want to ensure that your shards are not too small.

27

Ah.... thankyou, I'm getting it now :slight_smile:

The problem was me not understanding indexes properly and simply using the examples in the documentation for a laptop containing 3 years of event log data - you've opened my eyes :slight_smile:
Just out of curiosity, is it possible to use a yearly index?

28

You can certainly use a yearly index. As you probably have relatively little data, that would make perfect sense.

29

Excellent, thankyou both for your time.

Just one last request, can you point me to any resources that advise on what index to use for a particular scenario?

30

I haven't seen such doc. This will probably require ES consulting.

31

No problem Jimmy, thanks again.

32

You're welcome. :smiley:

33

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.