Winlogbeat many shards

[brascon01 (tony)]

Hi All,

i am new with ELK, i have a test cluster with single data node, i have configured one server to use winlogbeat now i am receiving a lots of small shards, i encountered similar issues as the below thread, i am trying to figure out what is best to change from a daily indexing to monthly or yearly index. Can someone please guide me to it.

Testing Elastic Stack and winlogbeat / query exceeds 1000 shards

@kernelpanic I figured out why: The ignore_older is only set for the Application log in the config you provided. Use the following if you would like a 72h set for each event log. event_logs: - name: Application ignore_older: 72h - name: Security ignore_older: 72h - name: System ignore_older: 72h

here is my cluster health:

"cluster_name" : "Diamond",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1682,
"active_shards" : 1682,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1682,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
Many thanks

[spinscale (Alexander Reelsen)]

Hey,

you can use the elasticsearch output configuration of beats to specify the index name incuding a timespace... see https://www.elastic.co/guide/en/beats/libbeat/current/config-file-format-namespacing.html

--Alex

[brascon01 (tony)]

Thank you Alex, appreciate your help, my current logstash beat conf output is:
output {
elasticsearch {
hosts => "192.168.1.1.10:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

document_type => "%{[@metadata][type]}"

}

do i have to change it to the below to perform a weekly indexing:

output {
elasticsearch {
hosts => "192.168.1.1.10:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.ww}"

document_type => "%{[@metadata][type]}"

}

[brascon01 (tony)]

Maybe this will help someone:

use this

index => "%{[@metadata][beat]}-%{+yyyy.ww}"

instead of

index => "%{[@metadata][beat]}-%{+YYYY.MM.ww}"

green open winlogbeat-2017.11