How to set shards based on size instead of date?

[robhep]

Hi, I have a 5 node cluster in a lab environment. 2 data nodes, 3 master eligible nodes, 3 kibana, and 2 logstash. It's a low volume setup with about a half a dozen systems feed data to it. It has the default setup with 5 primary and 1 replicate shards per index. However my indices are about 2-8 mb in size, so for about 36 days, I have a ton of shards. How can I reduce the number of primary shards down to 1 primary and 1 replicate?

{
"cluster_name" : "ELK-cluster",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 5,
"number_of_data_nodes" : 2,
"active_primary_shards" : 415,
"active_shards" : 830,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

Thanks,
Robert

[warkolm (Mark Walkom)]

You can update the default Logstash template so that it only uses a single shard, that's probably the easiest way.

[robhep]

Thanks for the suggestion.
I have a Ossec/Wazuh manager server with filebeat sending alerts to Logstash. I'm not sure where to add the line for defining the number of shards, primary and replica. This is the template from the Wazuh group.

cat /etc/logstash/conf.d/01-wazuh.conf
Wazuh - Logstash configuration file
Remote Wazuh Manager - Filebeat input
input {
beats {
port => 5000
codec => "json_lines"
ssl => true
ssl_certificate => "/etc/logstash/logstash.crt"
ssl_key => "/etc/logstash/logstash.key"
}
}
filter {
if [data][srcip] {
mutate {
add_field => [ "@src_ip", "%{[data][srcip]}" ]
}
}
if [data][aws][sourceIPAddress] {
mutate {
add_field => [ "@src_ip", "%{[data][aws][sourceIPAddress]}" ]
}
}
}
filter {
geoip {
source => "@src_ip"
target => "GeoLocation"
fields => ["city_name", "country_name", "region_name", "location"]
}
date {
match => ["timestamp", "ISO8601"]
target => "@timestamp"
}
mutate {
remove_field => [ "timestamp", "beat", "input_type", "tags", "count", "@version", "log", "offset", "type", "@src_ip", "host"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "wazuh-alerts-3.x-%{+YYYY.MM.dd}"
"number_of_shards" : 1 <-- here?
"number_of_replicas" : 1 <-- here?
document_type => "wazuh"
}
}

Link for above from Wazuh Documentation.
curl -so /etc/logstash/conf.d/01-wazuh.conf https://raw.githubusercontent.com/wazuh/wazuh/3.5/extensions/logstash/01-wazuh-remote.conf

Thanks,
Robert

[Christian_Dahlqvist]

If you want to change the number of primary shards and replicas, you need to do so through an https://www.elastic.co/guide/en/elasticsearch/reference/6.4/indices-templates.html. To get even more flexibility and cut indices based on size and/or age, you could switch to using the rollover index API, which is also described in this blog post.

[robhep]

Thanks Christian.