Closed+hot index or warm+hot index


I'm administrating an elastic stack at the moment and my current method right now is closing indexes older than 2 weeks and keeping the last 2 weeks in hot state. If people need to check logs older than 2 weeks they contact me and I just reopen them until they're finished. There is a considerable amount of daha flow to the stack and if I don't do reguler cleaning/closing I mostly get request timeouts from kibana. Is it better to change the older logs' states to warm/cold rather than closing? Will I notice a performance boost while querying?

If you are already getting request timeouts when searching hot/warm data, then that might indicate your cluster is already under stress. Are you aware that when you reopen a closed index it is reindexing based on the documents? See the docs on closed indices.

Have you considered using frozen indices instead of closing? It would allow occasional requests on-demand, without your intervention.

Have you read the guide on tuning for search speed?

Thanks for the valuable info @wylie .

I was wondering something else since I can't seem to find solid numbers anywhere. I think that my shard load can be downsized making logical groups for different indices via logstash.conf. For example let's say i have 100 different indices created each day with 1p and 1r shards and i make groups that include 3 indices each. Each group has its own identified ports and indexnames etc.

This will roughly make 33 logstash instances or 33 pipelines (again this is only a mock scenario). I will start to get a third of my shard load which will increase overall kibana/elasticsearch performance i suppose.

But will it cause logstash to consume much more memory or cpu?


I'm a Kibana developer, so I might not be able to help as much as an expert on those particular parts of the stack- feel free to ask again in the other parts of the forum. What I can point you to are some things that commonly cause performance issues:

  1. Having either too many or too few shards
  2. Not using ILM to automate the shard sizing

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.