Closed+hot index or warm+hot index

Elastic Stack Kibana
1

Hello,

I'm administrating an elastic stack at the moment and my current method right now is closing indexes older than 2 weeks and keeping the last 2 weeks in hot state. If people need to check logs older than 2 weeks they contact me and I just reopen them until they're finished. There is a considerable amount of daha flow to the stack and if I don't do reguler cleaning/closing I mostly get request timeouts from kibana. Is it better to change the older logs' states to warm/cold rather than closing? Will I notice a performance boost while querying?

2

If you are already getting request timeouts when searching hot/warm data, then that might indicate your cluster is already under stress. Are you aware that when you reopen a closed index it is reindexing based on the documents? See the docs on closed indices.

Have you considered using frozen indices instead of closing? It would allow occasional requests on-demand, without your intervention.

Have you read the guide on tuning for search speed?

3

Thanks for the valuable info @wylie .

I was wondering something else since I can't seem to find solid numbers anywhere. I think that my shard load can be downsized making logical groups for different indices via logstash.conf. For example let's say i have 100 different indices created each day with 1p and 1r shards and i make groups that include 3 indices each. Each group has its own identified ports and indexnames etc.

This will roughly make 33 logstash instances or 33 pipelines (again this is only a mock scenario). I will start to get a third of my shard load which will increase overall kibana/elasticsearch performance i suppose.

But will it cause logstash to consume much more memory or cpu?

Thanks.

4

I'm a Kibana developer, so I might not be able to help as much as an expert on those particular parts of the stack- feel free to ask again in the other parts of the forum. What I can point you to are some things that commonly cause performance issues:

  1. Having either too many or too few shards
  2. Not using ILM to automate the shard sizing
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.