Index pattern type in Elasticsearch

Elastic Stack Elasticsearch
1

HI Team,
I would like to take some suggestion to create index pattern in ELK.

As per my current configuration in production, i'm index data in ES by monthly index with 1 Primary Shard and 2 replicas.

Due to that i was faced below in ES,

" Elasticsearch throwing number of documents in the index cannot exceed 2147483519"

To avoid such issue i would to break monthly into weekly or Day basis.

I need your input which approach is best approach in production.

Looking forward your inputs on this query

2

There are a lot of factors at play.

  1. You could increase the number of primary shards for your index, which would allow you 2^31 documents per primary shard.
  2. You could goto a weekly index ~4 times the amount of shards you currently have per month
  3. You could goto a daily index ~30 times the amount of shards you currently have per month

Which is best is going to be based on many factors, including but not limited to:
Your cluster architecture, number of nodes, indexing pressure, read pressure, number of cpus available, amount of heap and ram available, speed of disks... The best approach would be to test, test, and test again.

3

I have below setup,

3 master node, 3 data node

total        used        free      shared  buff/cache   available
Mem:       65808032    35851440      367148        7704    29589444    29416536
Swap:       2097148      586240     1510908

How to increase the shards value in index

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.