I was wondering if there is a way to increase the size of the index for Kibana. I have one index that is 2,181,430,251 and it started to error out saying that I was at max size. I want to keep the size going but not sure how.
Lucene has a strict document limit per shard around 2 billion, so you would need to add shards to increase the size. This may require reindexing. Which version are you on?
I am running the latest version of everything. I am currently at 2,181,xxx,xxx records.
How can I add more shards?
Depending on how and when your index was created, you may be able to use the split index API. If that is not an option you will need to reindex your data into an index with a higher number of shards.
Well I have about 250 billion records so I will need to maximize my shards.
How large are your documents? How many indices and shards do you have at the moment? What is the size and specification of your cluster?
Great questions my documents range from 12mbs to 5kb and I have roughly 300,000 files that equals 113gb.
I am running everything standard so I have no adjusted any shards or anything.
I have total of 16TB of space for this.
I ran GET _cat/shards and it came back with this:
XXXXXXX_2 0 p STARTED 70722857 12.9gb X.X.x.x SOMENAME
XXXXXXX_2 0 r UNASSIGNED
XXXX 0 p STARTED 2147483519 300.7gb x.x.x.x SOMENAME
XXXX 0 r UNASSIGNED
.kibana_task_manager 0 p STARTED 2 29.9kb x.x.x. SOMENAME
.kibana_1 0 p STARTED 13 55.7kb x.x.x.x SOMENAME
System specs are
Proc: i7
Ram: 32
Hard Drive: 16TB + 1tb ssd
If 2 billion documents take up 300GB, 250 billion is may require up to over 37TB. I also suspect you might run into issues with heap pressure. I therefore suspect you will need a larger cluster to handle this. You may find this webinar useful.
ok back to the drawing board.
What happens if I create multiple "index" can I merge them all together?
No, but an index can have a lot of shards and you can query across multiple indices, so having a reasonable number of shards should not be a problem. Even though it depends on your data I would recommend keeping shards between 50GB and 100GB in size unless you verify that querying speed and relocation of shards work well at larger sizes.
Ok perfect. Thanks so much for the help I am sure that you will be seeing me around some.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.