Global ordinals on high cardinality fields with large indices

Elastic Stack Elasticsearch
1

Hi,

I've got a daily index with a couple of high cardinality fields. Even after reading some threads and the documentation available, I'm not sure how should I proceed. The index has ~600M docs in it every day and the most requested HC field has ~20M unique values. The index has 6 shards on 8 nodes. I'm trying to lower the general memory usage and I'm not sure if adding eager global ordinals to the affected fields' mapping would help in any way or if I change the execution hint to map would do any harm. Unfortunately, the queries are aggregating those unique values to different term buckets almost every time.
Any suggestions?

Thanks!

2

If daily indices are no longer written to you can reduce memory usage by forcemerging them down to a single segment as described in this webinar. It can take a while and result in a lot of disk I/O though.

1 Like
3

Thank you @Christian_Dahlqvist !
Even if it'd be a segment that's 300GB in size?

4

Is that your shard size??

5

Nope, I have 6 shards so one shard would only be around 50-60GB.

6

Then will be a segment per shard, but shards that size will take some time to forcemerge.

7

I see, thanks! I'll watch the webinar soon too. Would eager global ordinals help for the actively written shards?

8

I do not think so.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.