Global ordinals on high cardinality fields with large indices

[YvorL]

Hi,

I've got a daily index with a couple of high cardinality fields. Even after reading some threads and the documentation available, I'm not sure how should I proceed. The index has ~600M docs in it every day and the most requested HC field has ~20M unique values. The index has 6 shards on 8 nodes. I'm trying to lower the general memory usage and I'm not sure if adding eager global ordinals to the affected fields' mapping would help in any way or if I change the execution hint to map would do any harm. Unfortunately, the queries are aggregating those unique values to different term buckets almost every time.
Any suggestions?

Thanks!

[Christian_Dahlqvist]

If daily indices are no longer written to you can reduce memory usage by forcemerging them down to a single segment as described in this webinar. It can take a while and result in a lot of disk I/O though.

[YvorL]

Thank you @Christian_Dahlqvist !
Even if it'd be a segment that's 300GB in size?

[Christian_Dahlqvist]

Is that your shard size??

[YvorL]

Nope, I have 6 shards so one shard would only be around 50-60GB.

[Christian_Dahlqvist]

Then will be a segment per shard, but shards that size will take some time to forcemerge.

[YvorL]

I see, thanks! I'll watch the webinar soon too. Would eager global ordinals help for the actively written shards?

[Christian_Dahlqvist]

I do not think so.