I have an issue with Curator that I would like an opinion on. This is elk 6.2.0 with curator 5.4, but also happens on 6.1.2. I hope I am missing somethng obvious.
I have a cluster , that implements the hot/warm architecture. I am using the roll-over approach of handling the indexes- After 1 day , I migrate the indixes to the warm node. This works fine.
After 1 more day, I want to shrink and forcemerge. I want this to take place on my warm nodes and I want the indexes to remain on the warm nodes.
Curator picks one of my warm nodes as shrink node, moves the shards to the shrink node.Shrinks the index to 1 shard. After this is complete the index is moved to the HOT node. Then curator starts the post allocation step, which moves the index back to the WARM node.
So obviously, this is taking much longer than it should.
My skrink step looks like this at the moment:
action: shrink options: ignore_empty_list : true shrink_node: DETERMINISTIC node_filters: permit_masters: False number_of_shards: 1 number_of_replicas: 0 shrink_suffix: '-shrink' delete_after: false post_allocation: allocation_type: require key: box_type value: warm wait_for_active_shards: 1 extra_settings: settings: index.codec: best_compression index.routing.allocation.require.box_type: warm wait_for_completion: True wait_interval: 30 max_wait: -1 filters: - filtertype: count count: 2 pattern: '^(dns-active)-\d+$' reverse: true exclude: true - filtertype: age source: field_stats direction: older unit: days unit_count: 2 field: '@timestamp' stats_result: min_value
I have been trying to fix this with index templates , but no luck so far.
Anything I can do to avoid this copying of the index back to the HOT node and then back to the WARM node?