Strange shard routing behavior with index templates and shrink in a hot/warm architecture

[Kim-Kruse-Hansen]

Hi

I have an issue with Curator that I would like an opinion on. This is elk 6.2.0 with curator 5.4, but also happens on 6.1.2. I hope I am missing somethng obvious.

I have a cluster , that implements the hot/warm architecture. I am using the roll-over approach of handling the indexes- After 1 day , I migrate the indixes to the warm node. This works fine.

After 1 more day, I want to shrink and forcemerge. I want this to take place on my warm nodes and I want the indexes to remain on the warm nodes.

Curator picks one of my warm nodes as shrink node, moves the shards to the shrink node.Shrinks the index to 1 shard. After this is complete the index is moved to the HOT node. Then curator starts the post allocation step, which moves the index back to the WARM node.

So obviously, this is taking much longer than it should.

My skrink step looks like this at the moment:

actions:

5:

action: shrink

options:

  ignore_empty_list : true

  shrink_node: DETERMINISTIC

  node_filters:

    permit_masters: False

  number_of_shards: 1

  number_of_replicas: 0

  shrink_suffix: '-shrink'

  delete_after: false

  post_allocation:

    allocation_type: require

    key: box_type

    value: warm

  wait_for_active_shards: 1

  extra_settings:

    settings:

      index.codec: best_compression

      index.routing.allocation.require.box_type: warm

  wait_for_completion: True

  wait_interval: 30

  max_wait: -1

filters:

- filtertype: count

  count: 2

  pattern: '^(dns-active)-\d+$'

  reverse: true

  exclude: true

- filtertype: age

  source: field_stats

  direction: older

  unit: days

  unit_count: 2

  field: '@timestamp'

  stats_result: min_value

I have been trying to fix this with index templates , but no luck so far.

Anything I can do to avoid this copying of the index back to the HOT node and then back to the WARM node?

Regards
Kim

[theuntergeek (Aaron Mildenstein)]

Are you setting shard routing parameters in your index templates?

[Kim-Kruse-Hansen]

Hi
Yes , I currently have 2 index templates.

The first one at order 0 requires box_type: hot and then other at order 1 requires box:warm.

If I manually create an index that ends with -shrink , it gets correctly allocated to my warm nodes. Index without -shrink suffix will also get correctly allocated to my hot nodes.

Could it be the rebalancing that is kicking in somehow? The hot nodes have fewer indices than the warm nodes.

Regards
Kim

[theuntergeek (Aaron Mildenstein)]

This sounds more like a question for Elasticsearch. While Curator is indeed handling the API calls, it seems that the API calls and the shard routing would work this same way whether you called the APIs manually, or via Curator.

I'm not sure what happens inside Elasticsearch with templates when a shrink is called and the new index is "created." The answer could help inform what to do. I re-titled the topic to hopefully address the core issue better.