ILM is not Working properly for ECK

Hello all,

We've been running a ECK last May, 2020 and there are some weird issues we are seeing ES 7.6.2

Context:
ECK 1.1.0

Hot-Warm Architecture

4 HotNodes = 64G Ram - 5TB Storage Fast SSD
8 warm Nodes = 28G ram - 8TB Storage Local Storage
3 Master nodes - 4G ram 3G Storage
1 ingest node = 4G ram and 2G storage
3 Kibana nodes = 4G ram

We are ingesting close to 1.5TB a day and for some reason the data goes to our Warm Nodes instead of Hot nodes. our index is p44.api.log.2020.07.07 and if I look at the hot threads from the node, I can see that is being written there, and the Stack Monitoring is exactly pointing to warm Nodes, to fix it I have to force reroute to HOT.

Are there anything missing on our ILM?

64.3% (321.4ms out of 500ms) cpu usage by thread 'elasticsearch[prod-logs-es-warmv1-4][[p44.api.log.2020.07.07][2]: Lucene Merge Thread #1697]'
 2/10 snapshots sharing following 24 elements
   java.base@13.0.2/jdk.internal.misc.Unsafe.park(Native Method)
   java.base@13.0.2/java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:235)
   java.base@13.0.2/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2123)
   app//org.apache.lucene.index.MergePolicy$OneMergeProgress.pauseNanos(MergePolicy.java:156)
   app//org.apache.lucene.index.MergeRateLimiter.maybePause(MergeRateLimiter.java:148)
   app//org.apache.lucene.index.MergeRateLimiter.pause(MergeRateLimiter.java:93)
   app//org.apache.lucene.store.RateLimitedIndexOutput.checkRate(RateLimitedIndexOutput.java:78)
   app//org.apache.lucene.store.RateLimitedIndexOutput.writeBytes(RateLimitedIndexOutput.java:72)
   app//org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:52)
   app//org.apache.lucene.store.RAMOutputStream.writeTo(RAMOutputStream.java:90)

Stack MOnitoring

ILM

We Solved this by adding:

    {
        "routing" : {
          "allocation" : {
            "require" : {
              "data" : "hot",
              "_id":""
            }
          }
        }
     }

Effectively telling the index to start at HOT.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.