ILM Error duration cannot be negative

running elk stack version 7.16.2.

GET .siem-signals-default-000010/_ilm/explain

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "duration cannot be negative, was given [-8402614]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "duration cannot be negative, was given [-8402614]",
    "suppressed" : [
      {
        "type" : "illegal_state_exception",
        "reason" : "Failed to close the XContentBuilder",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unclosed object or array found"
        }
      }
    ]
  },
  "status" : 400
}

This ilm error is happening for .siem-signals-default-000010,.siem-signals-default-000011,.siem-signals-default-000012 indices. .siem-signals-default-000012 is the current index.
Other index is working fine
GET .siem-signals-default-000009/_ilm/explain

{
  "indices" : {
    ".siem-signals-default-000009" : {
      "index" : ".siem-signals-default-000009",
      "managed" : true,
      "policy" : ".siem-signals-default",
      "lifecycle_date_millis" : 1643002355452,
      "age" : "22.17d",
      "phase" : "hot",
      "phase_time_millis" : 1640410343993,
      "action" : "complete",
      "action_time_millis" : 1643002355652,
      "step" : "complete",
      "step_time_millis" : 1643002355652,
      "phase_execution" : {
        "policy" : ".siem-signals-default",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1622608539811
      }
    }
  }
}

Interesting, thanks for letting us know about this!

Can you also provide the output of GET /.siem-signals-default-000010/_settings so I can see the index settings?

Also, do you happen to have a full stacktrace in the logs? If so, can you paste it here?

I don't what happened , everything is working fine now i have not made any change or resolve anything

GET .siem-signals-default-000010/_settings

{
  ".siem-signals-default-000010" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : ".siem-signals-default",
          "rollover_alias" : ".siem-signals-default",
          "indexing_complete" : "true"
        },
        "routing" : {
          "allocation" : {
            "include" : {
              "_tier_preference" : "data_content"
            }
          }
        },
        "mapping" : {
          "total_fields" : {
            "limit" : "10000"
          }
        },
        "refresh_interval" : "1s",
        "number_of_shards" : "1",
        "blocks" : {
          "write" : "false"
        },
        "provided_name" : ".siem-signals-default-000010",
        "creation_date" : "1643002355508",
        "number_of_replicas" : "1",
        "uuid" : "oYRMTZM1TNSGqX2XCjiytQ",
        "version" : {
          "created" : "7160299"
        }
      }
    }
  }
}

no error is coming duration cannot be negative

GET .siem-signals-default-000010/_ilm/explain
{
  "indices" : {
    ".siem-signals-default-000010" : {
      "index" : ".siem-signals-default-000010",
      "managed" : true,
      "policy" : ".siem-signals-default",
      "lifecycle_date_millis" : 1644925175663,
      "age" : "17.5h",
      "phase" : "hot",
      "phase_time_millis" : 1644925115410,
      "action" : "complete",
      "action_time_millis" : 1644925175657,
      "step" : "complete",
      "step_time_millis" : 1644925175657,
      "phase_execution" : {
        "policy" : ".siem-signals-default",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_size" : "50gb",
              "max_age" : "30d"
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1622608539811
      }
    }
  }
}

This makes me think that perhaps it was due to clock synchronization. The ILM explain response does the following:

TimeValue.timeValueMillis(System.currentTimeMillis() - lifecycleDate);

Which can be negative and would cause the error above. Coincidentally, it appears that this was recently fixed on 8.x because the line has been changed to:

TimeValue.timeValueMillis(Math.max(0L, now.get() - indexCreationDate));

I think we can backport a fix for this so that it will be fixed in the future for 7.x also. Thanks for reporting this!

I opened Do not allow negative age in ILM explain by dakrone · Pull Request #84043 · elastic/elasticsearch · GitHub to fix this, thanks again for letting us know about it!

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.