_ilm/explain returns illegal_argument_exception: "duration cannot be negative, was given [-1613792563]"

Elastic Stack Elasticsearch
1

Hi guys,
I am running Kibana 7.15 and somehow the ilm information of each index got lost in the UI.
Usually (still like this in my test and monitoring cluster) when loading all indices via "Stack Management -> Index Management" the JSON response includes the ILM information.
e.g.:

{
      "health":"green",
      "status":"open",
      "name":"filebeat-7.11.1-2021.11.08",
      "uuid":"xPyqf9ZQS8WXnIEY2cb3HQ",
      "primary":"1",
      "replica":"0",
      "documents":"1220135",
      "size":"432.7mb",
      "isFrozen":false,
      "aliases":"none",
      "hidden":false,
      "isRollupIndex":false,
      "ilm":{
         "index":"filebeat-7.11.1-2021.11.08",
         "managed":true,
         "policy":"log-default-delete14",
         "lifecycle_date_millis":1636326001076,
         "age":"2.43d",
         "phase":"hot",
         "phase_time_millis":1636326001231,
         "action":"complete",
         "action_time_millis":1636326001431,
         "step":"complete",
         "step_time_millis":1636326001431,
         "phase_execution":{
            "policy":"log-default-delete14",
            "phase_definition":{
               "min_age":"0ms",
               "actions":{
                  "set_priority":{
                     "priority":100
                  }
               }
            },
            "version":1,
            "modified_date_in_millis":1616755251168
         }
      },
      "isFollowerIndex":false
   }

But in the production cluster the ilm part is missing and looks like the following:
e.g.

{
   "health":"green",
   "status":"open",
   "name":"linux_syslog_2021-08-30",
   "uuid":"s9GnmrQkSHecCZpe8C2FCQ",
   "primary":"1",
   "replica":"1",
   "documents":"15506551",
   "size":"5.4gb",
   "isFrozen":false,
   "aliases":"none",
   "hidden":false,
   "isRollupIndex":false,
   "isFollowerIndex":false
}

But ILM Explain provides all the information:

GET linux_syslog_2021-08-30/_ilm/explain
{
  "indices" : {
    "linux_syslog_2021-08-30" : {
      "index" : "linux_syslog_2021-08-30",
      "managed" : true,
      "policy" : "ilm_policy_linux",
      "lifecycle_date_millis" : 1630274400252,
      "age" : "72.48d",
      "phase" : "cold",
      "phase_time_millis" : 1632002404434,
      "action" : "complete",
      "action_time_millis" : 1632002728947,
      "step" : "complete",
      "step_time_millis" : 1632002728947,
      "phase_execution" : {
        "policy" : "ilm_policy_linux",
        "phase_definition" : {
          "min_age" : "20d",
          "actions" : {
            "allocate" : {
              "number_of_replicas" : 1,
              "include" : { },
              "exclude" : { },
              "require" : {
                "data" : "cold"
              }
            }
          }
        },
        "version" : 1,
        "modified_date_in_millis" : 1620831287614
      }
    }
  }
}

Kibana reports property "managed" is missing, after clicking on any index.
It is not possible to show the settings pane on the right anymore.

image
image1909×1062 130 KB

Any idea what happened and how to solve this issue?

Thanks in advance,
greetings

2

After lots of tests I found out, that it is related to indices.
We have lots of different user roles and for some of them the issue is present, for others not.
In other words, it seems to be related to indices and if there is one "affected index" visible to a user the issue occurs. If all indices are fine, settings pane is sliding in smoothly from the right.
Still searching for the exact thing which breaks Kibana to get ilm details...

3

I managed to identify one index, which has broken ilm and causes Kibana not to load the ilm information.

GET ocp_mdsa_qa_mdsa-category-service_202111/_ilm/explain
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "duration cannot be negative, was given [-1613850232]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "duration cannot be negative, was given [-1613850232]",
    "suppressed" : [
      {
        "type" : "illegal_state_exception",
        "reason" : "Failed to close the XContentBuilder",
        "caused_by" : {
          "type" : "i_o_exception",
          "reason" : "Unclosed object or array found"
        }
      }
    ]
  },
  "status" : 400
}

Any idea how to fix this?

4

Problem solved!

I automatically created indices with Curator with an origination date in the future and this breaks the ILM. After correcting the origination date, everything is fine again.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.