[Roll up index] Roll up index stopped processing documents

Elastic Stack Elasticsearch
2

Solved this thanks to @polyfractal response on [Rollup Job] NumberFormatException - #4 by polyfractal, and I quote:

This looks like a rollup issue, but it actually turns out to be a bug in the composite aggregation (which rollup uses). If the composite agg is iterating over a field in one index, then moves on to the next index in the search and finds that the field is unmapped in that new index... it accidentally mapped that field as numeric. This would throw an exception when the original field was a keyword like in your case, because the after key can't be cast to a numeric.
Since this is caused by unmapped fields, a temporary fix would be to go add the correct mapping to the indices missing the field(s), or just wait for 7.1 to land.

Since is not an option for me updating to ES 7.1, I updated the index mapping of the source indices of the roll_up jobs, here is an example:

Error log:

[2020-10-19T09:00:00,227][WARN ][o.e.x.r.j.RollupJobTask  ] [QyJqSRr] Rollup job [A] failed with an exception: 
java.lang.RuntimeException: 
Shard failures encountered while running indexer for job [A]: 
[shard [[QyJqSRrVS1alvBduaEDp5w][filebeat-6.6.2-2020.09.05][0]], 
reason [RemoteTransportException[[QyJqSRr][172.18.0.2:9300][indices:data/read/search[phase/query]]]; 
nested: NumberFormatException[For input string: "tracker-tag-absent"]; ], 
cause [java.lang.NumberFormatException: 
For input string: "tracker-tag-absent"

Input tracker-tab-absent is the value of the field message_metadata.organization.keyword, as you can check in the current position that this json shows you when you select your job and go to the JSON tab:
image

I updated the mapping of that particular index and it worked. You can also update it under the _template/filebeat index so that it applies to all indices. (in this case i'm using filebeat but use the template you need _template/<your_template>)

PUT filebeat-6.6.2-2020.09.05/_mapping/doc
{
  "properties": { 
    "message_metadata": {
      "properties": {
        "organization": { 
          "type": "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }  
    }
  }
}

That was the field message_metadata.organization.keyword mapped to the index. Notice I used the one that is aggretable (message_metadata.organization.keyword instead of message_metadata.organization) cause is the field that the roll up needs.

Cheers,

Caro.

1 Like