[Roll up index] Roll up index stopped processing documents

[zaratustra689 (Carolina Baldino)]

Hello,

I have running now 3 different jobs for their respective index. In one of them, docs are being processed but on the other 2, they are not.
The thing is that they 3 have the exactly same configuration for the roll up, they only differ in the amount of terms selected for the job:

• Roll up A has only 2 terms
• Roll up B has 7 terms
• Roll up C has 7 terms

They are all being rolled up under the same index pattern. Non of the terms have infinite (of tendency to infinite) amounts of possibilities (such as would be with a client_ip kind of term)

Roll ups configs for jobs A, B and C are the following:

But I noticed that for some reason jobs B and C stopped processing documents.. the stats are the very same since day ago:

• Roll up A: (is the only roll up that hasn't stopped processing documents )

  

• Roll up B:

  

• Roll up C

  

Roll ups B and C seems being "limitated" by something for some reason. Can you tell that "limitation" that is making stop the processed documents for those jobs?

I only suspect of the page_size, which is the same for the three jobs: the 1000 default. I find curious about the fact that the number of pages processed in B and C is a multiple *1000 of the rollups indexed. Is it related to the amount of terms? Is there any recommended page size related to the amount of terms selected for a job?

EDIT: I realized that indeces B and C have "search_failures"s:

Logs shows:

[2020-10-19T09:00:00,227][WARN ][o.e.x.r.j.RollupJobTask  ] [QyJqSRr] Rollup job [A] failed with an exception: 
java.lang.RuntimeException: Shard failures encountered while running indexer for job [A]: [shard [[QyJqSRrVS1alvBduaEDp5w][filebeat-corso-6.6.2-2020.09.05][0]], reason [RemoteTransportException[[QyJqSRr][172.18.0.2:9300][indices:data/read/search[phase/query]]]; nested: NumberFormatException[For input string: "tracker-tag-absent"]; ], cause [java.lang.NumberFormatException: For input string: "tracker-tag-absent"

tracker-tag-absent is the output of the log

Thank for reading,

Caro.

[zaratustra689 (Carolina Baldino)]

Solved this thanks to @polyfractal response on [Rollup Job] NumberFormatException, and I quote:

This looks like a rollup issue, but it actually turns out to be a bug in the composite aggregation (which rollup uses). If the composite agg is iterating over a field in one index, then moves on to the next index in the search and finds that the field is unmapped in that new index... it accidentally mapped that field as numeric. This would throw an exception when the original field was a keyword like in your case, because the after key can't be cast to a numeric.
Since this is caused by unmapped fields, a temporary fix would be to go add the correct mapping to the indices missing the field(s), or just wait for 7.1 to land.

Since is not an option for me updating to ES 7.1, I updated the index mapping of the source indices of the roll_up jobs, here is an example:

Error log:

[2020-10-19T09:00:00,227][WARN ][o.e.x.r.j.RollupJobTask  ] [QyJqSRr] Rollup job [A] failed with an exception: 
java.lang.RuntimeException: 
Shard failures encountered while running indexer for job [A]: 
[shard [[QyJqSRrVS1alvBduaEDp5w][filebeat-6.6.2-2020.09.05][0]], 
reason [RemoteTransportException[[QyJqSRr][172.18.0.2:9300][indices:data/read/search[phase/query]]]; 
nested: NumberFormatException[For input string: "tracker-tag-absent"]; ], 
cause [java.lang.NumberFormatException: 
For input string: "tracker-tag-absent"

Input tracker-tab-absent is the value of the field message_metadata.organization.keyword, as you can check in the current position that this json shows you when you select your job and go to the JSON tab:

I updated the mapping of that particular index and it worked. You can also update it under the _template/filebeat index so that it applies to all indices. (in this case i'm using filebeat but use the template you need _template/<your_template>)

PUT filebeat-6.6.2-2020.09.05/_mapping/doc
{
  "properties": { 
    "message_metadata": {
      "properties": {
        "organization": { 
          "type": "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }  
    }
  }
}

That was the field message_metadata.organization.keyword mapped to the index. Notice I used the one that is aggretable (message_metadata.organization.keyword instead of message_metadata.organization) cause is the field that the roll up needs.

Cheers,

Caro.