# \[Roll up index\] Roll up index stopped processing documents

**URL:** <https://discuss.elastic.co/t/roll-up-index-roll-up-index-stopped-processing-documents/252434>\
**Category:** Elasticsearch\
**Created:** [October 17, 2020, 10:20pm UTC](https://discuss.elastic.co/t/roll-up-index-roll-up-index-stopped-processing-documents/252434 "2020-10-17T22:20:02Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![zaratustra689](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaratustra689/32/43118_2.png) [@zaratustra689](https://discuss.elastic.co/u/zaratustra689)\
**Post date:** [October 17, 2020, 10:20pm UTC](https://discuss.elastic.co/t/roll-up-index-roll-up-index-stopped-processing-documents/252434/1 "2020-10-17T22:20:02Z")

</div>

Hello,

I have running now 3 different jobs for their respective index. In one of them, docs are being processed but on the other 2, they are not.  
The thing is that they 3 have the **exactly same configuration** for the roll up, they only differ in the amount of terms selected for the job:

- Roll up A has only 2 terms
- Roll up B has 7 terms
- Roll up C has 7 terms

They are all being rolled up under the same index pattern. Non of the terms have infinite (of tendency to infinite) amounts of possibilities (such as would be with a `client_ip` kind of term)

Roll ups configs for jobs A, B and C are the following:

> ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/b/4bc1ab953e91375538c83b8ca3ac1c5e2663cec6.png)

But I noticed that for some reason jobs B and C stopped processing documents.. the stats are the very same since day ago:

- Roll up A: (is the only roll up that hasn't stopped processing documents )

> ![image](https://us1.discourse-cdn.com/elastic/original/3X/1/0/10f0ed1f2f792f2f673bef9645e15620fc2e874c.png)

- Roll up B:

> ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/4/24b1155c2163d5d8c74896303494f28d54b43059.png)

- Roll up C

> ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/0/00de632e7e8d713ebfebe2473a0a56e34197fbab.png)

Roll ups B and C seems being "limitated" by something for some reason. Can you tell that "limitation" that is making stop the processed documents for those jobs?

I only suspect of the `page_size`, which is the same for the three jobs: the 1000 default. I find curious about the fact that the number of pages processed in B and C is a multiple \*1000 of the rollups indexed. Is it related to the amount of terms? Is there any recommended page size related to the amount of terms selected for a job?

**EDIT:** I realized that indeces B and C have "search\_failures"s:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/c/ac5da8d7c19954dc9ba4db275ccc7d7b688a4cf5.png)

Logs shows:

```auto
[2020-10-19T09:00:00,227][WARN][o.e.x.r.j.RollupJobTask] [QyJqSRr] Rollup job [A] failed with an exception: 
java.lang.RuntimeException: Shard failures encountered while running indexer for job [A]: [shard [[QyJqSRrVS1alvBduaEDp5w][filebeat-corso-6.6.2-2020.09.05][0]], reason [RemoteTransportException[[QyJqSRr][172.18.0.2:9300][indices:data/read/search[phase/query]]]; nested: NumberFormatException[For input string: "tracker-tag-absent"]; ], cause [java.lang.NumberFormatException: For input string: "tracker-tag-absent"

```

> `tracker-tag-absent` is the output of the log

Thank for reading,

Caro.

---

<div class="post-metadata">

**Author:** ![zaratustra689](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zaratustra689/32/43118_2.png) [@zaratustra689](https://discuss.elastic.co/u/zaratustra689)\
**Post date:** [October 23, 2020, 8:41pm UTC](https://discuss.elastic.co/t/roll-up-index-roll-up-index-stopped-processing-documents/252434/2 "2020-10-23T20:41:58Z")

</div>

Solved this thanks to @polyfractal response on [[Rollup Job] NumberFormatException - #4 by polyfractal](https://discuss.elastic.co/t/rollup-job-numberformatexception/178195/4), and I quote:

> This looks like a rollup issue, but it actually turns out to be a bug in the composite aggregation (which rollup uses). If the composite agg is iterating over a field in one index, then moves on to the next index in the search and finds that the field is unmapped in that new index... it accidentally mapped that field as numeric. This would throw an exception when the original field was a keyword like in your case, because the `after` key can't be cast to a numeric.  
> Since this is caused by unmapped fields, a temporary fix would be to go add the correct mapping to the indices missing the field(s), or just wait for 7.1 to land.

Since is not an option for me updating to ES 7.1, I updated the index mapping of the source indices of the roll\_up jobs, here is an example:

Error log:

```auto
[2020-10-19T09:00:00,227][WARN][o.e.x.r.j.RollupJobTask] [QyJqSRr] Rollup job [A] failed with an exception: 
java.lang.RuntimeException: 
Shard failures encountered while running indexer for job [A]: 
[shard [[QyJqSRrVS1alvBduaEDp5w][filebeat-6.6.2-2020.09.05][0]], 
reason [RemoteTransportException[[QyJqSRr][172.18.0.2:9300][indices:data/read/search[phase/query]]]; 
nested: NumberFormatException[For input string: "tracker-tag-absent"]; ], 
cause [java.lang.NumberFormatException: 
For input string: "tracker-tag-absent"

```

Input `tracker-tab-absent` is the value of the field `message_metadata.organization.keyword`, as you can check in the `current position` that this json shows you when you select your job and go to the JSON tab:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/1/211775a88fdafddde9359069a51f0b2410f9e186.png)

I updated the mapping of that particular index and it worked. You can also update it under the `_template/filebeat` index so that it applies to all indices. (in this case i'm using `filebeat` but use the template you need `_template/<your_template>`)

```auto
PUT filebeat-6.6.2-2020.09.05/_mapping/doc
{
  "properties": { 
    "message_metadata": {
      "properties": {
        "organization": { 
          "type": "text",
          "fields" : {
            "keyword" : {
              "type" : "keyword",
              "ignore_above" : 256
            }
          }
        }
      }  
    }
  }
}

```

That was the field `message_metadata.organization.keyword` mapped to the index. Notice I used the one that is aggretable (`message_metadata.organization.keyword` instead of `message_metadata.organization`) cause is the field that the roll up needs.

Cheers,

Caro.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 20, 2020, 8:41pm UTC](https://discuss.elastic.co/t/roll-up-index-roll-up-index-stopped-processing-documents/252434/3 "2020-11-20T20:41:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.