Secondary rollup of existing rollup index

Hi,
I have an index which is created from a Rollup job. Its content is as follows:

"hits": {
    "total": 4,
    "max_score": 1,
    "hits": [
      {
        "_index": "my_rollup_1m_2021.08.17",
        "_type": "_doc",
        "_id": "my_rollup_1m$09-UpSrS0X2Nnx3TyDfMtg",
        "_score": 1,
        "_source": {
          "stId.terms._count": 2,
          "timestamp.date_histogram.time_zone": "UTC",
          "timestamp.date_histogram.timestamp": 1629201600000,
          "count.sum.value": 20,
          "_rollup.version": 2,
          "stId.terms.value": 1,
          "timestamp.date_histogram.interval": "1h",
          "timestamp.date_histogram._count": 2,
          "_rollup.id": "my_rollup_1m"
        }
      },
    ....
}

Then I am trying to create a secondary level rollup job, that rolls up the index mentioned above. The secondary rollup job configuration is:

PUT _rollup/job/my_rollup_1h
{
  "id": "my_rollup_1h",
  "index_pattern": "my_rollup_1m_2021.08.17",
  "rollup_index": "<my_rollup_1h_{now/d}>",
  "cron": "0 39 * * * ?",
  "groups": {
    "date_histogram": {
      "interval": "1h",
      "field": "timestamp.date_histogram.timestamp",
      "time_zone": "UTC"
    },
    "terms": {
      "fields": [
        "stId.terms.value"
      ]
    }
  },
  "metrics": [
    {
      "field": "count.sum.value",
      "metrics": [
        "sum"
      ]
    }
  ],
  "timeout": "20s",
  "page_size": 10000
}

Secondary rollup job runs fine as seen from below status:

"status": {
    "job_state": "started",
    "current_position": {
      "st.terms.value.terms": 4,
      "timestamp.date_histogram.timestamp.date_histogram": 1629201600000
    },
    "upgraded_doc_id": true
  },
  "stats": {
    "pages_processed": 2,
    "documents_processed": 4,
    "rollups_indexed": 4,
    "trigger_count": 1,
    "index_time_in_ms": 1,
    "index_total": 1,
    "index_failures": 0,
    "search_time_in_ms": 5,
    "search_total": 2,
    "search_failures": 0
  }

But when i check the seconday rollup index, its always empty. Besides i see this exception in ES:

[2021-08-17T13:39:00,004][WARN ][o.e.x.c.i.AsyncTwoPhaseIndexer] [es-data-0] Error while attempting to bulk index documents: failure in bulk execution:
[0]: index [my_rollup_1h_2021.08.17], type [_doc], id [my_rollup_1h$2b4OsNouZ94vhB47tII2ww], message [MapperParsingException[failed to parse]; nested: ClassCastException[org.elasticsearch.index.mapper.DateFieldMapper incompatible with org.elasticsearch.index.mapper.ObjectMapper];]
[1]: index [my_rollup_1h_2021.08.17], type [_doc], id [my_rollup_1h$AU3POF2c3SwXnjzgga3d0g], message [MapperParsingException[failed to parse]; nested: ClassCastException[org.elasticsearch.index.mapper.DateFieldMapper incompatible with org.elasticsearch.index.mapper.ObjectMapper];]
[2]: index [my_rollup_1h_2021.08.17], type [_doc], id [my_rollup_1h$pQOaK-fGV4vl35xgviD4Og], message [MapperParsingException[failed to parse]; nested: ClassCastException[org.elasticsearch.index.mapper.DateFieldMapper incompatible with org.elasticsearch.index.mapper.ObjectMapper];]
[3]: index [my_rollup_1h_2021.08.17], type [_doc], id [my_rollup_1h$72Ci96Cz6XIVIoTHg3z-Rw], message [MapperParsingException[failed to parse]; nested: ClassCastException[org.elasticsearch.index.mapper.DateFieldMapper incompatible with org.elasticsearch.index.mapper.ObjectMapper];]

Seems like mapping exception when trying to index the processed data. Is there any thing else to be done when configuring the job in this scenario?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.