Rollup not triggered

Elastic Stack Elasticsearch
1

Hi,

We are trying to use the new rollup feature.
trigger_count increases with time, but others metrics are sticking to 0, and the index remain empty.
Are we missing something?

Thanks !

PUT _xpack/rollup/job/testrollup
    {
      "index_pattern": "dailyindex_539_*",
      "rollup_index": "test-rollup",
      "cron": "*/30 * * * * ?",
      "page_size": 1000,
      "groups": {
        "date_histogram": {
          "field": "@timestamp",
          "interval": "1m"
        }
      },
      "metrics": [
        {
          "field": "responsetime",
          "metrics": [
            "min",
            "max",
            "sum"
          ]
        }
      ]
    }

POST _xpack/rollup/job/testrollup/_start

GET _xpack/rollup/job/_all
{
  "jobs": [
    {
      "config": {
        "id": "testrollup",
        "index_pattern": "dailyindex_539_*",
        "rollup_index": "test-rollup",
        "cron": "*/30 * * * * ?",
        "groups": {
          "date_histogram": {
            "interval": "1m",
            "field": "@timestamp",
            "time_zone": "UTC"
          }
        },
        "metrics": [
          {
            "field": "responsetime",
            "metrics": [
              "min",
              "max",
              "sum"
            ]
          }
        ],
        "timeout": "20s",
        "page_size": 1000
      },
      "status": {
        "job_state": "started"
      },
      "stats": {
        "pages_processed": 0,
        "documents_processed": 0,
        "rollups_indexed": 0,
        "trigger_count": 424
      }
    }
  ]
}

And rollup index remain empty

GET _cat/indices/test-rollup,dailyindex_539_*?v
health status index                      uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   dailyindex_539_2018-08-23 rm-eGdpZRrqp6Wa-PEmHqA   3   1    6427212            0      6.7gb          3.3gb
green  open   test-rollup                   lOPCzTGIQEunDvW2YlYUTA   5   1          0            0      2.5kb          1.2kb
2

Any errors/warnings in the log? Since it's triggering, I suspect there's something going wrong and it's aborting after the first search / before indexing docs.

You might also try deleting the job, deleting test-rollup, recreate/restart and see if there are any errors/warnings immediately after job creation.

Sorry for the delay, slipped through my inbox filter :slight_smile:

3

Hi,

We couldn't find any error in logs on the master node.
We have tried to delete/recreate many times and we are getting correct acknowledge.

I'm wondering if we didn't reach that bug: https://github.com/elastic/elasticsearch/pull/31890
Here is the format of our timestamp: "@timestamp": "2018-08-28T14:28:48.386Z",

Benjamin

4

Very possible. What's the section in the mapping look like?

That bug manifests as SearchPhaseExecutionException warnings. See this thread for an example exception

Would you be able to upgrade to 6.4 to try with the bugfix in place?

5 
"properties": {
      "@timestamp": {
        "type": "date",
        "format": "dateOptionalTime"
      },

Sorry, we actually have errors, I missed it (I only checked the Master Node).
Here is the error :

[2018-08-28T19:00:00,035][DEBUG][o.e.a.s.TransportSearchAction] [es8-main] All shards failed for phase: [query]
org.elasticsearch.ElasticsearchParseException: failed to parse date field [1535475600000] with format [dateOptionalTime]
        at org.elasticsearch.common.joda.DateMathParser.parseDateTime(DateMathParser.java:213) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.joda.DateMathParser.parse(DateMathParser.java:66) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.parseToMilliseconds(DateFieldMapper.java:318) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.mapper.DateFieldMapper$DateFieldType.isFieldWithinQuery(DateFieldMapper.java:342) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.query.RangeQueryBuilder.getRelation(RangeQueryBuilder.java:459) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.query.RangeQueryBuilder.doRewrite(RangeQueryBuilder.java:466) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.query.AbstractQueryBuilder.rewrite(AbstractQueryBuilder.java:269) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.search.builder.SearchSourceBuilder.rewrite(SearchSourceBuilder.java:888) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.search.builder.SearchSourceBuilder.rewrite(SearchSourceBuilder.java:79) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.query.Rewriteable.rewrite(Rewriteable.java:67) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.query.Rewriteable.rewrite(Rewriteable.java:50) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.search.internal.ShardSearchLocalRequest$RequestRewritable.rewrite(ShardSearchLocalRequest.java:285) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.search.internal.ShardSearchLocalRequest$RequestRewritable.rewrite(ShardSearchLocalRequest.java:275) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.index.query.Rewriteable.rewrite(Rewriteable.java:67) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:627) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.search.SearchService.createSearchContext(SearchService.java:607) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.search.SearchService.createContext(SearchService.java:569) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:551) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:347) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:333) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:329) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:1019) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.3.0.jar:6.3.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_171]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_171]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.IllegalArgumentException: Unrecognized chars at the end of [1535475600000]: [0000]
        at org.elasticsearch.common.joda.DateMathParser.parseDateTime(DateMathParser.java:209) ~[elasticsearch-6.3.0.jar:6.3.0]

We plan to upgrade tomorrow morning.

6

Ah yep, looks like that bug. Upgrading should fix it.

I'll talk to Jim and see if we should backport this to 6.3.x as well.

7

Hey,

I confirm the upgrade fixed the issue.
Thanks !

Benjamin

8

Great! Feel free to ping me directly on any other Rollup issues you encounter. I've been keeping an eye on the forums, but a direct ping will notify me quicker :slight_smile:

Or if it looks like a bona fide bug, can always submit directly to github too.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.