Rollup job lost some data

I have a rollup job, it does daily aggregations. But for unknown reason, some data not included in the rollup index. The job was created at the beginning of each month and was running in the started state until the end of the month. The rollup index documents and size increased day by day.

But at the end of the month, if I create another rollup job with same settings and run it again, it will create another rollup index with 1 million documents difference. When searching both of them, I can find some documents only exist in the new rollup index.

I looked into the elasticsearch log, it only has a lot of warnings and some JVM GC log.
Are there any logs for rollup jobs except the log in /var/log/elasticsearch folder?
Anyone had same issue when using rollup jobs?
The ES version is 6.5.2.

Logs:

[2020-03-17T00:28:27,000][WARN ][o.e.x.c.i.AsyncTwoPhaseIndexer] [es-data-234234242342342] Schedule was triggered for job [rollup_readings_xxxx_v2_job], but prior indexer is still running.
[2020-03-17T11:00:02,688][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-234234242342342] [gc][16962357] overhead, spent [320ms] collecting in the last [1s]
    {
      "config" : {
        "id" : "rollup_xxxx_days_v2_job",
        "index_pattern" : "xxxx*_v2",
        "rollup_index" : "rollup_xxxx_days_v2",
        "cron" : "* * */12 * * ?",
        "groups" : {
          "date_histogram" : {
            "interval" : "1d",
            "field" : "created_at",
            "delay" : "12h",
            "time_zone" : "UTC"
          },
          "terms" : {
            "fields" : [
              "aid",
              "uid",
              "did",
              "sid",
              "evt"
            ]
          }
        },
        "metrics" : [
          {
            "field" : "value",
            "metrics" : [
              "avg",
              "min",
              "max",
              "sum",
              "value_count"
            ]
          },
          {
            "field" : "geo_lat",
            "metrics" : [
              "avg"
            ]
          },
          {
            "field" : "geo_lon",
            "metrics" : [
              "avg"
            ]
          },
          {
            "field" : "yy",
            "metrics" : [
              "value_count"
            ]
          }
        ],
        "timeout" : "20s",
        "page_size" : 9216
      }
    }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.