Problem while running multiple rollup jobs on same rollup-index

Hello,

I created one index as,

    PUT prod-index-20190101/_doc/1
    {
    "subject":"cpp",
    "event_at":"2019-01-01T00:00:00.000Z"
    }

    PUT prod-index-20190101/_doc/2
    {
      "subject”:”java”,
      "event_at":"2019-01-01T00:00:00.000Z"
    }

And it contains two documents as shown above.

And then I tried to run rollup job -1 on above index as,

    PUT _xpack/rollup/job/rj-subject-test-1
    {
    "index_pattern": "prod-index-*",
    "rollup_index": "rjsubject-index",
    "cron": "0 0/1 * * * ?",
    "page_size" :1000,
    "groups" : {
    "date_histogram": {
    "field": "event_at",
    "interval": "1d"
    },
    "terms": {
    "fields": ["subject.keyword"]
    }
    }

And now if I perform rollup-search on the rollup-index “rjsubject-index” as,

GET rjsubject-index/_rollup_search?size=0
{
"aggs":{
"date":{
"date_histogram": {
"field": "event_at",
"interval": "1d"
},
"aggs": {
"NAME": {
"terms": {
"field": "subject.keyword",
"size": 10
}
}
}
}
}
}

now I am getting response for above code as ,

"aggregations" : {
"date" : {
"meta" : { },
"buckets" : [
{
"key_as_string" : "2019-01-01T00:00:00.000Z",
"key" : 1546300800000,
"doc_count" : 2,
"NAME" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [
{
"key" : "cpp",
"doc_count" : 1
},
{
"key" : "java",
"doc_count" : 1
}
]
}
}
]
}
}

Which is fine.

Now I’ll create one more index as,

PUT prod-index-20190102/_doc/1
{
  "subject":"ES",
  "event_at":"2019-01-02T00:00:00.000Z"
}

PUT prod-index-20190102/_doc/2
{
  "subject":"Logstash",
  "event_at":"2019-01-02T00:00:00.000Z"
}

Which also contains 2 docs. And then I’ll delete previously created index “prod-index-20190101”

Now if I perform GET prod*/_search?size=20 I’ll get 2 docs as expected which are from index “prod-index-20190102”.

Now before running second rollup-job I’ll stop first rollup-job.

Now I’ll run second rollup-job on the same index pattern as first rollup-job and I’ll keep rollup-index also same as rollup-job 1’s rollup-index,

PUT _xpack/rollup/job/rj-subject-2
{
    "index_pattern": "prod-index-*",
    "rollup_index": "rjsubject-index",
    "cron": "0 0/1 * * * ?",
    "page_size" :1000,
    "groups" : {
      "date_histogram": {
        "field": "event_at",
        "interval": "1d"
      },
      "terms": {
        "fields": ["subject.keyword"]
      }
    }
}

And now after rollup-job 2’s completion if I perform rollup-search as,

GET rjsubject-index/_rollup_search?size=0
{
  "aggs":{
    "date":{
      "date_histogram": {
        "field": "event_at",
        "interval": "1d"
      },
      "aggs": {
        "NAME": {
          "terms": {
            "field": "subject.keyword",
            "size": 10
          }
        }
      }
    }
  }
}

And now I am getting response for above code as,

"aggregations" : {
    "date" : {
      "meta" : { },
      "buckets" : [
        {
          "key_as_string" : "2019-01-02T00:00:00.000Z",
          "key" : 1546387200000,
          "doc_count" : 2,
          "NAME" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "ES",
                "doc_count" : 1
              },
              {
                "key" : "Logstash",
                "doc_count" : 1
              }
            ]
          }
        }
      ]
    }
  }

So now my question is why I am not getting docs from index “prod-index-20190101” which contains subject names as “cpp” and “java” and which are rolled-up using rollupjob 1
(Note- as we have seen above, I was getting the docs in rollup-search before running the second job)

And also I performed normal search on rollup index as,
GET rjsubject-index/_search?size=20
And I am able to see,

"hits" : {
"total" : 4, …….}

So all 4 docs are present in given rollup-index then why they are not getting displayed in rollup-search?

ES version 6.6.1

Is there something I should try, am I missing something??
Thanks in advance.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.