Elastic cloud Kibana rollup is not working

Elastic Stack Elasticsearch
1

Dear Elastic Team Members,

Recently we've been testing Elastic Cloud Kibana features with the trail, Today we have encountered an issue that rollup is not working.
As we have an active index called metricbeat is being written. I did many times tested with Kibana rollup. but never had a chance to indexing data to a new index. but the rollup is executed successfully every 15s, Don't know why there is no data is written into the new index test123.

We'll make a contract with Elastic cloud soon. but some features I need to make sure it's working properly before we move on-premise ELK to Elastic Cloud.

I know I can ask for help on the Elastic Cloud portal,

But at the moment, I would like to ask for Elastic Team member here to help me out?

Thanks in advance

PS

The license I had for testing here.

{
  "license" : {
    "status" : "active",
    "uid" : "4552d43c-e532-47b1-9552-27fed12f7d1a",
    "type" : "platinum",
    "issue_date" : "2019-01-08T00:00:00.000Z",
    "issue_date_in_millis" : 1546905600000,
    "expiry_date" : "2020-04-30T23:59:59.999Z",
    "expiry_date_in_millis" : 1588291199999,
    "max_nodes" : 100,
    "issued_to" : "Elastic - INTERNAL",
    "issuer" : "Alex Piggott",
    "start_date_in_millis" : 1546300800000
  }
}

Index metricbeat

image
image.png1912×955 129 KB

image
image.png1717×974 91.9 KB

Rollup settings:

image
image.png1727×975 70.9 KB

index test123

image
image.png1040×101 7.57 KB

2

Any Updates on this?

Thanks in advance

3

@Robin_Guo Do you have data in the metricbeat* index pattern that is over a day old? The rollup job is configured with a delay of 1d, which means it will not roll up any data that has a timestamp after 1d ago.

Also, the full config of the job would be helpful in diagnosing (should be able to click the JSON tab in the job view).

4

Hi Elastic Team,

I have read the document about roll-up that the official website suggested as following:

https://www.elastic.co/guide/en/kibana/current/create-and-manage-rollup-job.html

Also , after I created a new roll-up job to test, We have a new index with a rollup data, when I create an index pattern to mapping that new index, from Discover on Kibana I didn't see there is any value for field Time, And there even don't have Time Range I can choose.

PS

  • The rollup job I made:

GET _xpack/rollup/job/metricbeat_test4


{
  "jobs" : [
    {
      "config" : {
        "id" : "metricbeat_test4",
        "index_pattern" : "metricbeat-*",
        "rollup_index" : "metricbeat_test4",
        "cron" : "0 11 * * * ?",
        "groups" : {
          "date_histogram" : {
            "interval" : "5m",
            "field" : "@timestamp",
            "delay" : "2d",
            "time_zone" : "UTC"
          },
          "terms" : {
            "fields" : [
              "beat.name.keyword",
              "tags.keyword",
              "dc.keyword",
              "os.keyword"
            ]
          }
        },
        "metrics" : [
          {
            "field" : "system.load.1",
            "metrics" : [
              "avg"
            ]
          },
          {
            "field" : "system.load.5",
            "metrics" : [
              "avg"
            ]
          },
          {
            "field" : "system.load.15",
            "metrics" : [
              "avg"
            ]
          },
          {
            "field" : "system.load.norm.1",
            "metrics" : [
              "avg"
            ]
          },
          {
            "field" : "system.load.norm.15",
            "metrics" : [
              "avg"
            ]
          },
          {
            "field" : "system.load.norm.5",
            "metrics" : [
              "avg"
            ]
          }
        ],
        "timeout" : "20s",
        "page_size" : 1000
      },
      "status" : {
        "job_state" : "started",
        "current_position" : {
          "@timestamp.date_histogram" : 1556690700000,
          "beat.name.keyword.terms" : "ukelastic-test7",
          "dc.keyword.terms" : "ld5",
          "os.keyword.terms" : "linux",
          "tags.keyword.terms" : "system"
        },
        "upgraded_doc_id" : true
      },
      "stats" : {
        "pages_processed" : 42,
        "documents_processed" : 516604,
        "rollups_indexed" : 7464,
        "trigger_count" : 19,
        "index_time_in_ms" : 1466,
        "index_total" : 23,
        "index_failures" : 0,
        "search_time_in_ms" : 999,
        "search_total" : 42,
        "search_failures" : 0
      }
    }
  ]
}
  • The new index metricbeat_test4 :

image
image.png1757×779 64.8 KB

image
image.png1758×693 56.3 KB

image
image.png1755×976 154 KB

  • The original index metricbeat :

image
image.png1760×981 111 KB

Could someone help me out?
What am I missed during I create rollup job? whey there is not value for the field time for rollup index metricbeat_test4?

Best Regards
Robin

5

Which version are you running?

6

Hi @Christian_Dahlqvist,
I'm using version 6.7 for the elastic cloud.

8

Hi elsatic team,

Any update or feedback about this?

Thanks

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.