Why are these rollup documents missing an ID?

Hello - I've run into a major problem and would really appreciate an explanation. Thank you for your input.

We are using rollups to process metricbeat data for analysis. We have three servers, say A, B and C. Having these server ID's is essential for keeping track of what we are storing in the rollups. Everything will use these ID's!

But several documents in the rollups do not contain an ID. The original metricbeat data always contains one of these ID's. So what are these documents with missing ID's?

Here is the rollup JSON (which I implemented using the API):

{
"config": {
"id": "cpu_minute_test_rollup",
"index_pattern": "metricbeat-*",
"rollup_index": "cpu_minute_test",
"cron": "0 * * * * ?",
"groups": {
"date_histogram": {
"calendar_interval": "1h",
"field": "@timestamp",
"delay": "1m",
"time_zone": "UTC"
},
"histogram": {
"interval": 5,
"fields": [
"system.cpu.total.pct"
]
},
"terms": {
"fields": [
"system.cpu.total.pct",
"fields.server.id",
"system.cpu.cores"
]
}
},
"metrics": [
{
"field": "system.cpu.total.pct",
"metrics": [
"value_count",
"sum",
"min",
"max",
"avg"
]
}
],
"timeout": "20s",
"page_size": 1000
},
"status": {
"job_state": "started",
"current_position": {
"@timestamp.date_histogram": 1565316000000,
"fields.server.id.terms": "A",
"system.cpu.cores.terms": 2,
"system.cpu.total.pct.histogram": 0,
"system.cpu.total.pct.terms": 0.787
},
"upgraded_doc_id": true
},
"stats": {
"pages_processed": 136,
"documents_processed": 6442666,
"rollups_indexed": 133985,
"trigger_count": 2,
"index_time_in_ms": 32935,
"index_total": 134,
"index_failures": 0,
"search_time_in_ms": 31237,
"search_total": 136,
"search_failures": 0
}
}

Here is what we expect documents to look like (output from rollup):

{
"_index": "cpu_minute_test",
"_type": "_doc",
"_id": "cpu_minute_test_rollup$MmisCtl54fwO4f2-fQABCg",
"_version": 1,
"_score": null,
"_source": {
"@timestamp.date_histogram.time_zone": "UTC",
"@timestamp.date_histogram.timestamp": 1563908400000,
"system.cpu.total.pct.histogram.interval": 5,
"system.cpu.total.pct.terms._count": 932,
"system.cpu.cores.terms.value": null,
"@timestamp.date_histogram.interval": "1h",
"@timestamp.date_histogram._count": 932,
"system.cpu.total.pct.terms.value": null,
"system.cpu.total.pct.value_count.value": 0,
"system.cpu.total.pct.histogram.value": null,
"system.cpu.total.pct.sum.value": 0,
"_rollup.version": 2,
"fields.server.id.terms.value": "A",
"fields.server.id.terms._count": 932,
"system.cpu.total.pct.histogram._count": 932,
"system.cpu.cores.terms._count": 932,
"_rollup.id": "cpu_minute_test_rollup"
},
"sort": [
-9223372036854776000
]
}

But several are missing this essential ID. They look like this:

{
"_index": "cpu_minute_test",
"_type": "_doc",
"_id": "cpu_minute_test_rollup$gKcjJihI5w0S1y7_ZuXplA",
"_version": 1,
"_score": null,
"_source": {
"@timestamp.date_histogram.time_zone": "UTC",
"@timestamp.date_histogram.timestamp": 1563908400000,
"system.cpu.total.pct.histogram.interval": 5,
"system.cpu.total.pct.terms._count": 1371,
"system.cpu.cores.terms.value": null,
"@timestamp.date_histogram.interval": "1h",
"@timestamp.date_histogram._count": 1371,
"system.cpu.total.pct.terms.value": null,
"system.cpu.total.pct.value_count.value": 0,
"system.cpu.total.pct.histogram.value": null,
"system.cpu.total.pct.sum.value": 0,
"_rollup.version": 2,
"fields.oss.id.terms.value": null,
"fields.oss.id.terms._count": 1371,
"system.cpu.total.pct.histogram._count": 1371,
"system.cpu.cores.terms._count": 1371,
"_rollup.id": "cpu_minute_test_rollup"
},
"sort": [
-9223372036854776000
]
}

What are these documents without an ID? Is this a normal thing or have I messed something up?

Thanks again!

I am a little confused why in this section it is not just only the server.id since that seems to be the term you want to group by. Did you create this roll-up via just the API or did you use the UI?

"terms": {
"fields": [
"system.cpu.total.pct",
"fields.server.id",
"system.cpu.cores"
]

Perhaps..

"terms": {
"fields": [
"fields.server.id"
]
}

Thanks for the reply.

If we use the UI the JSON file looks the same. I literally copied the JSON file from the UI and used it in the API. The other fields { "system.cpu.total.pct", and "system.cpu.cores"} need to show up in Kibana as well.

Making the change you suggested does not help. I just created a new rollup and indexed it. Some of the documents still have a blank "fields.server.id" field. Like this one:
{
"_index": "cpu_minute_test2",
"_type": "_doc",
"_id": "cpu_minute_test_rollup2$oh7B-Xiw6AQmOs_3HcRq0w",
"_version": 1,
"_score": null,
"_source": {
"@timestamp.date_histogram.time_zone": "UTC",
"@timestamp.date_histogram.timestamp": 1563908400000,
"system.cpu.total.pct.avg._count": 104,
"system.cpu.total.pct.histogram.interval": 5,
"@timestamp.date_histogram.interval": "1h",
"@timestamp.date_histogram._count": 104,
"system.cpu.total.pct.avg.value": 11.507,
"system.cpu.total.pct.value_count.value": 104,
"system.cpu.total.pct.histogram.value": 0,
"system.cpu.total.pct.max.value": 0.46,
"system.cpu.total.pct.sum.value": 11.507,
"_rollup.version": 2,
"fields.server.id.terms.value": null,
"fields.server.id.terms._count": 104,
"system.cpu.total.pct.histogram._count": 104,
"system.cpu.total.pct.min.value": 0,
"_rollup.id": "cpu_minute_test_rollup2"
},
"sort": [
-9223372036854776000
]
}

Curious ... what version of Elasticsearch and Kibana are you using?

I just created the following in Version 7.0 via the UI.

I aggregated on host.hostname.keyword... note the terms section is just what I want to aggregate on not the actual metrics.

Also Have you verified fields.server.id is in fact a term you may need to use fields.server.id.keyword although it should have errored if it was not a keyword and / or perhaps you have document in the source that are missing fields.server.id values?

{
  "id": "metric-rollup",
  "index_pattern": "metricbeat-*",
  "rollup_index": "metric-rollup-test",
  "cron": "0 * * * * ?",
  "page_size": 1000,
  "groups": {
    "date_histogram": {
      "interval": "60m",
      "delay": "1d",
      "time_zone": "UTC",
      "field": "@timestamp"
    },
    "terms": {
      "fields": [
        "host.hostname.keyword"
      ]
    }
  },
  "metrics": [
    {
      "field": "system.load.1",
      "metrics": [
        "avg",
        "max",
        "min"
      ]
    },
    {
      "field": "system.cpu.total.norm.pct",
      "metrics": [
        "avg",
        "max",
        "min"
      ]
    }
  ]
}

And My Results look like this, and I did not find any docs missing host.hostname.keyword.terms.value

   "hits" : [
      {
        "_index" : "metric-rollup-test",
        "_type" : "_doc",
        "_id" : "metric-rollup$8wwMVox2YiuTEPfb2f26sA",
        "_score" : 1.0,
        "_source" : {
          "@timestamp.date_histogram.time_zone" : "UTC",
          "@timestamp.date_histogram.timestamp" : 1558026000000,
          "host.hostname.keyword.terms._count" : 16,
          "system.load.1.max.value" : 2.890000104904175,
          "_rollup.version" : 2,
          "system.load.1.min.value" : 2.890000104904175,
          "system.load.1.avg.value" : 2.890000104904175,
          "@timestamp.date_histogram.interval" : "60m",
          "system.load.1.avg._count" : 1.0,
          "@timestamp.date_histogram._count" : 16,
          "host.hostname.keyword.terms.value" : "pcf-es-instance-1",
          "_rollup.id" : "metric-rollup"
        }
      },
      {
        "_index" : "metric-rollup-test",
        "_type" : "_doc",
        "_id" : "metric-rollup$Nzyt4e-3cYkVQ9wKkTuOrA",
        "_score" : 1.0,
        "_source" : {
          "@timestamp.date_histogram.time_zone" : "UTC",
          "@timestamp.date_histogram.timestamp" : 1558026000000,
          "host.hostname.keyword.terms._count" : 38,
          "system.load.1.max.value" : 0.18000000715255737,
          "_rollup.version" : 2,
          "system.load.1.min.value" : 0.15000000596046448,
          "system.load.1.avg.value" : 0.33000001311302185,
          "@timestamp.date_histogram.interval" : "60m",
          "system.load.1.avg._count" : 2.0,
          "@timestamp.date_histogram._count" : 38,
          "host.hostname.keyword.terms.value" : "pcf-load-0",
          "_rollup.id" : "metric-rollup"
        }
      },
      {
        "_index" : "metric-rollup-test",
        "_type" : "_doc",
        "_id" : "metric-rollup$UWSk_ybHucEvs5R0vY9tbQ",
        "_score" : 1.0,
        "_source" : {
          "@timestamp.date_histogram.time_zone" : "UTC",
          "@timestamp.date_histogram.timestamp" : 1558026000000,
          "host.hostname.keyword.terms._count" : 42,
          "system.load.1.max.value" : 0.03999999910593033,
          "_rollup.version" : 2,
          "system.load.1.min.value" : 0.03999999910593033,
          "system.load.1.avg.value" : 0.07999999821186066,
          "@timestamp.date_histogram.interval" : "60m",
          "system.load.1.avg._count" : 2.0,
          "@timestamp.date_histogram._count" : 42,
          "host.hostname.keyword.terms.value" : "pcf-mysql-0",
          "_rollup.id" : "metric-rollup"
        }
      },

So I guess I would I expect it to work as you want.... I am not saying what you are seeing is not happening perhaps it is an issue.

also it helps is you format your json using the </> button above

Thank you very much for the reply.

Our metricbeat data always has our server ID field - so it isn't that. But I'll look into your keyword idea and see if that works. When I have added .keyword to the end of our server ID field it throws an error that says:

{
    "type": "rollup_action_request_validation_exception",
    "reason": "Validation Failed: 1: Could not find a [numeric] or [keyword/text] field with name [server.id.keyword] in any of the indices matching the index pattern.;"
}

Perhaps our server ID was not added correctly? I should also add that the server ID is a keyword.

We are using Kibana and Elasticsearch 7.2

Perhaps you could try to create a job with aggregation host.hostname.keyword see if it works the way expected.

And Yes server.id will need to have a type keyword in the template / mapping I suspect it is just a text field. It can be both text and keyword but will definitely need a keyword to be aggregatable.

Can you post your modified metricbeat mapping? And in your index pattern it should show up as aggregatable..

Great. Thank you again for the reply.

I'll have to get that done Monday morning and will let you know how it went!

Hello again,

Can you suggest what the simplest way to turn our server ID into a keyword is? We spent some time trying various things to define it as a keyword but nothing worked. The server ID field still is just a text field.

Thank you!

Hi @EricJohnson

How did you add the field? Like this?
https://www.elastic.co/guide/en/beats/metricbeat/current/add-fields.html#add-fields

If you look at your server.id field in discover does it look like this?

You probably need to go refresh the index pattern.... have you done that?

After you do that you should be able to see something like this...

It may take a few minutes to show up in Discover

you can look at the actual mapping after you add the field

GET /metricbeat-7.2.0

and is you look you should see something like

"mynewcoolfield" : {
  "properties" : {
    "id" : {
      "type" : "keyword",
      "ignore_above" : 1024
    }
  }
},

Then you can give the metrics rollup another try

1 Like

Your responses have been extremely helpful. Thank you again for your time!

We decided to trash the old data and restart Metricbeat and the rollup issue went away. The server ID fields are keywords and seem to be behaving as they should. Now, fortunately, all of the documents have the correct fields. We will still need to add new keywords to existing systems - so I'll share what you posted to help everyone.

Thanks again!

1 Like