Script field returns wrong value

Elastic Stack Kibana
#1

Hi,

I'm
I'm trying to use the script field and I started with an existing field.
The value of the script field is different from the value of the field itself.
I'm using metricbeat "system.cpu.idle.pct" and have defined a script field for it as well: doc['system.cpu.idle.pct'].value

Their values are not matching.
Here is the request body and the screenshot.

Would you please kindly help me? What am I doing wrong?

Thanks,
Parisa.

cpuIdle.png1325×1021 22.3 KB 

{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        {
          "query_string": {
            "query": "*",
            "analyze_wildcard": true
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": 1488396703669,
              "lte": 1488397603670,
              "format": "epoch_millis"
            }
          }
        }
      ],
      "must_not": []
    }
  },
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "interval": "30s",
        "time_zone": "America/New_York",
        "min_doc_count": 1
      },
      "aggs": {
        "1": {
          "avg": {
            "field": "system.cpu.idle.pct"
          }
        },
        "3": {
          "avg": {
            "script": {
              "inline": "doc['system.cpu.idle.pct'].value",
              "lang": "painless"
            }
          }
        }
      }
    }
  }
}
#2

I'm unable to reproduce this locally, if you execute a query like this in console, what does it return?

POST metricbeat*/_search?size=0
{
  "aggs": {
    "1": {
      "avg": {
        "script": {
          "inline": "doc['system.cpu.idle.pct'].value",
          "lang": "painless"
        }
      }
    }
  }
}

Console reproduction steps I used: https://gist.github.com/spalger/16eb2597bd9add36b594e63e73aa4295

#3

The .metricbeat should be replaced with .?
If yes this is what I get back:

{
"took": 1,
"timed_out": false,
"_shards": {
"total": 0,
"successful": 0,
"failed": 0
},
"hits": {
"total": 0,
"max_score": 0,
"hits": []
}
}

#4

Sorry, .metricbeat should be replaced with metricbeat*

#5

Hi again,

When I put my index instead of metricbeat* and run it, it does not return anything.
It seems that it is taking forever to return.

If I change the index name with something invalid, it will return what I posted before.

#6

I think my data was too big there.
I ran it on a smaller instance and here is what I got:

{
"took": 15,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 15738,
"max_score": 0,
"hits": []
},
"aggregations": {
"1": {
"value": 0.002453615453043589
}
}
}

The problem is the value that is returning is from a different scale that the actual value.
On the screenshot that I have posted the scripted field has values but they are too small.

#7

Hi @spalger,

I have followed all your steps in the reproduce-discuss page and I still get the wrong answer. ([https://gist.github.com/spalger/16eb2597bd9add36b594e63e73aa4295]
Here is what I get:

{
"took": 277,
"timed_out": false,
"_shards": {
"total": 10,
"successful": 10,
"failed": 0
},
"hits": {
"total": 19437,
"max_score": 0,
"hits": []
},
"aggregations": {
"1": {
"value": 0.00251582034264547
}
}
}
Would you please kindly let me know how I can open a bug on Kibaba scripted field?

Thanks,
Parisa.

#8

Hi @spalger,

If I delete the metricbeat index and recreate it with your steps and run the command then the result is correct.
{
"took": 4,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 4,
"max_score": 0,
"hits": []
},
"aggregations": {
"1": {
"value": 0.035
}
}
}

However the problem is that metricbeat is the one collecting data for me and this does not work with the data collected from the beat.

#9

Hi,

I've found the solution and the reason this strange behaviour happens.
If I create the visualization over the "cpu stat" search instead of the index itself it will give me the correct result.

Taking a look at the request that is sending you'll see this:

{
"size": 0,
"aggs": {
"2": {
"date_histogram": {
"field": "@timestamp",
"interval": "30s",
"time_zone": "America/New_York",
"min_doc_count": 1
},
"aggs": {
"1": {
"avg": {
"script": {
"inline": "doc['system.cpu.idle.pct'].value",
"lang": "painless"
}
}
}
}
}
},
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "metricset.module: system AND metricset.name: cpu",
"analyze_wildcard": true
}
},
{
"range": {
"@timestamp": {
"gte": 1489434545480,
"lte": 1489435445480,
"format": "epoch_millis"
}
}
}
],
"must_not": []
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"*": {}
},
"require_field_match": false,
"fragment_size": 2147483647
}
}

The search filters if there is no value for the field and do not set it to value 0.
The metrics beat does not send values for all the timeslots. The timeslots that do not have the fields value will get the value zero for the script field. The search solves this by putting a filter.

I think This is not a bug on elasticsearch/ kibana. But metric beat should warn the users that this can happen using the script fields over the index itself.

#10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.