ELK watcher : painless script - stream.filter

Elastic Stack Elasticsearch
1

Below is the reslut of Elastic query with derivative aggregation

"utils_per_5m": {
"buckets": [
{
"key_as_string": "2020-04-15T21:10:00.000Z",
"doc_count": 1,
"utils": {
"value": 924
},
"key": 1586985000000
},
{
"key_as_string": "2020-04-15T21:15:00.000Z",
"doc_count": 1,
"utils": {
"value": 0
},
"utils_deriv": {
"value": -924
},
"key": 1586985300000
}
]
}

utils_per_5m.buckets array has two objects. Since using derivative, only the second object in the array will have util_deriv .
Usecase : filter utils_per_5m.buckets array (2 objects ) with the condition as
utils_per_5m.buckets.stream().filter(poll -> poll.utils_deriv.value != null && poll.utils_deriv.value < 0).collect(Collectors.toList())

while executing the script in painless, getting NPE, though null check in pace for poll.utils_deriv.value NULL prior to poll.utils_deriv.value < 0 .

But , poll.utils.value == 0 filter working , utils.value exists in both the utils_per_5m.buckets array objects.
utils_per_5m.buckets.stream().filter(poll -> poll.utils.value == 0).collect(Collectors.toList())

Can you clarify why poll.utils_deriv.value filter throwing NPE even after checking NULL , also second object in the array has matching data.

2

From a quick look, wouldn't you need to check poll.utils_deriv for null already? If poll.utils_deriv is null then poll.utils_deriv.value will run into a NPE.

PS: Please format your code for better readability :slight_smile:

3

works.. thanks for the insight

4

Here's my result

"aggregations": {
"name": {
"buckets": [
{
"doc_count": 8,
"port": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"doc_count": 2,
"utils_per_5m": {
"buckets": [
{
"key_as_string": "2020-04-17T18:00:00.000Z",
"doc_count": 1,
"utils": {
"value": 924
},
"key": 1587146400000
},
{
"key_as_string": "2020-04-17T18:05:00.000Z",
"doc_count": 1,
"utils": {
"value": 923
},
"utils_deriv": {
"value": -1
},
"key": 1587146700000
}
]
},
"key": "Port-1"
},
{
"doc_count": 2,
"utils_per_5m": {
"buckets": [
{
"key_as_string": "2020-04-17T18:00:00.000Z",
"doc_count": 1,
"utils": {
"value": 876
},
"key": 1587146400000
},
{
"key_as_string": "2020-04-17T18:05:00.000Z",
"doc_count": 1,
"utils": {
"value": 870
},
"utils_deriv": {
"value": -6
},
"key": 1587146700000
}
]
},
"key": "Port-2"
}
]
},
"key": "Avenal"
},{ }
]
}
}

painless script :
POST _scripts/deviceutil-parser
{
"script": {
"lang": "painless",
"source": "return ['host_port_util_map': ctx.payload.aggregations.name.buckets.stream().map(p -> [p.key, p.port.buckets.stream().map(bkts -> [bkts.key,bkts.utils_per_5m.buckets.stream().filter(poll -> poll.utils.value == 0 && (poll.utils_deriv != null && poll.utils_deriv.value < 0)).collect(Collectors.toList())]).collect(Collectors.toList())]).collect(Collectors.toList())];"
}
}

Script output:
[Avenal, [ [Port-1, ], [[Port-2, ] ]

Question : Since there is mo matching records, getting empty arrays. Is there a better way to filter the steam to return only matching data set ie non empty arrays. Pls ask if you need more info