Can't divide two fields when querying an index

Luis_P · August 3, 2021, 1:25am

Hello, I'm new to Elasticsearch and Kibana and I've been trying to create a watcher that alerts me when the free disk space ratio is lower than a defined value. I believe I've managed to build a working watcher however, I can't create a field with a division of two sum_bucket results needed for the total ratio. I'll post my watcher JSON below, only leaving the actions part out:

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          ".monitoring-es-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "timestamp": {
                      "gte": "{{ctx.trigger.scheduled_time}}||-5m"
                    }
                  }
                },
                {
                  "term": {
                    "type": "node_stats"
                  }
                }
              ]
            }
          },
          "aggs": {
            "nodes": {
              "terms": {
                "field": "source_node.name",
                "size": 100
              },
              "aggs": {
                "total_in_bytes": {
                  "max": {
                    "field": "node_stats.fs.total.total_in_bytes"
                  }
                },
                "available_in_bytes": {
                  "max": {
                    "field": "node_stats.fs.total.available_in_bytes"
                  }
                },
                "free_ratio": {
                  "bucket_script": {
                    "buckets_path": {
                      "available_in_bytes": "available_in_bytes",
                      "total_in_bytes": "total_in_bytes"
                    },
                    "script": "params.available_in_bytes / params.total_in_bytes"
                  }
                },
        "total_in_gb": {
          "bucket_script": {
            "buckets_path": {
              "total_in_bytes": "total_in_bytes"
            },
            "script": "Math.round((params.total_in_bytes/1073741824)*100)/100"
          }
        },
        "available_in_gb": {
          "bucket_script": {
            "buckets_path": {
              "available_in_bytes": "available_in_bytes"
            },
            "script": "Math.round((params.available_in_bytes/1073741824)*100)/100"
          }
        }
              }
            },
            "sum_total": {
              "sum_bucket": {
                "buckets_path": "nodes>total_in_gb"
              }
            },
            "sum_available": {
              "sum_bucket": {
                "buckets_path": "nodes>available_in_gb"
              }
            }
          },
          "size": 0
        }
      }
    }
  },
  "condition": {
    "script": {
    "lang": "painless",
    "source": "return ((ctx.payload.aggregations.sum_available.value / ctx.payload.aggregations.sum_total.value) < ctx.metadata.lower_bound);"
    }
  },
  "actions": {},
  "metadata": {
    "lower_bound": 1
  },
  "throttle_period_in_millis": 1800000
}

I want a field which is the division of sum_available by sum_total. This result would help me providing the ratio to the logs. I've tried several approaches with no success and I'd appreciate some help if it's possible. Thank you!

spinscale · August 3, 2021, 7:05am

Hey,

can you add some more context of what is missing in addition of pasting several hundred lines of a watch? Is the query working as expected, but the condition does not work? Or is the data provided not showing the data as needed? This would help to tackle the problem to figure out if this is a data problem or a watcher problem.

Thanks!

--Alex

Luis_P · August 3, 2021, 8:50am

Hello Alex,

Thank you for the reply!

The query is working as expected and so do the aggregations and the condition so the watcher itself is working as intended, the only thing I'm missing is an extra calculated field for info logging. I get the total, available disk space and free disk space ratio for each node. I get the sum, from all the nodes, of the total and available disk space, through a sum_bucket of the results for each node. What I'm trying to accomplish is obtaining a new field which divides the results of the two already working sum_bucket results. Next is the relevant part of the response to the query, I've left only one of the resulting buckets for reference:

"aggregations" : {
    "nodes" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "...",
          "doc_count" : 30,
          "total_in_bytes" : {
            "value" : 1.073605505024E12
          },
          "available_in_bytes" : {
            "value" : 1.001956671488E12
          },
          "free_ratio" : {
            "value" : 0.9332633512023596
          },
          "total_in_gb" : {
            "value" : 999.0
          },
          "available_in_gb" : {
            "value" : 933.0
          }
        }
      ]
    },
    "sum_total" : {
      "value" : 3097.0
    },
    "sum_available" : {
      "value" : 2880.0
    }
  }

This last part of the response I would like it to have an extra field like:

    },
    "sum_total" : {
      "value" : 3097.0
    },
    "sum_available" : {
      "value" : 2880.0
    },
    "total_free_ratio": {
       "value" : // this would be equal to sum_available / sum_total
    }

Once again, thank you!

Luis

spinscale · August 3, 2021, 9:17am

so, while you probably could figure this out with another bucket script aggregation, you could just do this within your condition as well by doing

ctx.total_free_ratio = ctx.payload.aggregations...sum_available / ctx.payload.aggregations...sum_total;
return your_condition_goes_here;

Just in case that helps you to get up and running faster

Luis_P · August 3, 2021, 9:21am

That's how I managed to have the watcher condition working, but I can't access the resulting ratio to use when sending an email when the condition is met. The value is not in the ctx.payload object. This is why I wanted the field. I tried using a bucket script aggregation with no success as when I put it next to the sum_bucket's it says that it needs to be inside another aggregation, so I'm lost on how to create the field with this value

spinscale · August 3, 2021, 9:28am

take another look at the snippet above, it stores a variable in the context to do exactly that

Luis_P · August 3, 2021, 10:56am

Oh you're right, I'm sorry! I only saw it as a variable to use in the condition and not as one "created" in the ctx object. However, I tested it and it's not creating the field on ctx object. This is the condition as it shows when I simulate the watcher:

"condition": {
    "script": {
      "source": "ctx.total_free_ratio = ctx.payload.aggregations.sum_available.value / ctx.payload.aggregations.sum_total.value; return (ctx.total_free_ratio < ctx.metadata.lower_bound);",
      "lang": "painless"
    }
  }

And this is the resulting object as shown in the email when I print {{ctx}} removing the buckets field to reduce the string:

{metadata={lower_bound=1, name=Disk Space Test 2, xpack={type=json}}, watch_id=_inlined_, payload={_shards={total=7, failed=0, successful=7, skipped=0}, hits={hits=[], total=95, max_score=null}, took=2, timed_out=false, aggregations={sum_total={value=3097.0}, nodes={doc_count_error_upper_bound=0, sum_other_doc_count=0, buckets=[]}, sum_available={value=2878.0}}}, id=_inlined__a45c0cab-bb50-4190-ae29-d7ccdb1d47ec-2021-08-03T09:53:05.7974636Z, trigger={triggered_time=2021-08-03T09:53:06.797Z, scheduled_time=2021-08-03T09:54:05.797Z}, vars={}, execution_time=2021-08-03T09:53:05.7974636Z}

There is no ctx.total_free_ratio field. I tried it on the email itself {{ctx.total_free_ratio}} but it show no value

spinscale · August 3, 2021, 11:12am

try setting within the ctx.payload variable not just ctx

Luis_P · August 3, 2021, 11:56am

Yes, that worked!! Thank you so much, I am now able to access and use that value for logging! I appreciate your attention on this issue!

Luis_P · August 3, 2021, 11:59am

I will leave as a suggestion perhaps having a pre-configured alert for server/cluster/node free disk space ratio similar to what already exists for CPU usage and Memory as I think it would be very useful without creating a watcher similar to mine.

system · August 31, 2021, 11:59am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.