Scripted fields versus scripts in discovery objects

[gyannea]

In Kibana I have found two ways to create an additional field based upon what is in the documents.

1. Use scripted fields in the index patterns. Since I want to convert a parameter that is in seconds to hours the script is

params._source.hubEvent.downtime / 3600.0;

with name 'downtimeInHrs'. Or

2. Place a full painless script into a search filter

{
  "script_fields": {
    "downtimeInHrs": {
      "script": {
        "lang": "painless",
        "source": "params._source.hubEvent.downtime * params.multiplier",
        "params": {
          "multiplier": 0.0002777777778
        }
      }
    }
  },
the filter 
...

Both generate the field 'downtimeInHrs' and in both cases they are visible in the discovered document and in the left hand column.

Using #1 I can use the field in visualizations. And the field is prefixed with '#'. I do not know what that '#' prefix means. However, if I do a search that does not contain a hubEvent.downtime, I get a null pointer exception. Doing an

if (params._source.hubEvent.downtime !=null)

does not solve the problem. There is still a null pointer exception.

Using #2 I cannot use the field in visualizations. In the left hand window of the discovery the field is prefixed by a '?'. There is no null pointer issue since the filter only works with docs that have a hubEvent.downtime field.

I am so confused. The docs do not explain how any of this works . I want to use the script in the discovery since my understanding is that it is much more efficient. But it is useless if I cannot use it in visualizations.

[lukas (Lukas Olson)]

What version of Kibana are you using?

I'm not sure I understand how (2) is working for you. When you say "search filter", what do you mean? Are you referring to the query bar or the add filter dialog?

Regardless, both of the methods you described are deprecated and will be removed at some point. It is recommended that you use runtime fields to accomplish what you're saying. These can be used both in Discover and inside visualizations.

[gyannea]

Lucas,

I am using version 7.1.1. It does not have the runtime fields options. That sounds like a pretty cool option for what I am trying to do and hopefully it won't be so confusing. That comes in at 7.13 I think. I am not sure if I have control over the version I get to use....

In any case, I create the base search using the 'add filter' dialog using Lucene. I place the painless script is at the start of the search JSON and it converts the document parameter hubEvent.downtime (in seconds) to a new field 'downtimeInHrs'). It looks as follows:

{
  "script_fields": {
    "downtimeInHrs": {
      "script": {
        "lang": "painless",
        "source": "params._source.hubEvent.downtime * params.multiplier",
        "params": {
          "multiplier": 0.0002777777778
        }
      }
    }
  },
  "highlightAll": true,
  "version": true,
  "query": {
    "language": "lucene",
    "query": ""
  },
  "filter": [
    {
      "aggs": {
        "downtime_stats": {
          "stats": {
            "field": "hubEvent.downtime"
          }
        },
        "total_downtime": {
          "sum": {
            "field": "hubEvent.downtime"
          }
        }
      },
      "from": 0,
      "query": {
        "bool": {
          "filter": [
            {
              "range": {
                "hubEvent.downtime": {
                  "gte": 10
                }
              }
            }
          ],
          "must": [
            {
              "match": {
                "hubEvent.code": "qmiUp"
              }
            },
            {
              "exists": {
                "field": "hubEvent.downtime"
              }
            }
          ]
        }
      },
      "size": 25,
      "sort": [
        {
          "hubEvent.downtime": {
            "order": "desc"
          }
        }
      ],
      "meta": {
        "type": "custom",
        "disabled": false,
        "negate": false,
        "alias": "downtime",
        "key": "aggs",
        "value": "{\"downtime_stats\":{\"stats\":{\"field\":\"hubEvent.downtime\"}},\"total_downtime\":{\"sum\":{\"field\":\"hubEvent.downtime\"}}}",
        "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index"
      },
      "$state": {
        "store": "appState"
      }
    }
  ],
  "indexRefName": "kibanaSavedObjectMeta.searchSourceJSON.index"
}

So within the search I assure that hubEvent.downtime is present.

This search works as it gets the docs I want and there is a 'downtimeInHrs' field present, both appended to the document and it is listed in the left hand column. I can add the new field as a table and the values are correct. However, clicking on the new field shows values but no visualization option. So something is not right.

[lukas (Lukas Olson)]

Ahh, I see. In the meantime, if you alter your script as follows, it should work for you in both Discover and Visualize:

return doc["hubEvent.downtime"].size() > 0 ? doc["hubEvent.downtime"].value / 3600.0 : null;

Let me know if that works!

[warkolm (Mark Walkom)]

@gyannea

I am using version 7.1.1

Please note that version is very much EOL and no longer supported, you should be looking to upgrade as a matter of urgency.

(Sorry I am not directly resolving your issue, but you are running a very old version!)

[gyannea]

This looks good. I have not yet got a null pointer exception. Also looking to update.

I am confused by the use of

doc['hubEvent.downtime']

versus

params._source.hubEvent.downtime

I could not get my simple one-line script to work using 'doc' but it did work with the 'params' variant. I do not understand what these two syntaxes are doing.

[gyannea]

I am looking to get the powers that be to upgrade, especially given what they want me to do. Would like to have that runtime option. Hope it is a simple as it sounds. It's certainly sounds like what I want to perform the task.

[gyannea]

@lukas I have spent days trying to figure out how to capture the time period the user selects through the Kibana interface, for example, a month, year, week to date, etc. I started a new topic and have not gotten any response. It may not be possible.

What I want to display is, for example, the percent of time a device was down in the period picked by the user. This is not an indexed data field. But Kibana uses the range in its search.

        {
          "range": {
            "date": {
              "format": "strict_date_optional_time",
              "gte": "2023-02-01T05:00:00.000Z",
              "lte": "2023-03-02T15:11:30.784Z"
            }
          }
        }

Kibana can get those values but I can't!

[lukas (Lukas Olson)]

Unfortunately there isn't a way I'm aware of to get those values in a scripted field.

What is the calculation you're trying to do? Can you give an example?

[gyannea]

The user enters a time period using the timepicker. I want to calculate the percent of time a given device did not have connectivity during that period.

There was a request for this feature back in 2017 (several added they would also like it). The response was to make an issue requesting the feature. Guess it never got addressed.