Scripted Fields + Filtering

Rob_wylde · January 25, 2021, 3:12am

I am trying to use painless to calculate the duration of 2 ISO8601 timestamps The field is correctly populated with the returned string i have defined. However the field can't not be filtered upon / with. This is my first time utilizing script fields and painless scripting.

Language: Painless
Type: String
Format: String
Transform: none

if (doc.containsKey('start_date') && doc.containsKey('last_update'))
{
    if (doc['start_date'].size() > 0 && doc['last_update'].size() > 0 )
    {
        def time_diff = doc['last_update'].value.millis - doc['start_date'].value.millis;
        def minutes = time_diff / 60000;
//        minutes > 30 ? 'Open': 'Closed'

        if ( minutes > 30 )
        {
            return 'Open'
        }else{
            return 'Closed'
        }
    }
}

When i go to filter by clicking the + or - button next to the field or using the 'add filter' button i get an error.

search_phase_execution_exception


_construct@https://domain.com/35949/bundles/core/core.entry.js:6:4859
Wrapper@https://domain.com/35949/bundles/core/core.entry.js:6:4249
_createSuperInternal@https://domain.com/35949/bundles/core/core.entry.js:6:3388
HttpFetchError@https://domain.com/35949/bundles/core/core.entry.js:6:6016
_callee3$@https://domain.com/35949/bundles/core/core.entry.js:6:59535
l@https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:380:1740519
s/o._invoke</<@https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:380:1740273
_/</e[t]@https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:380:1740876
fetch_asyncGeneratorStep@https://domain.com/35949/bundles/core/core.entry.js:6:52652
_next@https://domain.com/35949/bundles/core/core.entry.js:6:52992

Any suggestions?

Thanks for reading

stu · January 25, 2021, 11:00pm

Hi @Rob_wylde
What are your mappings for start_date and last_update? If they're date then you can do

def time_diff = doc['last_update'].value.toInstant().toEpochMilli() - 
                doc['start_date'].value.toInstant().toEpochMilli()

If you need to parse them because they're keyword, you'll need to parse them using some Datetime methods.

Something like:

ZonedDateTime start = ZonedDateTime.parse(doc['last_update.keyword'].value);
ZonedDateTime last = ZonedDateTime.parse(doc['last_update.keyword'].value);
def time_diff = start.toInstant().toEpochMilli() - 
                last.toInstant().toEpochMilli();

It's hard to tell, can you provide the exception returned by elasticsearch? It'll look something like:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "script_exception",
        "reason" : "runtime error",
        "script_stack" : []
...
       "script": ""
...
        "lang" : "painless",
        "position" : {
          "offset" : 375,
          "start" : 331,
          "end" : 475
        }

Rob_wylde · January 26, 2021, 5:24am

Stu,

Thanks for the reply. I've tried value.toInstant().toEpochMilli() as suggested as both fields are mapped as 'date'. Once again the scripted field gets populated but I can not filter on this field. I'm not sure how to get the exception error from elastic within kibana

Below is what i see in the Firefox console

statusCode	400
error	"Bad Request"
message	"status_exception"
attributes	Object { error: {…} }
error	Object { type: "status_exception", reason: "error while executing search", caused_by: {…} }
type	"status_exception"
reason	"error while executing search"
caused_by	Object { type: "search_phase_execution_exception", phase: "fetch", grouped: true, … }
type	"search_phase_execution_exception"
reason	""

Also and probably more likely the issue.

Firefox
Content Security Policy: The page’s settings blocked the loading of a resource at inline (“script-src”).
Chrome
Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/389hGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
^ A single error about an inline script not firing due to content security policy is expected!

Should be noted that i'm running ECE 2.6 with 7.10.1 .. I'm not seeing anything glaring in the logging and metrics kibana interface ( where i ship my logs ).I'm also using Nginx as a reverse proxy but I've tried filtering via this scripted field while bypassing nginx and experience the same behavior.

Thanks

stu · January 26, 2021, 3:47pm

Can you expand those objects? That's where the juicy parts of the error is.

Also and probably more likely the issue.

The error you quoted above indicates that the request has successful hit elasticsearch, so the CSP problem is likely unrelated.

Rob_wylde · January 26, 2021, 7:35pm

Stu,

XHRPOSThttps://domain.com/internal/search/ese
[HTTP/1.1 400 Bad Request 183ms]

    1

    {"statusCode":400,"error":"Bad Request","message":"status_exception","attributes":{"error":{"type":"status_exception","reason":"error while executing search","caused_by":{"type":"search_phase_execution_exception","reason":"","phase":"fetch","grouped":true,"failed_shards":[{"shard":0,"index":"trouble_conditions-2021","node":"Qnt6JaU9RW24Og8rpAqrjA","reason":{"type":"query_shard_exception","reason":"failed to create query: compile error","index_uuid":"Ck8UTBhATpKnUdhSGRmqPg","index":"trouble_conditions-2021","caused_by":{"type":"script_exception","reason":"compile error","script_stack":["... rn s.get() == v;}compare(() -> { if (doc.containsK ...","                             ^---- HERE"],"script":"boolean compare(Supplier s, def v) {return s.get() == v;}compare(() -> { if (doc.containsKey('start_date') && doc.containsKey('last_update'))\n{\n    if (doc['start_date'].size() > 0 && doc['last_update'].size() > 0 )\n    {\n        def time_diff = doc['last_update'].value.toInstant().toEpochMilli() - doc['start_date'].value.toInstant().toEpochMilli();\n        def minutes = time_diff / 60000;\n        minutes > 30 ? \"Ongoing\": \"Closed\"\n    }\n} }, params.value);","lang":"painless","position":{"offset":65,"start":40,"end":90},"caused_by":{"type":"illegal_argument_exception","reason":"not all paths return a value for lambda"}}}}],"caused_by":{"type":"null_pointer_exception","reason":"Cannot invoke \"org.elasticsearch.search.aggregations.InternalAggregations.getSerializedSize()\" because \"reducePhase.aggregations\" is null"}}}}}

Apologies for the formatting

BenB196 · January 26, 2021, 7:58pm

Instead of using two if statements, you might want to try:

if (!doc['start_date'].empty && !doc['last_update'].empty)

Your script would look like:

if (!doc['start_date'].empty && !doc['last_update'].empty)
{
    def time_diff = doc['last_update'].value.millis - doc['start_date'].value.millis;
    def minutes = time_diff / 60000;

//  minutes > 30 ? 'Open': 'Closed'
    if ( minutes > 30 )
    {
        return 'Open'
    } else {
        return 'Closed'
    }
}

This topic explains a similar issue: Painless, if two field exists then calculate

Rob_wylde · January 26, 2021, 8:05pm

Ben,

I've tried your code and i get the same results but I also now get ttScope error

TypeError: ttScope is null
    hide https://domain.com/35949/bundles/plugin/kibanaLegacy/kibanaLegacy.plugin.js:1
    hideTooltipBind https://domain.com/35949/bundles/plugin/kibanaLegacy/kibanaLegacy.plugin.js:1
    $digest https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    $apply https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    _callee$ https://domain.com/35949/bundles/plugin/discover/discover.chunk.4.js:1
    l https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:380
    _invoke https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:380
    t https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:380
    create_doc_table_react_asyncGeneratorStep https://domain.com/35949/bundles/plugin/discover/discover.chunk.4.js:1
    _next https://domain.com/35949/bundles/plugin/discover/discover.chunk.4.js:1
    promise callback*create_doc_table_react_asyncGeneratorStep https://domain.com/35949/bundles/plugin/discover/discover.chunk.4.js:1
    _next https://domain.com/35949/bundles/plugin/discover/discover.chunk.4.js:1
    create_doc_table_react_asyncToGenerator https://domain.com/35949/bundles/plugin/discover/discover.chunk.4.js:1
    create_doc_table_react_asyncToGenerator https://domain.com/35949/bundles/plugin/discover/discover.chunk.4.js:1
    injectAngularElement https://domain.com/35949/bundles/plugin/discover/discover.chunk.4.js:1
    convertDirectiveToRenderFn https://domain.com/35949/bundles/plugin/discover/discover.chunk.4.js:1
    DocTableLegacy https://domain.com/35949/bundles/plugin/discover/discover.chunk.4.js:1
    ps https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    El https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    unstable_runWithPriority https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:442
    Hi https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    Sl https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    ol https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    Gi https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    unstable_runWithPriority https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:442
    Hi https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    Gi https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    qi https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    el https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    $l https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    Gl https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    render https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:434
    renderComponent https://domain.com/35949/bundles/plugin/kibanaLegacy/kibanaLegacy.plugin.js:1
    $digest https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    evalAsync https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    completeTask https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    r https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    setTimeout handler*cn/o.defer https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    $evalAsync https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    get https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    d https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    e https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    o https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    promise callback*e https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    h https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    d https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    d https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    $digest https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    $apply https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    callInDigest https://domain.com/35949/bundles/plugin/kibanaLegacy/kibanaLegacy.plugin.js:1
    next https://domain.com/35949/bundles/plugin/kibanaLegacy/kibanaLegacy.plugin.js:1
    __tryOrUnsub https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21
    next https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21
    _next https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21
    next https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21
    debouncedNext https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:480
    oe https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:480
    _execute https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:340
    execute https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:340
    flush https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:340
    setInterval handler*__kbnSharedDeps__</i</t.prototype.requestAsyncId https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:340
    schedule https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:340
    schedule https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:355
    schedule https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:340
    _next https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:480
    next https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21
    notifyNext https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:340
    _next https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21
    next https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:21
    next https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:283
    handleStateUpdate https://domain.com/35949/bundles/plugin/data/data.plugin.js:1
    addFilters https://domain.com/35949/bundles/plugin/data/data.plugin.js:1
    filterQuery https://domain.com/35949/bundles/plugin/discover/discover.chunk.7.js:1
    wrapped https://domain.com/35949/bundles/plugin/kibanaLegacy/kibanaLegacy.plugin.js:1
    $eval https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    $apply https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    wrapped https://domain.com/35949/bundles/plugin/kibanaLegacy/kibanaLegacy.plugin.js:1
    inlineFilter https://domain.com/35949/bundles/plugin/discover/discover.chunk.4.js:1
    fn https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js line 426 > Function:4
    i https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    $eval https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    $apply https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    compile https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:426
    dispatch https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:404
    handle https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:404
    add https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:404
    Le https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:404
    each https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:393
    each https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:393
    Le https://domain.com/35949/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:404
kbn-ui-shared-deps.js:426:78980

Thanks for the suggestion

stu · February 1, 2021, 4:40pm

@Rob_wylde

We need the error from elasticsearch, not kibana, to diagnose this issue.

From what you posted above:

"caused_by":{
  "type":"null_pointer_exception","reason":"Cannot invoke \"org.elasticsearch.search.aggregations.InternalAggregations.getSerializedSize()\" because \"reducePhase.aggregations\" is null"}

Tells me your script is returning null in some cases. You'll need to return a default value if your conditionals do not match. Something like:

if (!doc['start_date'].empty && !doc['last_update'].empty)
{
...
}
return 'Closed'

system · March 1, 2021, 4:40pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.