Scripted Fields - if field has value conditional

[wendt]

Good morning/afternoon/evening!

I'm on Kibana 6.3.0 and am struggling to implement a solution using a scripted field.

I have an index that has two text fields: body and selftext.

Each document will have one or the other of these fields populated. For the purpose of some dashboards, I want to display these in one field: message.

The general logic I'm trying to follow is:

if body [exists|has value|is not null|length > 1] then message = body
else if selftext [exists|has value|is not null|length > 1] then message = selftext

I've tried using the doc.['fieldname'].empty and doc.containsKey('fieldname') functions so far, but the first ends up causing a "failed to fetch shards" error, and the latter seems to always evaluate to true.

Below is how I used doc.containsKey('fieldname'). With this, if body exists in a document, message will populate accordingly. If selftext exists... message remains empty. If I swap the order they're evaluated in, the results will switch as well. That leads me to believe the first statement is always evaluated as true, or is at least exiting without evaluating the second statement.

if (doc.containsKey('body')){params['_source']['body']}
else if(doc.containsKey('selftext')){params['_source']['selftext']}

Anyways, if anyone has any suggestions, they'd be welcome! If there is more detail I can provide, just ask. I appreciate the time.

[Nathan_Reese]

Give this a try if (doc['body'].value != null) return doc['body'].value; return doc['selftext'].value

If you want to test it out on the documents I used, enter the below commands in Dev Tools application.

I like to build scripts in Dev Tools and then I migrate working scripts over to Index Patterns. In Dev Tools, you can run the script immediate and get feedback on how it works. The last command executes a search with the script and is the one to run when experimenting with scripted fields.

PUT testindex
{
    "settings" : {
        "index" : {
            "number_of_shards" : 1, 
            "number_of_replicas" : 0
        }
    }
}

PUT testindex/_mapping/_doc 
{
  "properties": {
    "body": {
      "type": "keyword"
    },
    "selftext": {
      "type": "keyword"
    }
  }
}

PUT testindex/_doc/1
{
    "body" : "text from body field"
}

PUT testindex/_doc/2
{
    "selftext" : "text from selftext"
}

GET testindex/_search
{
  "query": {
    "match_all": {}
  },
  "script_fields" : {
    "myScriptedField" : {
      "script" : {
        "lang": "painless",
        "source": "if (doc['body'].value != null) return doc['body'].value; return doc['selftext'].value"
      }
    }
  }
}

[wendt]

Thank you!

First - I hadn't tried testing scripts in dev tools before. That's extremely helpful.

Secondly - the solution works perfectly when I follow your steps, but fails when I try it on the live index. I think this stems from a difference in how the fields are mapped. My mapping for body and selftext both look like the below, with the type specified as "text" as opposed to "keyword".

"body": {
  "type": "text",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 256
    }
  }
}

I do see body.keyword and selftext.keyword in the field list, but they aren't populated with anything in my view in discover.

If I run the the query you provided against body.keyword and selftext.keyword in the live index, it returns a string in the scripted field ~4/10 times. I would expect to see a value returned every time just based off the nature of the data.

Would it be best to reindex and specify the type as "keyword", or is there an alternative solution that you know of?

[Nathan_Reese]

The text type indexes data for full text search. That means the value is passed through an analyzer to convert the string into a list of individual terms before being indexed.

The keyword type just indexes the value as a single string and does not allow for full text searching.

How your data needs to be stored is a function of how you intend to use it. Do you need full search? Or do you intend to search by the complete value? text indexes use a lot more resources but provide a much richer search experience.

Here is the example reworked for text type. Read the last paragraph of https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-script-fields.html to get a better understanding of doc vs _source

DELETE testindex

PUT testindex
{
    "settings" : {
        "index" : {
            "number_of_shards" : 1, 
            "number_of_replicas" : 0
        }
    }
}

PUT testindex/_mapping/_doc 
{
  "properties": {
    "body": {
      "type": "text",
      "fielddata": true
    },
    "selftext": {
      "type": "text",
      "fielddata": true
    }
  }
}

PUT testindex/_doc/1
{
    "body" : "text from body field"
}

PUT testindex/_doc/2
{
    "selftext" : "text from selftext"
}

GET testindex/_search
{
  "query": {
    "match_all": {}
  },
  "script_fields" : {
    "myScriptedField" : {
      "script" : {
        "lang": "painless",
        "source": "if (params['_source']['body'] != null) return params['_source']['body']; return params['_source']['selftext']"
      }
    }
  }
}

[wendt]

Thanks Nathan! That did exactly what I needed it to.

Thank you as well for the links to resources. I definitely need to get a better handle on data types and on scripted fields in general.

Have a great day!