Painless scripted field values not returning on only two of the values in my docs but all other values even nested values return in kibana 6.4.2

Elastic Stack Kibana
1

Hello I am trying to do some data manipulation on a scripted field in Kibana. I'm having an issue where I can return all other key values in the log except the message and error.message keys. They return 0 results even with data in the fields.

Using Kibana Version: 6.4.2

Here is a example document:

      {
    "_index": "filelogbeat-production-6.4.2-2019.02.09",
    "_type": "doc",
    "_id": "Dje50mgB3uHiSvgnLMBn",
    "_score": 1,
    "_source": {
      "offset": 309376,
      "prospector": {
        "type": "log"
      },
      "source": """C:\inetpub\logs\ct-data-service\error.log""",
      "message": "================================================================================",
      "fileset": {
        "module": "iis",
        "name": "access"
      },
      "error": {
        "message": "Provided Grok expressions do not match field value: [================================================================================]"
      },
      "input": {
        "type": "log"
      },
      "@timestamp": "2019-02-09T14:46:07.646Z",
      "beat": {
        "hostname": "Removed",
        "name": "Removed",
        "version": "6.4.2"
      },
      "host": {
        "name": "Removed"
      }
    }
  },

I can reference

doc['host.name'].value; //these return values
doc['offset'].value;
doc['source'].value;

Here is the scripted field return on the source key.

[
 {
  "_id": "RVW50mgBvqEbiJJnH8Vp",
  "ipAddress": [
   "C:\\inetpub\\logs\\LogFiles\\W3SVC5\\u_extend192.log"
  ]
 },

I've tried scripting if the field exist with:

if (doc.containsKey('message')) {
    return doc['message'].value;
}

or

if (doc.containsKey('error')) {
    return doc['error'].value;
}

or

if (doc.containsKey('error.message')) {
    return doc['error.message'].value;
}

returns

First 10 results

[]

This still returns no results.
With iis logs message field isn't always present so I ran an es query in dev tools:

GET /filelogbeat-production-6.4.2-2019.02.09/_search
{
  "query": {
      "exists": {
    "field": "message"
  }
  }
}

This is how I got the sample doc. I assume using the .contains is a way to only run the scripted field when the message field is present.

Even though there is data I can't seem to access the message value with scripted fields it always returns null so does the error.message field.

I'm not sure why this is returning no values.

Also, can you do try catch blocks in painless language? I can't seem to get try catches to work.

Also in the preview results editor I can add the additional fields for message error.message and @timestamp from the drop down and only timestamp gets added to the output value.

I use doc['source'].value in the scripted field box and when previewing I added message and error.message fields from the drop down of additional fields.

This is what returns, there is no error or message results but I know for sure because of my query above that data exist and I can obviously see it in my dashboard. Its also under the string type in the drop down.

[
 {
  "_id": "RVW50mgBvqEbiJJnH8Vp",
  "@timestamp": "2019-02-05T04:24:05.000Z",
  "ipAddress": [
   "C:\\inetpub\\logs\\LogFiles\\W3SVC5\\u_extend192.log"
  ]
 },
2

if I do the following:

def path = doc['source'].value;
if (path != null) {
    return "value " + path;
}else{
   return "no value";
}

I get:

[
 {
  "_id": "RVW50mgBvqEbiJJnH8Vp",
  "ipAddress": [
   "value C:\\inetpub\\logs\\LogFiles\\W3SVC5\\u_extend192.log"
  ]
 },

if I do:

def path = doc['message'].value;
if (path != null) {
    return "value " + path;
}else{
   return "no value";
}

I get:

First 10 results

[]
3

if I do this no values are returned

if (doc.containsKey('message')) {
def path = doc['message'].value;
if (path != null) {
    return "value " + path;
}else{
   return "no value";
}
}else{
return "no key";
}

if I do

if (doc.containsKey('message')) {
    return "values";
}

it returns values so seems like the message field causes some error that isn't displayed in the output or something I have no idea.

4

I tried so many different methods in the provided online scripting console for doing scripted fields with the preview but wasn't getting any errors. Then I looked up how to do the elasticsearch query and run inline painless scripting to see if it had any issues accessing the field.

GET /filelogbeat-production-6.4.2-2019.02.11/_search {
		"query": {
			"bool": {
				"must": [{
					"exists": {
						"field": "message"
					}
				}],
				"filter": [{
					"script": {
						"script": {
							"inline": "doc['message'].value"
						}
					}
				}]
			}
		}
	}​

running this query in the dev tools​

"type": "search_phase_execution_exception", "reason": "all shards failed", "phase": "query", "grouped": true, "failed_shards": [{
	"shard": 0,
	"index": "filelogbeat-production-6.4.2-2019.02.11",
	"node": "hQksfgxQROiRSoSnDW3liQ",
	"reason": {
		"type": "script_exception",
		"reason": "runtime error",
		"script_stack": ["org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:670)", "org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:115)", "org.elasticsearch.index.query.QueryShardContext.lambda$lookup$0(QueryShardContext.java:280)", "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:88)", "org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:85)", "java.security.AccessController.doPrivileged(Native Method)", "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:85)", "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)", "doc['message'].value", "    ^---- HERE"],
		"script": "doc['message'].value",
		"lang": "painless",
		"caused_by": {
			"type": "illegal_argument_exception",
			"reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [message] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
		}
	}
}]
}, "status": 500
}​

for the filebeat created mapping for the index all message fields have the following mapping:

		"message": {
			"type": "text",
			"norms": false
		},

Error it gives:

"caused_by": {
	"type": "illegal_argument_exception",
	"reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [message] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
}

Could this be causing my issue in the painless preview field but not showing the exception?

Is there an issue using text fields with scripted fields?

5

You need to use doc[mesage].empty as a check to see that that the field exists in the document.

6 
if(doc['message'].empty){
 return "empty"
}else{
return doc['message'].value
}

This returns 0 values but there are values in the db.

This means the empty didn't trigger because it would've returned "empty"

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.