Scripted fields - doc[message.keyword].value reading as empty - field is not empty

Elastic Stack Kibana
1

Hello everyone,

I have a field that holds all the log information (multiple fields inside it).
Field looks like this: logging.level="some info", @timestamp="some info", message="some info", trace_id="some info"

I am trying, using scripted fields, to split the string into appropriate fields, but I encounter an error: A document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field!

The thing is that the field is not empty at all.

I have fixed it using the following: if (doc['message.keyword'].size() == 0) return "";

The error is gone but the newly created scripted field is showing as empty for obvious reasons.

Its seeing all fields as empty for some reasons.

This is my script without the line mentioned above.

String oldMessage = doc['message.keyword'].value;
def newScriptedField = "";
def index1 = oldMessage.indexOf('logging.level=');
def index2 = oldMessage.indexOf(', @');

if(oldMessage.contains("logging.level=")) {
    newScriptedField = oldMessage.substring(index1 + 14, index2);
    return newScriptedField;
} else {
    return '';
}
1 Like
2

Perhaps this is helpful - Kibana Painless scripted field checks if field exists or is empty and returns default, otherwise value · GitHub

3

Now I`m getting the following error:

   "reason": {
    "type": "script_exception",
    "reason": "compile error",
    "script_stack": [
     "if(doc['message.keyword'] ...",
     "^---- HERE"
    ],
    "script": "if(doc['message.keyword'].size() == 0) return \"\";\n\nif(!doc.containsKey('message') || doc['message'].empty) {\n    return 'N/A'\n} else {\n    return doc['message'].value\n}\n\ndef oldMessage = doc['message.keyword'].value;\ndef newScriptedField = \"\"; \n//def oldMessage = params['_source']['message'];\ndef index1 = oldMessage.indexOf('logging.level=');\ndef index2 = oldMessage.indexOf(', @');\n\nif(oldMessage.contains(\"logging.level=\")) {\n    newScriptedField = oldMessage.substring(index1 + 14, index2);\n    return newScriptedField;\n} else {\n    return '';\n}",
    "lang": "painless",
    "position": {
     "offset": 0,
     "start": 0,
     "end": 25
    },
    "caused_by": {
     "type": "illegal_argument_exception",
     "reason": "Unreachable statement."
    }
   }
  }
 ]
}
4

Try removing if(doc['message.keyword'].size() == 0) return from your script.

5

I've tried, I get the exact same thing

6

How are you using the script?

What version of the Elastic stack are you using?

7

Buy adding a scripted field in the index pattern configuration. The latest version.

8

I'd recommend trying a runtime field instead, since scripted fields are deprecated - Manage index pattern data fields | Kibana Guide [master] | Elastic

I need the full script and error to help.

9

What I've shown you above is the full script and the full error, pasting them blow again:

//That is the line without which I'm getting an error saying that: I need this line for the script to work.
//With it, it sees the message field as empty and the new scripted field is empty as a result.
//Changing the line in any way will result in the same error mentioned above.
//Found some other discussions on elastic, that it's related to a bug from version 6.3, that wasn't fixed (not sure if it's related).

if(doc['message.keyword'].size() == 0) return "";

//that is the line you showed me to add, and didn't help
if(!doc.containsKey('message') || doc['message'].empty) {
return 'N/A' } else {
return doc['message'].value 
}

def oldMessage = doc['message.keyword'].value;
def newScriptedField = ""; 
def oldMessage = params['_source']['message'];
def index1 = oldMessage.indexOf('logging.level=');
def index2 = oldMessage.indexOf(', @');

if(oldMessage.contains("logging.level=")) {
newScriptedField = oldMessage.substring(index1 + 14, index2);
return newScriptedField;
} else {
return '';
}

And that is the error (other information that is not included is not related to the script, is just sensitive information from logs, which I deleted, all important stuff is below):

  "reason": {
    "type": "script_exception",
    "reason": "compile error",
    "script_stack": [
     "if(doc['message.keyword'] ...",
     "^---- HERE"
    ],
    "script": "if(doc['message.keyword'].size() == 0) return \"\";\n\nif(!doc.containsKey('message') || doc['message'].empty) {\n    return 'N/A'\n} else {\n    return doc['message'].value\n}\n\ndef oldMessage = doc['message.keyword'].value;\ndef newScriptedField = \"\"; \n//def oldMessage = params['_source']['message'];\ndef index1 = oldMessage.indexOf('logging.level=');\ndef index2 = oldMessage.indexOf(', @');\n\nif(oldMessage.contains(\"logging.level=\")) {\n    newScriptedField = oldMessage.substring(index1 + 14, index2);\n    return newScriptedField;\n} else {\n    return '';\n}",
    "lang": "painless",
    "position": {
     "offset": 0,
     "start": 0,
     "end": 25
    },
    "caused_by": {
     "type": "illegal_argument_exception",
     "reason": "Unreachable statement."
    }
   }
  }
 ]
}
10

I have tried using params['_source']['message'] instead of doc['message.keyword'].value and it worked, but it wasn't filtering in Discover and couldn't see the logs as expected

1 Like
11

Do you think it would be possible to do the splitting of the string inside the .yml configuration for the filebeat? And adding the new fields over there? Cannot find the way to do that, do not now if its really possible. Looking for every way to do that

12

Did you remove the filter and recreate it? There were some funny tricks to making filters work with scripted fields.

13

Yes, still didn't work

14

I think logstash would be your best option for parsing logs on ingest. Parsing on ingest would make for faster queries.

15

At any rate, we need to get past syntax errors. Remind me of your current script and the error you're getting. If its the script you posted previously, remove the line I mentioned and post the error. It should be a different error.

Additionally, we can look at the output in Discover and verify that we're seeing the correct values and worry about filtering once they're correct.

16

Ok, so, with the current script:

def newScriptedField = ""; 
def oldMessage = params['_source']['message'];
def index1 = oldMessage.indexOf('logging.level=');
def index2 = oldMessage.indexOf(', @');

if(oldMessage.contains("logging.level=")) {
    newScriptedField = oldMessage.substring(index1 + 14, index2);
    return newScriptedField;
} else {
    return '';
}

Everything works fine, here is the newly created script:

Screenshot 2021-08-16 100232
Screenshot 2021-08-16 1002321538×89 21.8 KB

But, when I trying to filter or search for the newly created field, with ERROR for example, Im getting the current error:

{
  "took": 6160,
  "timed_out": false,
  "_shards": {
    "total": 6,
    "successful": 5,
    "skipped": 5,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "filebeat-*****",
        "node": "*******",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "oldMessage = params['_source']['message'];\r\ndef ",
            "                   ^---- HERE"
          ],
          "script": "boolean compare(Supplier s, def v) {return s.get() == v;}compare(() -> { def newScriptedField = \"\"; \r\ndef oldMessage = params['_source']['message'];\r\ndef index1 = oldMessage.indexOf('logging.level=');\r\ndef index2 = oldMessage.indexOf(', @');\r\n\r\nif(oldMessage.contains(\"logging.level=\")) {\r\n    newScriptedField = oldMessage.substring(index1 + 14, index2);\r\n    return newScriptedField;\r\n} else {\r\n    return '';\r\n} }, params.value);",
          "lang": "painless",
          "position": {
            "offset": 125,
            "start": 106,
            "end": 154
          },
          "caused_by": {
            "type": "null_pointer_exception",
            "reason": "Cannot invoke \"Object.getClass()\" because \"callArgs[0]\" is null"
          }
        }
      }
    ]
  },
  "hits": {
    "total": 0,
    "max_score": 0,
    "hits": []
  }
}
17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.