# Scripted fields - doc\[message.keyword\].value reading as empty - field is not empty

**URL:** <https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953>\
**Category:** Kibana\
**Created:** [August 10, 2021, 5:29pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953 "2021-08-10T17:29:53Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![acastravet](https://avatars.discourse-cdn.com/v4/letter/a/b5e925/32.png) [@acastravet](https://discuss.elastic.co/u/acastravet)\
**Post date:** [August 10, 2021, 5:29pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/1 "2021-08-10T17:29:53Z")

</div>

Hello everyone,

I have a field that holds all the log information (multiple fields inside it).  
Field looks like this: logging.level="some info", @timestamp="some info", message="some info", trace\_id="some info"

I am trying, using scripted fields, to split the string into appropriate fields, but I encounter an error: A document doesn't have a value for a field! Use doc[].size()==0 to check if a document is missing a field!

The thing is that the field is not empty at all.

I have fixed it using the following: if (doc['message.keyword'].size() == 0) return "";

The error is gone but the newly created scripted field is showing as empty for obvious reasons.

Its seeing all fields as empty for some reasons.

This is my script without the line mentioned above.

```auto
String oldMessage = doc['message.keyword'].value;
def newScriptedField = "";
def index1 = oldMessage.indexOf('logging.level=');
def index2 = oldMessage.indexOf(', @');

if(oldMessage.contains("logging.level=")) {
    newScriptedField = oldMessage.substring(index1 + 14, index2);
    return newScriptedField;
} else {
    return '';
}

```

---

<div class="post-metadata">

**Author:** ![mattkime](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattkime/32/43522_2.png) [@mattkime](https://discuss.elastic.co/u/mattkime)\
**Post date:** [August 10, 2021, 8:47pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/2 "2021-08-10T20:47:41Z")

</div>

Perhaps this is helpful - [Kibana Painless scripted field checks if field exists or is empty and returns default, otherwise value · GitHub](https://gist.github.com/jclosure/8e103dee2f7e9491845a2c0bb64c6b7a)

---

<div class="post-metadata">

**Author:** ![acastravet](https://avatars.discourse-cdn.com/v4/letter/a/b5e925/32.png) [@acastravet](https://discuss.elastic.co/u/acastravet)\
**Post date:** [August 11, 2021, 6:42am UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/3 "2021-08-11T06:42:29Z")

</div>

Now I`m getting the following error:

```auto
   "reason": {
    "type": "script_exception",
    "reason": "compile error",
    "script_stack": [
     "if(doc['message.keyword'] ...",
     "^---- HERE"
    ],
    "script": "if(doc['message.keyword'].size() == 0) return \"\";\n\nif(!doc.containsKey('message') || doc['message'].empty) {\n return 'N/A'\n} else {\n return doc['message'].value\n}\n\ndef oldMessage = doc['message.keyword'].value;\ndef newScriptedField = \"\"; \n//def oldMessage = params['_source']['message'];\ndef index1 = oldMessage.indexOf('logging.level=');\ndef index2 = oldMessage.indexOf(', @');\n\nif(oldMessage.contains(\"logging.level=\")) {\n newScriptedField = oldMessage.substring(index1 + 14, index2);\n return newScriptedField;\n} else {\n return '';\n}",
    "lang": "painless",
    "position": {
     "offset": 0,
     "start": 0,
     "end": 25
    },
    "caused_by": {
     "type": "illegal_argument_exception",
     "reason": "Unreachable statement."
    }
   }
  }
 ]
}

```

---

<div class="post-metadata">

**Author:** ![mattkime](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattkime/32/43522_2.png) [@mattkime](https://discuss.elastic.co/u/mattkime)\
**Post date:** [August 11, 2021, 4:46pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/4 "2021-08-11T16:46:42Z")

</div>

Try removing `if(doc['message.keyword'].size() == 0) return ` from your script.

---

<div class="post-metadata">

**Author:** ![acastravet](https://avatars.discourse-cdn.com/v4/letter/a/b5e925/32.png) [@acastravet](https://discuss.elastic.co/u/acastravet)\
**Post date:** [August 11, 2021, 5:01pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/5 "2021-08-11T17:01:22Z")

</div>

I've tried, I get the exact same thing

---

<div class="post-metadata">

**Author:** ![mattkime](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattkime/32/43522_2.png) [@mattkime](https://discuss.elastic.co/u/mattkime)\
**Post date:** [August 11, 2021, 5:02pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/6 "2021-08-11T17:02:38Z")

</div>

How are you using the script?

What version of the Elastic stack are you using?

---

<div class="post-metadata">

**Author:** ![acastravet](https://avatars.discourse-cdn.com/v4/letter/a/b5e925/32.png) [@acastravet](https://discuss.elastic.co/u/acastravet)\
**Post date:** [August 11, 2021, 5:06pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/7 "2021-08-11T17:06:26Z")

</div>

Buy adding a scripted field in the index pattern configuration. The latest version.

---

<div class="post-metadata">

**Author:** ![mattkime](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattkime/32/43522_2.png) [@mattkime](https://discuss.elastic.co/u/mattkime)\
**Post date:** [August 11, 2021, 5:10pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/8 "2021-08-11T17:10:16Z")

</div>

I'd recommend trying a runtime field instead, since scripted fields are deprecated - [Manage index pattern data fields | Kibana Guide [master] | Elastic](https://www.elastic.co/guide/en/kibana/master/managing-index-patterns.html)

I need the full script and error to help.

---

<div class="post-metadata">

**Author:** ![acastravet](https://avatars.discourse-cdn.com/v4/letter/a/b5e925/32.png) [@acastravet](https://discuss.elastic.co/u/acastravet)\
**Post date:** [August 11, 2021, 5:32pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/9 "2021-08-11T17:32:02Z")

</div>

What I've shown you above is the full script and the full error, pasting them blow again:

```auto
//That is the line without which I'm getting an error saying that: I need this line for the script to work.
//With it, it sees the message field as empty and the new scripted field is empty as a result.
//Changing the line in any way will result in the same error mentioned above.
//Found some other discussions on elastic, that it's related to a bug from version 6.3, that wasn't fixed (not sure if it's related).

if(doc['message.keyword'].size() == 0) return "";

//that is the line you showed me to add, and didn't help
if(!doc.containsKey('message') || doc['message'].empty) {
return 'N/A' } else {
return doc['message'].value 
}

def oldMessage = doc['message.keyword'].value;
def newScriptedField = ""; 
def oldMessage = params['_source']['message'];
def index1 = oldMessage.indexOf('logging.level=');
def index2 = oldMessage.indexOf(', @');

if(oldMessage.contains("logging.level=")) {
newScriptedField = oldMessage.substring(index1 + 14, index2);
return newScriptedField;
} else {
return '';
}

```

And that is the error (other information that is not included is not related to the script, is just sensitive information from logs, which I deleted, all important stuff is below):

```auto
  "reason": {
    "type": "script_exception",
    "reason": "compile error",
    "script_stack": [
     "if(doc['message.keyword'] ...",
     "^---- HERE"
    ],
    "script": "if(doc['message.keyword'].size() == 0) return \"\";\n\nif(!doc.containsKey('message') || doc['message'].empty) {\n return 'N/A'\n} else {\n return doc['message'].value\n}\n\ndef oldMessage = doc['message.keyword'].value;\ndef newScriptedField = \"\"; \n//def oldMessage = params['_source']['message'];\ndef index1 = oldMessage.indexOf('logging.level=');\ndef index2 = oldMessage.indexOf(', @');\n\nif(oldMessage.contains(\"logging.level=\")) {\n newScriptedField = oldMessage.substring(index1 + 14, index2);\n return newScriptedField;\n} else {\n return '';\n}",
    "lang": "painless",
    "position": {
     "offset": 0,
     "start": 0,
     "end": 25
    },
    "caused_by": {
     "type": "illegal_argument_exception",
     "reason": "Unreachable statement."
    }
   }
  }
 ]
}

```

---

<div class="post-metadata">

**Author:** ![acastravet](https://avatars.discourse-cdn.com/v4/letter/a/b5e925/32.png) [@acastravet](https://discuss.elastic.co/u/acastravet)\
**Post date:** [August 11, 2021, 5:35pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/10 "2021-08-11T17:35:01Z")

</div>

I have tried using `params['_source']['message']` instead of `doc['message.keyword'].value` and it worked, but it wasn't filtering in Discover and couldn't see the logs as expected

---

<div class="post-metadata">

**Author:** ![acastravet](https://avatars.discourse-cdn.com/v4/letter/a/b5e925/32.png) [@acastravet](https://discuss.elastic.co/u/acastravet)\
**Post date:** [August 11, 2021, 5:43pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/11 "2021-08-11T17:43:58Z")

</div>

Do you think it would be possible to do the splitting of the string inside the .yml configuration for the filebeat? And adding the new fields over there? Cannot find the way to do that, do not now if its really possible. Looking for every way to do that

---

<div class="post-metadata">

**Author:** ![mattkime](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattkime/32/43522_2.png) [@mattkime](https://discuss.elastic.co/u/mattkime)\
**Post date:** [August 12, 2021, 3:10am UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/12 "2021-08-12T03:10:14Z")

</div>

> [@acastravet](#):
>
> tried using `params['_source']['message']` instead of `doc['message.keyword'].value` and it worked, but it wasn't filtering in Discover and couldn't see the logs as expected

Did you remove the filter and recreate it? There were some funny tricks to making filters work with scripted fields.

---

<div class="post-metadata">

**Author:** ![acastravet](https://avatars.discourse-cdn.com/v4/letter/a/b5e925/32.png) [@acastravet](https://discuss.elastic.co/u/acastravet)\
**Post date:** [August 12, 2021, 7:18am UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/13 "2021-08-12T07:18:22Z")

</div>

Yes, still didn't work

---

<div class="post-metadata">

**Author:** ![mattkime](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattkime/32/43522_2.png) [@mattkime](https://discuss.elastic.co/u/mattkime)\
**Post date:** [August 12, 2021, 3:06pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/14 "2021-08-12T15:06:03Z")

</div>

I think logstash would be your best option for parsing logs on ingest. Parsing on ingest would make for faster queries.

---

<div class="post-metadata">

**Author:** ![mattkime](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattkime/32/43522_2.png) [@mattkime](https://discuss.elastic.co/u/mattkime)\
**Post date:** [August 12, 2021, 3:09pm UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/15 "2021-08-12T15:09:10Z")

</div>

At any rate, we need to get past syntax errors. Remind me of your current script and the error you're getting. If its the script you posted previously, remove the line I mentioned and post the error. It should be a different error.

Additionally, we can look at the output in Discover and verify that we're seeing the correct values and worry about filtering once they're correct.

---

<div class="post-metadata">

**Author:** ![acastravet](https://avatars.discourse-cdn.com/v4/letter/a/b5e925/32.png) [@acastravet](https://discuss.elastic.co/u/acastravet)\
**Post date:** [August 16, 2021, 7:05am UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/16 "2021-08-16T07:05:48Z")

</div>

Ok, so, with the current script:

```auto
def newScriptedField = ""; 
def oldMessage = params['_source']['message'];
def index1 = oldMessage.indexOf('logging.level=');
def index2 = oldMessage.indexOf(', @');

if(oldMessage.contains("logging.level=")) {
    newScriptedField = oldMessage.substring(index1 + 14, index2);
    return newScriptedField;
} else {
    return '';
}

```

Everything works fine, here is the newly created script:

 ![Screenshot 2021-08-16 100232](https://us1.discourse-cdn.com/elastic/original/3X/e/8/e8510367a661c8f5fe1a6a550571097c7a3f9c21.jpeg)

But, when I` trying to filter or search for the newly created field, with ERROR for example, I`m getting the current error:

```auto
{
  "took": 6160,
  "timed_out": false,
  "_shards": {
    "total": 6,
    "successful": 5,
    "skipped": 5,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": "filebeat- *****",
        "node": " *******",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "oldMessage = params['_source']['message'];\r\ndef ",
            " ^---- HERE"
          ],
          "script": "boolean compare(Supplier s, def v) {return s.get() == v;}compare(() -> { def newScriptedField = \"\"; \r\ndef oldMessage = params['_source']['message'];\r\ndef index1 = oldMessage.indexOf('logging.level=');\r\ndef index2 = oldMessage.indexOf(', @');\r\n\r\nif(oldMessage.contains(\"logging.level=\")) {\r\n newScriptedField = oldMessage.substring(index1 + 14, index2);\r\n return newScriptedField;\r\n} else {\r\n return '';\r\n} }, params.value);",
          "lang": "painless",
          "position": {
            "offset": 125,
            "start": 106,
            "end": 154
          },
          "caused_by": {
            "type": "null_pointer_exception",
            "reason": "Cannot invoke \"Object.getClass()\" because \"callArgs[0]\" is null"
          }
        }
      }
    ]
  },
  "hits": {
    "total": 0,
    "max_score": 0,
    "hits": []
  }
}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 13, 2021, 7:06am UTC](https://discuss.elastic.co/t/scripted-fields-doc-message-keyword-value-reading-as-empty-field-is-not-empty/280953/17 "2021-09-13T07:06:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.