# Kibana - scripted fileds - painles
**URL:** https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483
**Category:** Kibana
**Created:** [April 18, 2018, 8:33am UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483 "2018-04-18T08:33:20Z")
**Posts on this page:** 12
**Page:** 1
### Author:  [@bayerb](https://discuss.elastic.co/u/bayerb)
#### Post date: [April 18, 2018, 8:33am UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483/1 "2018-04-18T08:33:20Z")
When I insert the scripted field:
```auto
if (doc['system.process.cmdline'].value =~ /.*java.*-DJENKINS_HOME.*/) {
def m = /.*java.*-DJENKINS_HOME.*instance_(.*?)\//.matcher(doc['system.process.cmdline'].value);
if (m.matches()) {
return m.group(1);
}
else {
return "unknown";
}
}
} else {
return "not jenkins";
}
```
In discover I get the warning message
> Courier Fetch: 5 of 65 shards failed.
and no entries appear(0 hits).
In elastics log the following warning entry appears:
> [2018-04-18T08:30:12,951][WARN][o.e.d.a.a.i.t.p.PutIndexTemplateRequest] Deprecated field [template] used, replaced by [index\_patterns]
After I remove this scripted field, everzthing is get back to normal. What is wrong\<
Thanks!
---
### Author:  [@timroes](https://discuss.elastic.co/u/timroes)
#### Post date: [April 18, 2018, 8:39am UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483/2 "2018-04-18T08:39:08Z")
That script is invalid, which usually will cause a shard failure.
There is at least one curly bracket too much (I changed syntax styling above so you can more easily spot the additional closing curly bracket before the last `else`).
Also As long as you're field can contain `null` or not be present in some documents, you need to make sure you check for `null` on the `.value` before trying to regex compare it.
Cheers,
Tim
---
### Author:  [@bayerb](https://discuss.elastic.co/u/bayerb)
#### Post date: [April 18, 2018, 9:12am UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483/3 "2018-04-18T09:12:45Z")
Thanks for the suggestions. This was my next trial:
```
def cmdline = doc['system.process.cmdline'].value;
if (cmdline == null) {
return "not jenkins";
}
if (cmdline =~ /.*java.*-DJENKINS_HOME.*/) {
def m = /.*java.*-DJENKINS_HOME.*instance_(.*?)\//.matcher(doc['system.process.cmdline'].value);
if (m.matches()) {
return m.group(1);
}
else {
return "unknown";
}
}
else {
return "not jenkins";
}
```
Also without success. What is wrong here?
---
### Author:  [@timroes](https://discuss.elastic.co/u/timroes)
#### Post date: [April 18, 2018, 9:19am UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483/4 "2018-04-18T09:19:01Z")
Could you please provide more information on the error ES is giving you. This should be possible by one of the following ways:
1. Use the [Elasticsearch API](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/search-request-script-fields.html#search-request-script-fields) to query your index directly and check the output.
2. You can use the Browsers Dev Tool (by pressing F12) to inspect the request made by Discover (should be an `_msearch` request) and copy the full response of that.
Cheers,
Tim
---
### Author:  [@bayerb](https://discuss.elastic.co/u/bayerb)
#### Post date: [April 18, 2018, 9:35am UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483/5 "2018-04-18T09:35:20Z")
This is the request:
```
GET /_search
{
"query" : {
"match_all": {}
},
"script_fields" : {
"test1" : {
"script" : {
"lang": "painless",
"source": "
def cmdline = doc['system.process.cmdline'].value;
if (cmdline == null) {
return \"not jenkins\";
}
if (cmdline =~ /.*java.*-DJENKINS_HOME.*/) {
def m = /.*java.*-DJENKINS_HOME.*instance_(.*?)\//.matcher(doc['system.process.cmdline'].value);
if (m.matches()) {
return m.group(1);
}
else {
return \"unknown\";
}
}
else {
return \"not jenkins\";
}
"
}
}
}
}
```
this is the response:
```
{
"error": {
"root_cause": [
{
"type": "parsing_exception",
"reason": "[script] failed to parse field [source]",
"line": 9,
"col": 27
}
],
"type": "parsing_exception",
"reason": "[script] failed to parse field [source]",
"line": 9,
"col": 27,
"caused_by": {
"type": "unchecked_i_o_exception",
"reason": "com.fasterxml.jackson.core.JsonParseException: Illegal unquoted character ((CTRL-CHAR, code 13)): has to be escaped using backslash to be included in string value\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@667ef57e; line: 9, column: 29]",
"caused_by": {
"type": "json_parse_exception",
"reason": "Illegal unquoted character ((CTRL-CHAR, code 13)): has to be escaped using backslash to be included in string value\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@667ef57e; line: 9, column: 29]"
}
}
},
"status": 400
}
```
---
### Author:  [@timroes](https://discuss.elastic.co/u/timroes)
#### Post date: [April 18, 2018, 9:39am UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483/6 "2018-04-18T09:39:00Z")
To use ultiline strings in Kibanas Dev Tools you can use triple quotes (`"""`) and also remove the escaped quotes in this case, since otherwise JSON doesn't support that syntax. Also make sure you are only querying the index pattern, that you have selected in Discover:
```auto
GET //_search
{
"query": {
"match_all": {}
},
"script_fields": {
"test1": {
"script": {
"lang": "painless",
"source": """
def cmdline = doc['system.process.cmdline'].value;
if (cmdline == null) {
return "not jenkins";
}
if (cmdline =~ /.*java.*-DJENKINS_HOME.*/) {
def m = /.*java.*-DJENKINS_HOME.*instance_(.*?)\//.matcher(doc['system.process.cmdline'].value);
if (m.matches()) {
return m.group(1);
}
else {
return "unknown";
}
}
else {
return "not jenkins";
}
"""
}
}
}
}
```
---
### Author:  [@bayerb](https://discuss.elastic.co/u/bayerb)
#### Post date: [April 18, 2018, 12:36pm UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483/7 "2018-04-18T12:36:00Z")
Request:
```
GET metricbeat-*/_search
{
"query" : {
"match_all": {}
},
"script_fields" : {
"test1" : {
"script" : {
"lang": "painless",
"source": """
def cmdline = doc['system.process.cmdline'].value;
if (cmdline == null) {
return \"not jenkins\";
}
if (cmdline =~ /.*java.*-DJENKINS_HOME.*/) {
def m = /.*java.*-DJENKINS_HOME.*instance_(.*?)//.matcher(doc['system.process.cmdline'].value);
if (m.matches()) {
return m.group(1);
}
else {
return \"unknown\";
}
}
else {
return \"not jenkins\";
}
"""
}
}
}
}
```
response:
```
{
"error": {
"root_cause": [
{
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"... ine == null) {\r\n return \\\"not jenkins\\\";\r\n}\r\nif ( ...",
" ^---- HERE"
],
"script": " def cmdline = doc['system.process.cmdline'].value;\r\nif (cmdline == null) {\r\n return \\\"not jenkins\\\";\r\n}\r\nif (cmdline =~ /.*java.*-DJENKINS_HOME.*/) { \r\n def m = /.*java.*-DJENKINS_HOME.*instance_(.*?)//.matcher(doc['system.process.cmdline'].value);\r\n if (m.matches()) {\r\n return m.group(1);\r\n }\r\n else {\r\n return \\\"unknown\\\";\r\n }\r\n }\r\n else {\r\n return \\\"not jenkins\\\";\r\n }",
"lang": "painless"
}
],
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": 0,
"index": "metricbeat-6.2.3-2018.04.04",
"node": "6I0aP6ApSPeKmSUfgeFMqA",
"reason": {
"type": "script_exception",
"reason": "compile error",
"script_stack": [
"... ine == null) {\r\n return \\\"not jenkins\\\";\r\n}\r\nif ( ...",
" ^---- HERE"
],
"script": " def cmdline = doc['system.process.cmdline'].value;\r\nif (cmdline == null) {\r\n return \\\"not jenkins\\\";\r\n}\r\nif (cmdline =~ /.*java.*-DJENKINS_HOME.*/) { \r\n def m = /.*java.*-DJENKINS_HOME.*instance_(.*?)//.matcher(doc['system.process.cmdline'].value);\r\n if (m.matches()) {\r\n return m.group(1);\r\n }\r\n else {\r\n return \\\"unknown\\\";\r\n }\r\n }\r\n else {\r\n return \\\"not jenkins\\\";\r\n }",
"lang": "painless",
"caused_by": {
"type": "illegal_argument_exception",
"reason": "unexpected character [\\].",
"caused_by": {
"type": "lexer_no_viable_alt_exception",
"reason": null
}
}
}
}
]
},
"status": 500
}
```
---
### Author:  [@timroes](https://discuss.elastic.co/u/timroes)
#### Post date: [April 18, 2018, 1:24pm UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483/8 "2018-04-18T13:24:04Z")
You should remove the escaped quotes when using triple quotes. Try copy over my request.
---
### Author:  [@bayerb](https://discuss.elastic.co/u/bayerb)
#### Post date: [April 18, 2018, 1:54pm UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483/9 "2018-04-18T13:54:18Z")
Your request was fine. Based on its response it turned out, that I had to set script.painless.regex.enabled to true. Now The field appears in elasticsearch, but it is empty.
A system.process.cmdline looks like this:
`/devbase/jdk1.8.0/bin/java -DJENKINS_HOME=/devbase/jenkins/instance_od/conf -Djava.io.tmpdir=/devbase/jenkins/instance_od/tmp -Xmx3g -jar /devbase/jenkins/bin/jenkins-lts_2.73.3.war --httpPort=8110 --ajp13Port=-1 --prefix=/od --sessionTimeout=480 --httpListenAddress=127.0.0.1`
In this case for example I would like to see the value "od".
---
### Author:  [@timroes](https://discuss.elastic.co/u/timroes)
#### Post date: [April 18, 2018, 4:33pm UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483/10 "2018-04-18T16:33:15Z")
The problem for you most likely will be `ignore_above`. If you haven't changed the default mapping, string fields will have a setting of 256 for `ignore_above`, which means all values above that length won't be indexed. You can still see them in the document \_source (or in Kibana's Discover view), but they won't appear in the so called inverse index, that is used to build aggregations and search for data.
Also `doc` in painless scripts access only these doc values, and not the original source values, meaning that string is most likely cut off due to a wrong (the default) `ignore_above` setting in your index. If you need to be able to access that long values, please make sure your mapping doesn't have `ignore_above` in there.
Could you please paste the mapping for the metricbeat index, by running `GET /metricbeat-*/_mapping` in the dev tools?
---
### Author:  [@bayerb](https://discuss.elastic.co/u/bayerb)
#### Post date: [April 19, 2018, 7:09am UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483/11 "2018-04-19T07:09:36Z")
The full mapping is too long, but here is the relevant part:
```
"cmdline": {
"type": "keyword",
"ignore_above": 1024
},
```
I removed this setting with
```
PUT metricbeat-*/_mapping/doc
{
"properties": {
"cmdline": {
"type": "keyword"
}
}
}
```
But unfortunately the field is still empty. Any idea?
Thanks!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 17, 2018, 7:09am UTC](https://discuss.elastic.co/t/kibana-scripted-fileds-painles/128483/12 "2018-05-17T07:09:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.