I am trying to replicate a field using the scripted field in Kibana (5.0 GA).
However, it causes below error when I search.
Below is my scripted field definition.
"type" is the filed automatically generated by filebeat so it is a string type.
I have set security level to true but made no difference.
script.inline: true
script.stored: true
script.file : true
Do I have any grammer mistake in the painless language?
Using 5.0 in conjunction with filebeat, I was able to setup the same scripted field in your example without any issue.
Let's see what the error is which is causing the results to fail. Unfortunately, in this state, it's not the easiest to access. We can either inspect the request, which in Chrome is available in View > Developer > Developer Tools and on the Network tab.
Or, we can always make the request manually. For example if you have curl available, create a fail called request.json with the following:
{
"query": {
"match_all": {}
},
"script_fields": {
"foo": {
"script": {
"inline": "doc['type'].value",
"lang": "painless"
}
}
}
}
Then from the same directory, run curl -d @request.json "http://localhost:9200/filebeat-*/_search" | python -m json.tool.
If Elasticsearch is running at the different location, you will need to update that.
Another option, is to view the Elasticsearch logs for the error.
Can you also provide me the mappings for the filebeat index?
For example, http://localhost:9200/filebeat-*/_mapping
I do not have a filebeat mapping imported because I import the logs from logstash . All the logs go through logstash in our environment and index using the dynamic template logstash .
Below is my current mapping definition. The mapping definition was too long to post so I reproduced on other machine with same error.
{
"syslog-messages": {
"mappings": {
"messages": {
"properties": {
"@timestamp": {
"type": "date"
},
"@version": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"facility": {
"type": "long"
},
"facility_label": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"hoge": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"host": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"logsource": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"pid": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"priority": {
"type": "long"
},
"program": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"severity": {
"type": "long"
},
"severity_label": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"tags": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"test": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"timestamp": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}I got the result of the query.
[root@BS-POC-Test-Env y-watanabe]# curl -d @request.json "http://localhost:9200/syslog-messages/_search" | python -m json.tool
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1295 100 1130 100 165 95957 14011 --:--:-- --:--:-- --:--:-- 100k
{
"_shards": {
"failed": 1,
"failures": [
{
"index": "syslog-messages",
"node": "mnn9dEZNSVi9epNPq4BqAw",
"reason": {
"caused_by": {
"reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [host] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory.",
"type": "illegal_argument_exception"
},
"lang": "painless",
"reason": "runtime error",
"script": "doc['host'].value",
"script_stack": [
"org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:335)",
"org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:111)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:87)",
"org.elasticsearch.search.lookup.LeafDocLookup$1.run(LeafDocLookup.java:84)",
"java.security.AccessController.doPrivileged(Native Method)",
"org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:84)",
"doc['host'].value",
" ^---- HERE"
],
"type": "script_exception"
},
"shard": 0
}
],
"successful": 4,
"total": 5
},
"hits": {
"hits": [],
"max_score": 1.0,
"total": 1868
},
"timed_out": false,
"took": 2
}
Curious, have you resolved your issue?
In case you are still working with Painless for scripted fields in Kibana, there is now a blog available with examples and best practices: https://www.elastic.co/blog/using-painless-kibana-scripted-fields
No . Though my current environment is 100% same with my original post but I still get the Courier fetch error.
Below error was reproduced in v5.1.1.
Query was done through below route.
Kibana(v5.1.1) -> tribe node(v5.1.1) -> data node(v5.1.1)
Logs come in to data node through below route.
filebeat(v5.0.2-1) -> logstash(v5.0.2-1) -> datanode(v5.1.1)
I have no xpack in front.
Nothing special with the scripted field definition really.
There was no error in the "Developer Tool" in my Chrome Browser.
Let me turn on the DEBUG log on the tribe and data node side.
Below is the error I get in my tribe node.
[2016-12-14T10:55:01,361][DEBUG][o.e.a.s.TransportSearchAction] [node-client] [160257] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [node-3][10.1.0.14:9300][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.script.ScriptException: runtime error
at org.elasticsearch.painless.ScriptImpl.convertToScriptException(ScriptImpl.java:181) ~[?:?]
at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:128) ~[?:?]
at org.elasticsearch.search.fetch.subphase.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:52) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:163) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:502) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:356) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:353) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1385) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: java.lang.IllegalArgumentException: No field found for [sometype] in mapping with types []
at org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:80) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.painless.Executable$Script.execute(doc['sometype'].value='somevalue' @ <inline script>:5) ~[?:?]
at org.elasticsearch.painless.ScriptImpl.run(ScriptImpl.java:123) ~[?:?]
at org.elasticsearch.search.fetch.subphase.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:52) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:163) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:502) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:356) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.action.search.SearchTransportService$13.messageReceived(SearchTransportService.java:353) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1385) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) ~[elasticsearch-5.1.1.jar:5.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.1.1.jar:5.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]
It seems that this is the root cause.
Caused by: java.lang.IllegalArgumentException: No field found for [sometype] in mapping with types []
Do you need a field defined to use the scripted field?
Yes, the field you are trying to locate in doc_values must exist in the mapping for the scripted field to work.
I finally got it working.
I made field mapping for the field which will be created in scripted field.
Below example is f2 for the target field.
[root@elk-analytic elasticsearch]# curl -XGET localhost:9200/someindex/_mapping?pretty
{
"someindex" : {
"mappings" : {
"sometype" : {
"properties" : {
"date" : {
"type" : "date"
},
"f1" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"f2" : {
"type" : "integer"
},
"f3" : {
"type" : "long"
}
}
}
}
}
}
Scripted field definition.
Result
However, I had below warning when I first created the scripted field. Can this be ignored?
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.
