Get value from json object

Elastic Stack Kibana
1

Hi!
I have Kibana 7.3.2. I want to use own scripted field.
I spent 5 hours for looking for the good try to get value from json object for a my scripted field.
I have a next field with json object:
{ "timestamp": "2020-03-18 12:01:23.1811",
"correlationId": "825ee105-0eab-4f56-9c96-939c621d9c18",
"level": "INFO", "logger":
"Contracts",
"message": "WorkerService.Work :825ee105-0eab-4f56-9c96-939c621d9c18",
"json": {"SysInfo": {"TimeStamp": 1582013206597,
"InfId": 1,"Version": "v1","Method": "GetRsaId",
"Parameters": {"cisContractId": "ab06ca8c-92df-4405-92ec-c1fd53cbfe41",
"cisAddendumId": "cfd35e89-e342-4678-9419-13da5fdc5f36",
"docType": "Contract",
"docSeries": "XXX",
"docNumber": "3456789120"},
"TicketId": "6f9619ff-8b86-d011-b42d-00cf4cf964ff",
"Comment": "something"}} }

I used this:

  1. doc['message.keyword.correlationId']
  2. doc['message.keyword.correlationId'].value
  3. doc['message.keyword']['correlationId']
  4. doc['message.keyword']['correlationId'].value

But all tries fell.

Can u help with the my issue?

2

Hi @Avend544 -- What does your error output say?

One other thing you could try is using params._source:

params._source.message.correlationId
3

Hi @lukeelmers ! Thanks for your quick answer!
I get next error, when i try open the my index in the discover panel:

image
image1278×691 40.6 KB

The your example doesn't work. I attached message of kibana logs. I guess problem in another level of the kibana's message.

image
image1274×688 58.1 KB

When i call field like that : doc['message.keyword'] i get full string, but I want get field from message property. For example "correlationId".

4

When i call field like that : doc['message.keyword'] i get full string, but I want get field from message property.

What does your mapping for message look like? Is it a string?

If so, that's most likely your problem -- you'll need to parse that string in your scripted field first (which isn't going to be very performant).

You might instead consider ingesting this as a nested field, assuming you don't need to build visualizations on top of this data. Visualizations support for nested fields is coming soon, but currently they are only supported in Discover.

5

Yes, it is a string field. Can do i convert this field to the JSON or only use parse?
What script language can understand scripted field?
I using https://groovyconsole.appspot.com/ but not all methods works in the painless script, which work in groovy.
Can u give a link to the all method supported in the Painless

6

I try parse a string:
`String docNumberdConst = "docNumber";
def message = doc['message.keyword'];
String message1 = (String)message;
if(message1.contains(docNumberdConst)){
int indx = message1.indexOf(docNumberdConst);
return String.valueOf(indx);
}

return "empty";`

But when i try cast painless string to java string (String)message;, i get a next error:

Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"},{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"},{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"fetch","grouped":true,"failed_shards":[{"shard":0,"index":"contracts-copy-svc-2020.03.18","node":"B42056aPTIm5sWSxuQamCA","reason":{"type":"script_exception","reason":"runtime error","script_stack":["message1 = (String)message;\r\n"," ^---- HERE"],"script":"String docNumberdConst = "docNumber";\r\ndef message = doc['message.keyword'];\r\nString message1 = (String)message;\r\nif(message1.contains(docNumberdConst)){\r\n int indx = message1.indexOf(docNumberdConst);\r\n return String.valueOf(indx);\r\n}\r\n\r\nreturn "empty";\r\n","lang":"painless","caused_by":{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"}}},{"shard":0,"index":"contracts-copy-svc-2020.03.25","node":"lHWpRR4iSOeBcjGgEHeplA","reason":{"type":"script_exception","reason":"runtime error","script_stack":["message1 = (String)message;\r\n"," ^---- HERE"],"script":"String docNumberdConst = "docNumber";\r\ndef message = doc['message.keyword'];\r\nString message1 = (String)message;\r\nif(message1.contains(docNumberdConst)){\r\n int indx = message1.indexOf(docNumberdConst);\r\n return String.valueOf(indx);\r\n}\r\n\r\nreturn "empty";\r\n","lang":"painless","caused_by":{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"}}},{"shard":0,"index":"contracts-copy-svc-2020.03.26","node":"B42056aPTIm5sWSxuQamCA","reason":{"type":"script_exception","reason":"runtime error","script_stack":["message1 = (String)message;\r\n"," ^---- HERE"],"script":"String docNumberdConst = "docNumber";\r\ndef message = doc['message.keyword'];\r\nString message1 = (String)message;\r\nif(message1.contains(docNumberdConst)){\r\n int indx = message1.indexOf(docNumberdConst);\r\n return String.valueOf(indx);\r\n}\r\n\r\nreturn "empty";\r\n","lang":"painless","caused_by":{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"}}}],"caused_by":{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"}},"status":400}

Error: Request to Elasticsearch failed: {"error":{"root_cause":[{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"},{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"},{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"fetch","grouped":true,"failed_shards":[{"shard":0,"index":"contracts-copy-svc-2020.03.18","node":"B42056aPTIm5sWSxuQamCA","reason":{"type":"script_exception","reason":"runtime error","script_stack":["message1 = (String)message;\r\n","                   ^---- HERE"],"script":"String docNumberdConst = \"docNumber\";\r\ndef message = doc['message.keyword'];\r\nString message1 = (String)message;\r\nif(message1.contains(docNumberdConst)){\r\n    int indx = message1.indexOf(docNumberdConst);\r\n    return String.valueOf(indx);\r\n}\r\n\r\nreturn \"empty\";\r\n","lang":"painless","caused_by":{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"}}},{"shard":0,"index":"contracts-copy-svc-2020.03.25","node":"lHWpRR4iSOeBcjGgEHeplA","reason":{"type":"script_exception","reason":"runtime error","script_stack":["message1 = (String)message;\r\n","                   ^---- HERE"],"script":"String docNumberdConst = \"docNumber\";\r\ndef message = doc['message.keyword'];\r\nString message1 = (String)message;\r\nif(message1.contains(docNumberdConst)){\r\n    int indx = message1.indexOf(docNumberdConst);\r\n    return String.valueOf(indx);\r\n}\r\n\r\nreturn \"empty\";\r\n","lang":"painless","caused_by":{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"}}},{"shard":0,"index":"contracts-copy-svc-2020.03.26","node":"B42056aPTIm5sWSxuQamCA","reason":{"type":"script_exception","reason":"runtime error","script_stack":["message1 = (String)message;\r\n","                   ^---- HERE"],"script":"String docNumberdConst = \"docNumber\";\r\ndef message = doc['message.keyword'];\r\nString message1 = (String)message;\r\nif(message1.contains(docNumberdConst)){\r\n    int indx = message1.indexOf(docNumberdConst);\r\n    return String.valueOf(indx);\r\n}\r\n\r\nreturn \"empty\";\r\n","lang":"painless","caused_by":{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"}}}],"caused_by":{"type":"class_cast_exception","reason":"class_cast_exception: cannot explicitly cast def [org.elasticsearch.index.fielddata.ScriptDocValues.Strings] to java.lang.String"}},"status":400}
1 Like
7

I try use method indexOf but i cant find my pattern value "docNumber" from doc['message.keyword'];
When i use indexOf with doc['message.keyword']; method return -1
But when i use it on another string like that "asdasdasda docNumber asdasdas" i get index by my pattern.
I don't understand why i don't get index of pattern...

Code:
String docNumberdConst = "docNumber"; def message1 = doc['message.keyword']; int i = message1.indexOf(docNumberdConst); if(i >= 0){ return "exist"; } return "EMPTY " + i;

8

The painless specification and the painless API reference should have the information you are looking for.

My bad, but I just realized I was mistaken in this comment above -- I forgot that painless doesn't actually allow JSON parsing in its API. (In part because it would be really terrible from a performance perspective as I mentioned).

So your best bet is going to be parsing those message strings to objects at ingest time as stated in the comment above... these two threads have more on this topic:

The only other alternative I can think of would be regex matching for a substring, which is also going to slow things down, and definitely isn't considered a best practice for a situation like this.

9

@lukeelmers, Can u help with regex example ?
i try this, but it doesn't work....
def m = /\|docNumber([^\|]+)/.matcher(doc['message.keyword']);
if (m.matches()) {
return m.group(1);
} else {
return "EMPTY";
}
I got a next exception: "Script is invalid. View script preview for details". If i can open some view with error text, then tell to me pls

And what's problem in parse doc['message.keyword'] why i can't use it? Why Painlees string different by Java string?

10

I attached the screenshot with my filed and it's type.

image
image1288×379 18.9 KB

May be it help u give a solution to me.
If i can change type of field, then can u say how to make it. It will help me for take value like a string and will perform this with Java api.

11

I found a solution. I just separated my json object on filebeat layer, and after that i get value like from ordinary field.
I add that string in filebeat.yml
#----------------------------- Logstash output --------------------------------
output.logstash:
hosts: ["elog.kibana.ru:5001"]
and after that elastic will do automatic seporating JSON object and recognizing type of field

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.