Kibana scripted field: doc['some_field'] doesnt work when the field has a lot of data

[Felipsz (Felipe Gomes)]

Hi everyone,

Im creating some scripted fields to improve the visualization of the data that comes into Kibana 7.5.

I have a big field that its a JSON, and it has a lot of data, and i want to separate this data in multiple scripted fields. To exemplify, lets say that the JSON looks something like this:

{
  "data": {
    "data1": "123",
    "data2": "plane",
    "data3": "car",
     ......
    "dataXX": "321",
   }
}

And i want scripted fields to look like this: data1: 123, data2: plane, data3: car, dataXX: 321.

I know that the scripted field doesnt have an API to deal with JSON, so what i do is just use the .substring function to split the string, no problem at all. The thing is, when i use the doc['my_json_field.keyword'].value function only works when the data is small.

To exemplify, i've created the scripted field "data1", you can see the code of this script below, it just copies the data from the field "proxyRequest.message.content"

def data1 = doc['proxyRequest.message.content.keyword'];
return data1;

You can see the outputs below, when i have some big data, and when i have a smaller one (i've hidden the data with this red lines for obvious reasons):

I've tried to use params['_source']['proxyRequest.message.content'], but it doesnt output any data at all.

Any ideias of how to make the doc['some_field'] work for big amounts of data?

Thanks.

[Felipsz (Felipe Gomes)]

Just forgot to add, I've seen some people with the same issue as me, in the following topics:
https://discuss.elastic.co/t/scripted-field-read-field-data-from-a-document-using-doc-some-field-value-return-null-in-some-cases/187385/8

Scripted field: Read field data from a document using doc['some_filed'].value statement is not working for bigger field data

Hi Everyone, I am new to kibana. Need your help to solve the following problem which am facing from a day. I am working on kibana scripted field approach to transform a field value by accessing it using "doc[some_field].value" statement and perform some computation using groovy script then finally return a new field. Problem: "doc[some_field].value" is returning null in-case if the field value is large. Kibana version is : 6.0.1 field name: message.keyword => doc[message.keyword].value langu…

[Felipsz (Felipe Gomes)]

Hi,

Any help from anyone?

Im starting to think that this issue is caused by a limitation of Kibana, but im not sure.

[stu (Stuart Tettemer)]

Can you post your mappings? I'm guessing you have ignore_above.

I've tried to use params['_source']['proxyRequest.message.content'], but it doesnt output any data at all.

I'm wondering if params['_source']['proxyRequest']['message']['content'] works for you?

The _source contains the document closer to the raw JSON, whereas doc flattens the document, which is why the syntax is doc['proxyRequest.message.content'] for doc.

You can look at the contents of params['_source'] by doing Debug.explain(params['_source']).