Kibana scripted field: doc['some_field'] doesnt work when the field has a lot of data

Elastic Stack Elasticsearch
1

Hi everyone,

Im creating some scripted fields to improve the visualization of the data that comes into Kibana 7.5.

I have a big field that its a JSON, and it has a lot of data, and i want to separate this data in multiple scripted fields. To exemplify, lets say that the JSON looks something like this:

{
  "data": {
    "data1": "123",
    "data2": "plane",
    "data3": "car",
     ......
    "dataXX": "321",
   }
}

And i want scripted fields to look like this: data1: 123, data2: plane, data3: car, dataXX: 321.

I know that the scripted field doesnt have an API to deal with JSON, so what i do is just use the .substring function to split the string, no problem at all. The thing is, when i use the doc['my_json_field.keyword'].value function only works when the data is small.

To exemplify, i've created the scripted field "data1", you can see the code of this script below, it just copies the data from the field "proxyRequest.message.content"

def data1 = doc['proxyRequest.message.content.keyword'];
return data1;

You can see the outputs below, when i have some big data, and when i have a smaller one (i've hidden the data with this red lines for obvious reasons):

problem
problem1055×368 19.7 KB

I've tried to use params['_source']['proxyRequest.message.content'], but it doesnt output any data at all.

Any ideias of how to make the doc['some_field'] work for big amounts of data?

Thanks.

2

Just forgot to add, I've seen some people with the same issue as me, in the following topics:
https://discuss.elastic.co/t/scripted-field-read-field-data-from-a-document-using-doc-some-field-value-return-null-in-some-cases/187385/8

3

Hi,

Any help from anyone?

Im starting to think that this issue is caused by a limitation of Kibana, but im not sure.

4

Can you post your mappings? I'm guessing you have ignore_above.

I've tried to use params['_source']['proxyRequest.message.content'], but it doesnt output any data at all.

I'm wondering if params['_source']['proxyRequest']['message']['content'] works for you?

The _source contains the document closer to the raw JSON, whereas doc flattens the document, which is why the syntax is doc['proxyRequest.message.content'] for doc.

You can look at the contents of params['_source'] by doing Debug.explain(params['_source']).

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.