How to disable the addition of flattened source data to fields in Elasticsearch Kibana 7.16.3

Elastic Stack Kibana
1

We are using Elasticsearch, Logstash and Kibana version 7.16.3. Previously we were using version 7.10.x, and we used to have a minimum set of data in fields . However, after upgrading to the latest version, it is flattening the json data in _source and adding it to fields . The data is ingested to Elasticsearch using logstash and Node.js rest apis. Our _source json is pretty huge (containing array of arrays and array of objects) and because of this new feature, our kibana load times for json data has significantly increased. Is there any way to disable this new feature (if it cannot be disabled in Elasticsearch then atleast in kibana).

Example of how the data used to look before and after the ELK update. Note: I am posting a sample of how the data looks (not actual data). The schema of actual data includes multiple array of arrays and array of objects and each of those arrays usually have a lot of data.

ELK 7.10.x data sample:

{
    "_index": "example-index",
    "_type": "logs",
    "_id": "7b86b5e2e847b86b5e2e846b5e284",
    "_version": 6,
    "_score": 1,
    "_source": {
        "locations": {
            "coordinates": [
                [
                    0,
                    10
                ],
                [
                    10,
                    20
                ]
            ],
            "type": "linestring"
        },
        "Id": "7b86b5e2e847b86b5e2e846b5e284",
        "@timestamp": "2022-02-02T00:38:41.423Z",
        "object": {
            "bbox": {
                "topleftx": 10,
                "bottomrighty": 300,
                "bottomrightx": 20,
                "toplefty": 200
            }
        }
    },
    "fields": {
        "@timestamp": [
            "2022-02-02T00:38:41.423Z"
        ]
    },
    "sort": [
        1643769673658
    ]
}

ELK 7.16.3 data sample:

{
    "_index": "example-index",
    "_type": "logs",
    "_id": "7b86b5e2e847b86b5e2e846b5e284",
    "_version": 6,
    "_score": 1,
    "_source": {
        "locations": {
            "coordinates": [
                [
                    0,
                    10
                ],
                [
                    10,
                    20
                ]
            ],
            "type": "linestring"
        },
        "Id": "7b86b5e2e847b86b5e2e846b5e284",
        "@timestamp": "2022-02-02T00:38:41.423Z",
        "object": {
            "bbox": {
                "topleftx": 10,
                "bottomrighty": 300,
                "bottomrightx": 20,
                "toplefty": 200
            }
        }
    },
    "fields": {
        "locations.coordinates": [
            0,
            10,
            10,
            20
        ],
        "Id.keyword": [
            "7b86b5e2e847b86b5e2e846b5e284"
        ],
        "Id": [
            "f2c2c45b656679825f7b86b5e2e847669edf37af"
        ],
        "locations.type.keyword": [
            "linestring"
        ],
        "locations.type": [
            "linestring"
        ],
        "@timestamp": [
            "2022-02-02T00:38:41.423Z"
        ],
        "object.bbox.toplefty": [
            200
        ],
        "object.bbox.topleftx": [
            10
        ],
        "object.bbox.bottomrighty": [
            300
        ],
        "object.bbox.bottomrightx": [
            20
        ]
    }
}
2

Hi,

What do you mean by "kibana load time"? Get documents via Dev Tools? Or Dashboard loading time? And also please clarify how to get the data given in the post. Usual search query doesn't return any fields in my environment 7.16.2.

You said "Our _source json is pretty huge", but the new feature just adds fields and does nothing about the _source field. It confused me. Of course you can easily omit _source from the output by _source parameter of the query, and I'm sure this is not what you asking about.

3

Hi,

The new feature that adds fields causes the json view of Discover tab to load slower than usual (sometimes even crashes the chrome browser tab).

Our _source being huge causes more issues because each of the attributes in the _source json gets flattened and added to the field, thereby increasing the load time of json view in discover tab.

We would like to keep _source and if possible disable fields. On top of the loading issues, _source follows the schema that we have defined whereas the flattened data in field might confuse the users.

1 Like
4

This was asked recently and there's an issue in GitHub for this, I'll see if I can find it (but it's there somewhere).

2 Likes
5

Thanks for letting me know. Can I get a link for this issue and is there an ETA when it will get resolved? I would like to inform my team regarding the same.

I would also like to know whether fields is stored in Elasticsearch or its dynamically computed. If it is stored in Elasticsearch then we may also need to evaluate our storage requirements. Our _source is huge and if fields are also stored then our current storage capacity may not be good enough.

6

I haven't been able to find it sorry. And we don't provide ETAs on features or fixes.

7

Cool, no issues. If you remember, could you clarify if that feature was to disable fields at kibana level or Elasticsearch level. Also, we would like to know if fields are stored in Elasticsearch or its dynamically computed. Based on that we might need to re-evaluate our storage requirements. Thanks!

8

Hello @aiyerganapathy, you can disable the retrieval based on fields through the Kibana Advanced Setting discover:searchFieldsFromSource. It could be valuable to file an issue against Kibana with this description, since it sounds like there could be performance regression and you might not be the only user impacted.

Also to answer your question, there are no separate concrete fields that are stored separately from document _source. Instead, Elasticsearch loads _source, flattens it, parses the values, and returns these as fields. You can find more information here: Retrieve selected fields from a search | Elasticsearch Guide [7.16] | Elastic.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.