Index Pattern Not Refreshing

[misterek]

I have an index pattern that won't refresh. I have quite a few fields that aren't recognized. When I look at the index pattern, it recognizes 3630 fields. Clicking refresh sometimes recognizes the new fields (the total will be north of 8000). However, within 10-20 seconds, it resets back to 3630 fields. It kind of feels like it's recognizing them but not persisting the new fields.

The fields do exist correctly in ElasticSearch. It seems to be Kibana that's having an issue recognizing them. Any thoughts on how I might dig a little deeper into what Kibana is doing?

[stephenb (Stephen Brown)]

Hi @misterek Welcome to the community.

Index patterns can get large with that many fields take a look at the Kibana Settings here and in particular

server.maxPayloadBytes: The maximum payload size in bytes for incoming server requests. Default: 1048576

try setting this in your kibana.yml to 2 or 5 mb

server.maxPayloadBytes: 2097152

[stephenb (Stephen Brown)]

@misterek What version of the Stack are you on? I understand there may be a fix for this in 7.11 just curious what version you are on.

[misterek]

Thank you @stephenb ! We are currently on 7.10.1. If I follow you, sounds like this might be a known defect that'll be fixed?

[stephenb (Stephen Brown)]

Defect.... up for debate

You should be able to apply the setting I gave above and it should work.
The engineers re-worked the way index patterns work in Kibana for several reasons.

There is a lot of other good stuff in 7.11.1 when you are ready.

[misterek]

Thanks @stephenb ! I will give this a shot and report back how it goes.

[misterek]

Ok, probably a dumb question, but is there a way to verify that the config is set in kibana? I went all the way up to 10 mb server.maxPayloadBytes: 10485760 and I'm still seeing the same behavior.

On the plus side, it's easily reproducible.

1. Click refresh index pattern.
2. Click on 'saved objects' (or, probably anything really)
3. Go back into the index pattern, and suddenly there are 8xxx fields!
4. Refresh the browser page
5. Sadly there are now 3xxx.

Feels like you're on the right track with persisting, but would love to verify that that setting is actually being picked up.

Also, I've never seen an error message related to this. Would there be somewhere I might see an error message?

[misterek]

Problem solved!

I believe that I had two issues. I don't think I realized that refreshing an index pattern was a GET/PUT combination from the browser. It turns out that we had a max body size configured of 1MB. So I think that I had 2 issues -- I never saw an error message because the PUT was failing, and my index size was greater than the max payload (it ended up being 1.73MB)

Thanks @stephenb for the help!