Kibana field_stats request takes 20000 ms if number of Indices and shards are more


(Sushant Sood) #1

Hi Team , i am getting high response times for the field stats request , if i am having around 240 shards with daily indices . Can you please suggest how can i reduce these response times with that many number of open shards.


(Mark Walkom) #2

How much heap does ES have?


(Sushant Sood) #3

ES has 7 data nodes with 30 GB of heap every node.


(Christian Dahlqvist) #4

How are you measuring this latency? Is this just for the field_stats call or for the following aggregation as well?


(Sushant Sood) #5

Its just the Field Stats , I tested multiple options like kept all the 240 shards opened in memory and executed the request from Kibana for the default 15 mins results .Then I closed certain Indices and then again reloaded the kibana UI and checked the field_stats response in Network sniffer in Developer Tools of Mozilla Firefox .


(Christian Dahlqvist) #6

That sounds odd. What is the load on the cluster? Are you seeing a lot of long GC? Which version of Elasticsearch are you on?


(Sushant Sood) #7

I am using ES 2.1 , Kibana 4.3.0 , Load on the cluster is like i am indexing around 25-30 GB of data daily in Elasticsearch and I have not yet analysed the GC runs rather i have tried to reduce the number of shards per Index , earlier I was using 5 shards per Index and now I have reduced it to 1 Shard per index and tried creating Daily Index.
If I have 240 Open shards also , this issue should not come ? I thing I am using NAS storage . Would it cause an issue in field_stats response ?


(Sushant Sood) #8

Hi team,
Please suggest what would be the option to resolve the slowness of field_stats.


(Sushant Sood) #9

Hi Team, Please suggest what could cause the field_stats to be slow with more number of Shards.


(Mark Walkom) #10

What sort of NAS storage is this?


(Sushant Sood) #11

It is nfs4 storage.


(Mark Walkom) #12

But are you mounting something locally that ES then runs on?


(Sushant Sood) #13

No I have not mounted anything locally. When I reduced the shards from 240 to 50 , field stats response came down drastically from 20000 ms to 2700 ms


(Mark Walkom) #14

Then how is it setup?


(Sushant Sood) #15

oh sorry i thought you were asking if i mounted anything else apart from my nfs . I have mounted the NFSshare on my hypervisors and my Elasticsearch is deployed in VMs created by those hypervisors and Elasticsearch is using the same nfsmount for storing the indices.


(Diranged) #16

I've recently opened an issue on Github to dig into this (https://github.com/elastic/kibana/issues/9386) .. we're seeing the same issue, but its upwards of 35s to return the query. I strongly suspect that its our "warm" data nodes that hold the majority of our data, but is old "closed" indexes. I think that they're slower to respond, and this query requires all the nodes in the cluster to respond it seems.


(Sushant Sood) #17

Thanks for the reference , i will try to follow github issue .


(system) #18