Daily indices for a period of 1 month with wildcard search taking long time

Elastic Stack Kibana
1

Hi Team, i am having 1TB of data over the span of 1 month where we create daily indices and currently we are creating around 20 daily indices with total data of 30-35 GB per day . Field-stats api is taking 50 secs to get the 1000 shards for the period of 1 month and our backend store is Network Storage . Can you please let us know if field stats will take 50 secs in this scenario from Kibana.

2

Our Elasticsearch version is 2.1 and kibana is 4.3.0

3

Sounds like you have too many shards, that won't be helping.

4

Hi we are creating index per app and we have multiple apps. So instead of creating 20 primary shards per day , do you suggest us to consolidate the shards and use shard routing to route the small apps logs to 1 shard?

5

Just create daily/weekly indices with a lot less shards

6

Also if 1 of my app has a need of 1 TB data per day , in that case i will have to create more number of shards, In that case i would not have solution to reduce number of shards . Please suggest.

7

I missed that you have network storage, that will be slow.

You're best option is to remove the network storage and add more nodes.

8

Thanks for providing inputs . Can we achieve some sort of improvement by moving from NAS to SAN . Currently network storage is only option.

9

Maybe, you'd have to test.
We don't usually recommend network based storage.