In preparation for the removal of types I'm experimenting with ES6.0.0-rc1 to upgrade from our ancient 2.4.6 install. I'm trying to take our index with hundreds of types and split it into multiple indexes (since it's 1 type per index now). I'm coming across speed issues with Kibana and the queries it runs. Was wondering if anyone had any insight on how to improve performance?
Existing System
- ES 2.4.6
- 1 index, 7 shards
- 2 nodes
- 1 index with approx 400 types
- 630M documents
- approx 10s to view discover tab (7 days) in Kibana
New System
- ES 6.0.0-rc1
- approx 400 indices, 1 shard each
- 2 nodes
- 1 index per type
- 630M documents
- approx 50s to view discover tab (7 days) in Kibana
Since the documents in each type have different combinations of fields and mappings (~1000 fields) I don't really want to have them in the same index as it ends up being extremely sparse and I end up with mapping conflicts everywhere.
Do I try to shoehorn the data back into a single index with mapping conflict hell, sparse data and and continue that way? Or maybe there's a better way to do this? Hope that makes sense!
For reference, here is the query Kibana runs:
{"version":true,"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"_source":{"excludes":[]},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"3h","time_zone":"Europe/London","min_doc_count":1}}},"stored_fields":["*"],"script_fields":{},"docvalue_fields":["@timestamp","qs.newsDate"],"query":{"bool":{"must":[{"match_all":{}},{"range":{"@timestamp":{"gte":1507503600000,"lte":1508108399999,"format":"epoch_millis"}}}],"filter":[],"should":[],"must_not":[]}},"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"*":{}},"fragment_size":2147483647}}