Poor query performance after removing types for ES6


#1

In preparation for the removal of types I'm experimenting with ES6.0.0-rc1 to upgrade from our ancient 2.4.6 install. I'm trying to take our index with hundreds of types and split it into multiple indexes (since it's 1 type per index now). I'm coming across speed issues with Kibana and the queries it runs. Was wondering if anyone had any insight on how to improve performance?

Existing System

  • ES 2.4.6
  • 1 index, 7 shards
  • 2 nodes
  • 1 index with approx 400 types
  • 630M documents
  • approx 10s to view discover tab (7 days) in Kibana

New System

  • ES 6.0.0-rc1
  • approx 400 indices, 1 shard each
  • 2 nodes
  • 1 index per type
  • 630M documents
  • approx 50s to view discover tab (7 days) in Kibana

Since the documents in each type have different combinations of fields and mappings (~1000 fields) I don't really want to have them in the same index as it ends up being extremely sparse and I end up with mapping conflicts everywhere.

Do I try to shoehorn the data back into a single index with mapping conflict hell, sparse data and and continue that way? Or maybe there's a better way to do this? Hope that makes sense!

For reference, here is the query Kibana runs:

{"version":true,"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"_source":{"excludes":[]},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"3h","time_zone":"Europe/London","min_doc_count":1}}},"stored_fields":["*"],"script_fields":{},"docvalue_fields":["@timestamp","qs.newsDate"],"query":{"bool":{"must":[{"match_all":{}},{"range":{"@timestamp":{"gte":1507503600000,"lte":1508108399999,"format":"epoch_millis"}}}],"filter":[],"should":[],"must_not":[]}},"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"*":{}},"fragment_size":2147483647}}


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.