Hi All,
We have migrated from solr(3.6) to es(0.20.5). We will create nearly 80
indexes per day with total size as ~300GB. Among the 80 indexes one index
size is 30 to 40GB(150 millions records ) , some indexes have 2 to 5GB. We
used 8 solr machines (4 indexer + 4 optimizer). Indexer job will create
indexes on each hour with merge factor as 1000 . Once the hour rotated the
previous hour index will be scp to optimizer machine, here we fully
optimize the hour index and merge it to the day index. With this current
setup we achieved best query performance other than one big index. Because
when we loading the big index most of the we are facing OOM. So we decided
to move es. In es too we have 8 datanodes and 2 master machines and 4
client nodes. We approximately know which indexes will have more size so we
decided to have no.of.shards/per index based on their size. We set 5 shards
with 1 replication for the big index alone , remaining have 1 shard + 1
replication.In solr when we query we get results in less than a second for
small indexes but in es it takes 3 to 4 secs.We used QUERY_AND_FETCH type
for 1P+1R indexes and QUERY_THEN_FETCH type for 5P+1R indexes.I have shared
my configuration below , can any one suggest why we are getting 3 secs in
es for small indexes too ? and big index will take 200 secs to 500 secs,
how to reduce this ? FYI: We are moving in 0.90.5.
In solr we set 4GB for optimizer machines , In es we set 8GB for all nodes
(master + data)
In elasticsearch.yml
index.refresh_interval: 30s
index.merge.policy.max_merge_at_once: 3 (Because we are not optimizing in
es. slow indexing will be acceptable.)
index.merge.policy.segments_per_tier: 3
indices.store.throttle.type: merge
indices.store.throttle.max_bytes_per_sec: 50mb
index.cache.field.type: soft
index.cache.field.max_size: 5000000
index.cache.field.expire: 15m
action.disable_delete_all_indices: true
index :
analysis :
analyzer :
default_index :
type : custom
tokenizer : standard
filter : [ word_delimiter, lowercase, snowball ]
default_search :
type : custom
tokenizer : standard
filter : [ word_delimiter, lowercase, snowball ]
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.