Elasticsearch performance decrement after new index creation

Elastic Stack Elasticsearch
1

Hi all,

We have a 16 nodes cluster with some indexes (let say 9 active indexes and
10 empty and not used indexes) with 51 primary shards and 1 replica.
At startup we had 3 active indexes, reading from one and indexing into next
two. Search speed and indexing were satisfying for us.
I've created 6 new indexes and started searching and indexing on them.
Performance about 4 time decreased, even destination indexes are almost
empty and it must be faster at indexing startup.
The strange thing is that some nodes have high cpu usage(near 20-30 %)
because of ES threads when there is no indexing and searching request.
I read some threads in the forum regarding ES high cpu usage and set the
network.tcp.blocking to true, but didn't improve.

The only thing I've changed is that previously 100,000 filter
criteria(using or bool filters) were set to the search query and now its 1
million, however the loading and indexing documents are same as before.

ES configuration:

bootstrap.mlockall: true
discovery.zen.fd.ping_interval: 10
http.port: 9200
network.publish_host: ....147
index.cache.filter.max_size: 10
discovery.zen.minimum_master_nodes: 9
discovery.zen.fd.ping_timeout: 60s
index.cache.field.expire: 1m
gateway.recovery_after_nodes: 15
gateway.expected_nodes: 16
discovery.zen.fd.ping_retries: 10
index.cache.filter.expire: 1m
index.number_of_replicas: 1
transport.tcp.compress: true
discovery.zen.ping.unicast.hosts: ["cluster ips"]
network.tcp.blocking: true
gateway.recovery_after_time: 5m
index.number_of_shards: 51
indices.cache.filter.size: 64mb
discovery.zen.ping.timeout: 60s
index.query.bool.max_clause_count: 65536
node.name: ....
discovery.zen.ping.multicast.enabled: false
action.auto_create_index: false
index.cache.field.type: resident
index.refresh_interval: -1

Thanks for any help,
Vahid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

These are also additional info may help.
Using elasticsearch-0.20.5.

Profiled ES process and the following thread uses most of the cpu usage
while ES is idle:
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run()
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run()
org.elasticsearch.common.netty.channel.socket.oio.OioWorker.run()

org.elasticsearch.common.netty.channel.socket.oio.AbstractOioWorker.run()

org.elasticsearch.common.netty.channel.socket.oio.OioWorker.process()
java.io.PushbackInputStream.read()

.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.