Hi all,
I believe ES is bottlenecking. I researched this in logstash-users. I have
a redis-logstash-ES cluster with logs pre-indexed by the client. The error
I am receiving in logstash is:
{:timestamp=>"2013-08-08T03:43:20.705000+0000", :message=>"Too many active
ES requests, blocking now.", :inflight_requests=>50,
:max_inflight_requests=>50, :level=>:info}
The thread that referred me to a possible ES bottleneck is:
https://groups.google.com/forum/?fromgroups=#!searchin/logstash-users/too$20many$20es$20requests/logstash-users/zNCSD90auxY/UL0NGm3xX4cJ
The ES hosts have 128 gb RAM, quad CPUs with 8 cores each and 1.9 TB disk
available in RAID 10. ES has 67 GB RAM dedicated to it. Logstash gets 2-10
GB RAM. Disk has 93% available.
ES Setup:
cluster.name: "vault_elasticsearch"
node.name: "host-02"
node.max_local_storage_nodes: 1
index.number_of_shards: 3
index.number_of_replicas: 2
bootstrap.mlockall: true
gateway.recover_after_nodes: 2
gateway.recover_after_time: 10s
gateway.expected_nodes: 3
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["host-01:9300", "host-02:9300",
"host-03:9300"]
indices.memory.index_buffer_size: 50%
index.translog.flush_threshold_ops: 50000
index.refresh_interval: 60s
index.merge.policy.use_compound_file: false
I searched around various ES resources to come up with this
configuration. I am just learning about ES operations. What are your
recommendations to avoid this problem?
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.