Logstash to elasticsearch complaining state not recovered and no master

nivinm · July 10, 2015, 9:55pm

Hello,

I am changing my ELK configuration from logstash > kafka > river > elasticsearch. To logstash1 > kafka > logstash2 > elasticsearch, all 4 layers have a different set of servers.

For the logstash2 set I am using the following config file:
input

{ 
kafka 
	{zk_connect => "nvnkafn01.nvnnas.com:2181,nvnkafn02.nvnnas.com:2181,nvnkafn03.nvnnas.com:2181" 
	topic_id => 'windowseventlog'}
} 
output 
{ 
elasticsearch 
	{host => "nvnelsn01.nvnnas.com:9300,nvnelsn02.nvnnas.com:9300,nvnelsn03.nvnnas.com:9300"
	protocol => transport
	cluster => nvnels}
}

(I've tried changing the protocol from transport to http and even removing it to no avail.)

And getting the following error:

{:timestamp=>"2015-07-10T16:37:00.080000-0500", :message=>"Failed to flush outgoing items", :outgoing_count=>456, :exception=>org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];, :backtrace=>["org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(org/elasticsearch/cluster/block/ClusterBlocks.java:151)", "org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(org/elasticsearch/cluster/block/ClusterBlocks.java:141)", "org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(org/elasticsearch/action/bulk/TransportBulkAction.java:210)", "org.elasticsearch.action.bulk.TransportBulkAction.access$000(org/elasticsearch/action/bulk/TransportBulkAction.java:73)", "org.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(org/elasticsearch/action/bulk/TransportBulkAction.java:148)", "org.elasticsearch.action.support.TransportAction$ThreadedActionListener$2.run(org/elasticsearch/action/support/TransportAction.java:137)", "java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor.java:1142)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor.java:617)", "java.lang.Thread.run(java/lang/Thread.java:745)"], :level=>:warn}
{:timestamp=>"2015-07-10T16:38:01.127000-0500", :message=>"Got error to send bulk of actions: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];", :level=>:error}

Strange that it would complain about "state not recovered" and "no master", my elasticsearch cluster looks good:

 curl -XGET http://localhost:9200/_cluster/health?pretty
   {
  "cluster_name" : "nvnels",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 1,
  "active_shards" : 2,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0
}

Any advice? Attaching HQ screenshot.