Failed to flush outgoing items

Elastic Stack Elasticsearch
1

Hello,
I tried set up ELB stack in single Vagrant machine based on Ubuntu 14.04 amd64 box. Whole stack was functional (ES 1.5.2, LS 1.5.0-1_all, Kibana 4.0.2-linux-x64) until I install Watcher plugin. It seems that Logstash couldn't connect to ES:

{:timestamp=>"2015-06-09T12:13:42.021000+0000", :message=>"Got error to send bulk of actions to elasticsearch server at logger : blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];", :level=>:error, :file=>"logstash/o
utputs/elasticsearch.rb", :line=>"464", :method=>"flush"}
{:timestamp=>"2015-06-09T12:13:42.022000+0000", :message=>"Failed to flush outgoing items", :outgoing_count=>2, :exception=>org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/
2/no master];, :backtrace=>["org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(org/elasticsearch/cluster/block/ClusterBlocks.java:151)", "org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(org/elasticsearch/cluster/block/Cluster
Blocks.java:141)", "org.elasticsearch.action.bulk.TransportBulkAction.executeBulk(org/elasticsearch/action/bulk/TransportBulkAction.java:210)", "org.elasticsearch.action.bulk.TransportBulkAction.access$000(org/elasticsearch/action/bulk/TransportBulkAction.java:73)", "or
g.elasticsearch.action.bulk.TransportBulkAction$1.onFailure(org/elasticsearch/action/bulk/TransportBulkAction.java:148)", "org.elasticsearch.action.support.TransportAction$ThreadedActionListener$2.run(org/elasticsearch/action/support/TransportAction.java:137)", "java.ut
il.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor.java:1142)", "java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor.java:617)", "java.lang.Thread.run(java/lang/Thread.java:745)"], :level=>:warn, :fi
le=>"stud/buffer.rb", :line=>"231", :method=>"buffer_flush"}

When I removed Watcher plugin my ELB stack start working.

LS config:

input {
  file {
    path => "/var/log/syslog"
    type => "syslog"
  }

  udp {
    port => 30001
  }

  heartbeat {
  }
}

output {
    elasticsearch {
        host => "logger"
        cluster => "logstash"
    }
}

ES config:

cluster.name: "logstash"
node.name: "logger"
network.host: logger

/etc/hosts:

127.0.0.1	localhost
127.0.1.1 logger.vbox logger
2

Could you try switching "protocol => transport" and checking that it works?

3

Wow, it works! You are my hero.

(by the way, with this new config I see much more responsive logstash process; without protocol => transport I wasn't able to restart logstash with service logstash restart, I must kill it manually and then start it)

Thank you very much

4

I was having the same issue, even after I uninstalled the plugin.

Just to clarify, this what your logstash config should look like to be fixed:

elasticsearch {
    host => "127.0.0.1"
    manage_template => false
    index => "..."
    cluster => "..."
    protocol => "transport"
}

Could you elaborate on what's going on? Why did the plugin mess it up?