# Output to elasticsearch gives errors but works with stdout

**URL:** <https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530>\
**Category:** Logstash\
**Created:** [July 1, 2016, 12:30pm UTC](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530 "2016-07-01T12:30:34Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![infinitetutts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/infinitetutts/32/10632_2.png) [@infinitetutts](https://discuss.elastic.co/u/infinitetutts)\
**Post date:** [July 1, 2016, 12:30pm UTC](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530/1 "2016-07-01T12:30:34Z")

</div>

I think the error could be related to the ruby code, how do I fix this ?

Config:

```
input {
  beats {
    port => 5044
    ssl => false
  }
}

filter {
  if [type] == "apache" {
ruby {
code => " if event['message']
event['message'] = event['message'].gsub('\x','Xx')
event['message'] = event['message'].gsub('\x','XXx')
end
"
}

json {
      source => "message"
    }
}
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

```

Logstash log:

> {:timestamp=\>"2016-07-01T13:23:30.475000+0100", :message=\>"Connection refused", :class=\>"Manticore::SocketException", :backtrace=\>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:37:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in `call\_once'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `perform\_request'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:257:in `perform\_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/sniffer.rb:32:in `hosts'", "org/jruby/ext/timeout/Timeout.java:147:in `timeout'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/sniffer.rb:31:in `hosts'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:79:in `reload_connections!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:72:in `sniff!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.0-java/lib/logstash/outputs/elasticsearch/http\_client.rb:60:in `start_sniffing!'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.0-java/lib/logstash/outputs/elasticsearch/http\_client.rb:60:in `start_sniffing!'", "org/jruby/RubyKernel.java:1479:in `loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.0-java/lib/logstash/outputs/elasticsearch/http\_client.rb:59:in `start\_sniffing!'"], :level=\>:error}  
> {:timestamp=\>"2016-07-01T13:23:52.470000+0100", :message=\>"CircuitBreaker::rescuing exceptions", :name=\>"Beats input", :exception=\>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=\>:warn}  
> {:timestamp=\>"2016-07-01T13:23:52.470000+0100", :message=\>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=\>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=\>:warn}  
> {:timestamp=\>"2016-07-01T13:23:52.471000+0100", :message=\>"CircuitBreaker::rescuing exceptions", :name=\>"Beats input", :exception=\>LogStash::Inputs::Beats::InsertingToQueueTakeTooLong, :level=\>:warn}  
> {:timestamp=\>"2016-07-01T13:23:52.471000+0100", :message=\>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=\>LogStash::Inputs::BeatsSupport::CircuitBreaker::HalfOpenBreaker, :level=\>:warn}  
> {:timestamp=\>"2016-07-01T13:23:53.471000+0100", :message=\>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect\_backoff\_sleep=\>0.5, :level=\>:warn}  
> {:timestamp=\>"2016-07-01T13:23:53.472000+0100", :message=\>"CircuitBreaker::Open", :name=\>"Beats input", :level=\>:warn}  
> {:timestamp=\>"2016-07-01T13:23:53.473000+0100", :message=\>"Beats input: The circuit breaker has detected a slowdown or stall in the pipeline, the input is closing the current connection and rejecting new connection until the pipeline recover.", :exception=\>LogStash::Inputs::BeatsSupport::CircuitBreaker::OpenBreaker, :level=\>:warn}  
> {:timestamp=\>"2016-07-01T13:23:53.972000+0100", :message=\>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect\_backoff\_sleep=\>0.5, :level=\>:warn}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 3, 2016, 12:37am UTC](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530/2 "2016-07-03T00:37:10Z")

</div>

What's the connection with elasticcache and this?

---

<div class="post-metadata">

**Author:** ![infinitetutts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/infinitetutts/32/10632_2.png) [@infinitetutts](https://discuss.elastic.co/u/infinitetutts)\
**Post date:** [July 3, 2016, 8:35am UTC](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530/4 "2016-07-03T08:35:18Z")

</div>

Oh Sorry I meant ElasticSearch not Elastic Cache !!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 4, 2016, 5:36am UTC](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530/5 "2016-07-04T05:36:30Z")

</div>

> :message=\>"Connection refused"

This should be quite clear; there's nothing answering on localhost:9200. Is Elasticsearch running? If yes, is is really listening on that port or could there be a firewall blocking the connection (unlikely)?

---

<div class="post-metadata">

**Author:** ![infinitetutts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/infinitetutts/32/10632_2.png) [@infinitetutts](https://discuss.elastic.co/u/infinitetutts)\
**Post date:** [July 4, 2016, 7:47am UTC](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530/6 "2016-07-04T07:47:36Z")

</div>

```
lsof -i :9200
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 32619 elasticsearch 489u IPv6 341712524 0t0 TCP localhost:9200 (LISTEN)

```

As soon as I remove the the ruby code the output does go to elasticsearch

---

<div class="post-metadata">

**Author:** ![infinitetutts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/infinitetutts/32/10632_2.png) [@infinitetutts](https://discuss.elastic.co/u/infinitetutts)\
**Post date:** [July 4, 2016, 9:13am UTC](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530/7 "2016-07-04T09:13:04Z")

</div>

Ok so iv elastablished that there is a break in communication to elasticsearch:

`{:timestamp=>"2016-07-04T10:09:50.361000+0100", :message=>"Beats input: the pipeline is blocked, temporary refusing new connection.", :reconnect_backoff_sleep=>0.5, :level=>:warn}`

I have no idea why this is happening now all of a sudden, I tried sending it a minimal amount of logs to cross out that its not overloaded with logs. But I am still having this issue.

Do you please have any tips on what to check or how to fix ?

---

<div class="post-metadata">

**Author:** ![infinitetutts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/infinitetutts/32/10632_2.png) [@infinitetutts](https://discuss.elastic.co/u/infinitetutts)\
**Post date:** [July 4, 2016, 9:16am UTC](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530/8 "2016-07-04T09:16:05Z")

</div>

The connection does get established, but it breaks somewhere.

```
 lsof -i :9200
    COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
    java 17582 elasticsearch 506u IPv6 396166915 0t0 TCP localhost:9200 (LISTEN)
    java 17582 elasticsearch 6953u IPv6 396167077 0t0 TCP localhost:9200->localhost:42094 (ESTABLISHED)
    java 17582 elasticsearch 6954u IPv6 396168016 0t0 TCP localhost:9200->localhost:42105 (ESTABLISHED)
    java 17582 elasticsearch 6955u IPv6 396167082 0t0 TCP localhost:9200->localhost:42101 (ESTABLISHED)
    java 17582 elasticsearch 6956u IPv6 396167085 0t0 TCP localhost:9200->localhost:42103 (ESTABLISHED)
    java 17582 elasticsearch 6972u IPv6 396167099 0t0 TCP localhost:9200->localhost:42120 (ESTABLISHED)
    java 17582 elasticsearch 6973u IPv6 396167101 0t0 TCP localhost:9200->localhost:42121 (ESTABLISHED)
    java 17582 elasticsearch 6974u IPv6 396169012 0t0 TCP localhost:9200->localhost:42122 (ESTABLISHED)
    java 17582 elasticsearch 6975u IPv6 396169014 0t0 TCP localhost:9200->localhost:42123 (ESTABLISHED)
    java 17582 elasticsearch 6976u IPv6 396169016 0t0 TCP localhost:9200->localhost:42124 (ESTABLISHED)
    java 17918 logstash 43u IPv6 396168013 0t0 TCP localhost:42094->localhost:9200 (ESTABLISHED)
    java 17918 logstash 44u IPv6 396060623 0t0 TCP localhost:42101->localhost:9200 (ESTABLISHED)
    java 17918 logstash 45u IPv6 396178530 0t0 TCP localhost:42103->localhost:9200 (ESTABLISHED)
    java 17918 logstash 46u IPv6 396165809 0t0 TCP localhost:42107->localhost:9200 (ESTABLISHED)
    java 17918 logstash 47u IPv6 396155511 0t0 TCP localhost:42104->localhost:9200 (ESTABLISHED)
    java 17918 logstash 51u IPv6 396118384 0t0 TCP localhost:42112->localhost:9200 (ESTABLISHED)
    java 17918 logstash 52u IPv6 396171847 0t0 TCP localhost:42109->localhost:9200 (ESTABLISHED)
    java 17918 logstash 53u IPv6 396137975 0t0 TCP localhost:42110->localhost:9200 (ESTABLISHED)
    java 17918 logstash 54u IPv6 396177333 0t0 TCP localhost:42111->localhost:9200 (ESTABLISHED)
    java 17918 logstash 55u IPv6 396118385 0t0 TCP localhost:42113->localhost:9200 (ESTABLISHED)
    java 17918 logstash 59u IPv6 396162483 0t0 TCP localhost:42117->localhost:9200 (ESTABLISHED)
    java 17918 logstash 60u IPv6 396107470 0t0 TCP localhost:42120->localhost:9200 (ESTABLISHED)
```

---

<div class="post-metadata">

**Author:** ![infinitetutts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/infinitetutts/32/10632_2.png) [@infinitetutts](https://discuss.elastic.co/u/infinitetutts)\
**Post date:** [July 4, 2016, 11:23am UTC](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530/9 "2016-07-04T11:23:14Z")

</div>

I think the problem could be here,  
Elasticsearch log:

> [2016-07-04 12:22:02,389][WARN][cluster.routing.allocation.decider] [Ultra-Marine] high disk watermark [90%] exceeded on [YMlUds73Q9WSvS2bk7WKAA][Ultra-Marine][/var/lib/elasticsearch/elasticsearch/nodes/0] free: 23.6gb[8.7%], shards will be relocated away from this node  
> [2016-07-04 12:22:02,389][INFO][cluster.routing.allocation.decider] [Ultra-Marine] rerouting shards: [high disk watermark exceeded on one or more nodes]

---

<div class="post-metadata">

**Author:** ![infinitetutts](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/infinitetutts/32/10632_2.png) [@infinitetutts](https://discuss.elastic.co/u/infinitetutts)\
**Post date:** [July 4, 2016, 1:53pm UTC](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530/10 "2016-07-04T13:53:27Z")

</div>

I deleted all my elasticsearches indexes to fix this.  
`curl -XDELETE 'http://localhost:9200/filebeat-*'`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:49am UTC](https://discuss.elastic.co/t/output-to-elasticsearch-gives-errors-but-works-with-stdout/54530/11 "2017-07-06T04:49:45Z")

</div>