Elasticsearch output problem

Elastic Stack Logstash
1

Hi guys
I've got a problem after did upgrade to the latest logstash, elasticsearch and kibana.
Elasticsearch with kibana works fine. I see data, searches, visualisation and dashboards work perfect. But logstash even doesn't start when i set elasticsearch output.

in logstash log i have this stacktrace

{  :timestamp=>"2016-04-20T14:59:51.423000+0000"
	,   :message=>"Pipeline aborted due to error"
	,   :exception=>#<NoMethodError: undefined method `set_validate_after_inactivity' for #<Java::OrgApacheHttpImplConn::PoolingHttpClientConnectionManager:0x1b1c4a69>>
	,   :backtrace=>
		[  "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.5-java/lib/manticore/client.rb:403:in `pool'"
		,   "org/jruby/RubyKernel.java:1858:in `tap'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.5-java/lib/manticore/client.rb:402:in `pool'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.5.5-java/lib/manticore/client.rb:208:in `initialize'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/manticore.rb:100:in `__build_connections'"
		,   "org/jruby/RubyArray.java:2414:in `map'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/http/manticore.rb:90:in `__build_connections'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/transport/base.rb:34:in `initialize'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport/client.rb:115:in `initialize'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.15/lib/elasticsearch/transport.rb:26:in `new'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/http_client.rb:129:in `build_client'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/http_client.rb:20:in `initialize'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:40:in `build'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch.rb:132:in `build_client'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.5.5-java/lib/logstash/outputs/elasticsearch/common.rb:14:in `register'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/output_delegator.rb:75:in `register'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:181:in `start_workers'"
		,   "org/jruby/RubyArray.java:1613:in `each'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:181:in `start_workers'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/pipeline.rb:136:in `run'"
		,   "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.1-java/lib/logstash/agent.rb:465:in `start_pipeline'"
		]  
	,   :level=>:error
	}

output config looks simple:

output {
  elasticsearch {
    hosts => ["10.1.2.3:9200"]
    index => "cwl-%{+YYYY.MM.dd}"
    document_type => "%{@type}"
    workers => 1
  }
}

if i use stdout output everything works, when change to elasticsearch - dies. Logstash 1.5 didn't make me such an issue.

2

I have the exact same issue on logstash 5.0.0-alpha. I can reproduce this on logstash 2.1.3 with Oracle Java 1.8 as well. This looks to be jruby related or more specifically to the manticore component however any help on how to avoid this will be greatly appreciated

Environment:
java -v
java version "1.8.0_77"
Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

jruby -v
jruby 1.7.23 (1.9.3p551) 2015-11-24 f496dd5 on Java HotSpot(TM) 64-Bit Server VM 1.8.0_77-b03 +jit [linux-amd64]

uname -a
Linux ak-ubuntu 3.13.0-85-generic #129-Ubuntu SMP Thu Mar 17 20:50:15 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

lsb_release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.4 LTS"

Config

`

input { 

tcp {
        port => 5000
        type => "apache_access_log"
    }
    tcp {
        port => 5001
        type => "apache_error_log"
    }
}

filter {

    if [type] == "apache_access_log" {
        grok {
            match => {
                "message" => "%{HOSTNAME:nodename} %{IP:nodeip} %{COMBINEDAPACHELOG}"
            }
        }
    }
    date {
        match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
        locale => "en"
    }
    if ([type] == "apache_error_log") {
        grok {
            match => ["message", "%{HOSTNAME:nodename} %{IP:nodeip} %{APACHE_ERROR_LOG}"]
        }
        date {
            match => ["timestamp", "EEE MMM d HH:mm:ss.SSSSSS yyyy"]
            locale => "en"
        }
    }
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
    }
    stdout {
        codec => json
    }
}`
3

On looking some more this looks to be perms related for me. The jruby bundle @ bundle/jruby/1.9/gems/manticore-0.5.2-java was installed as root. Running logstash as sudo seems to resolve the issue. Ideally it will be great to have this check as part of the logstash startup script itself

4

In my case it is something differ. I did chown logstash.logstash for logstash folder and subfolders. I even tried run it as root. Same result. If i use 1.5.x logstash everything amazing, just upgrade to logstash 2.x whatever build fails ....

Im using amazon linux with OpenJDK and ruby from logstash build.

java -version
java version "1.7.0_99"
OpenJDK Runtime Environment (amzn-2.6.5.0.66.amzn1-x86_64 u99-b00)
OpenJDK 64-Bit Server VM (build 24.95-b01, mixed mode)

/opt/logstash/vendor/jruby/bin/jruby -v
jruby 1.7.23 (1.9.3p551) 2015-11-24 f496dd5 on OpenJDK 64-Bit Server VM 1.7.0_99-mockbuild_2016_03_28_23_53-b00 +jit [linux-amd64]