Logstash cant connect to Elasticsearch on localhost

Elastic Stack Logstash
1

Hi,

I have setup Logstash, ES and Kibana on EC2 server and used localhost:9200 in all three setup yml file. curl command is also working fine. but once I am pushing msg using jdbc I am getting below error.

JDBC working file and displaying the result on local screen using below code snippet

"output {
stdout { codec => json_lines }
}"

conf.d -

input {
jdbc {
jdbc_connection_string => "jdbc:postgresql://oratopg.cspwou0nzxz1.us-east-1.rds.amazonaws.com:5432/postgres"
jdbc_user => "dhairyaj"
jdbc_password => "Rsadmin11"
jdbc_validate_connection => true
jdbc_driver_library => "/home/ubuntu/jdbc/postgresql-9.4.1209.jar"
jdbc_driver_class => "org.postgresql.Driver"
statement => "SELECT * from pg_stat_activity"
schedule => "* * * * *"
}
}
output {
elasticsearch {

protocol => "http"

     index => "pg_stat_activity"
    document_type => "pg_stat_activity"
    document_id => "%{uid}"
    hosts => ["http://localhost:9200"]
}

}

I have also tried both localhost and 127.0.0.1:9200 in conf file but its not working.

error:

Attempted to send a bulk request to Elasticsearch configured at '["http://localhost:9200"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided? {:error_message=>"[406] {"error":"Content-Type header [text/plain; charset=ISO-8859-1] is not supported","status":406}", :error_class=>"Elasticsearch ::Transport::Transport::Errors::NotAcceptable", :backtrace=>["/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/ transport/transport/base.rb:201:in __raise_transport_error'", "/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearc h/transport/transport/base.rb:312:inperform_request'", "/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/tran sport/transport/http/manticore.rb:67:in perform_request'", "/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/t ransport/client.rb:128:inperform_request'", "/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/bulk.rb:9 0:in bulk'", "/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:innon_threadsafe_bulk'", "/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_clie nt.rb:38:in bulk'", "org/jruby/ext/thread/Mutex.java:149:insynchronize'", "/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7 .1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in bulk'", "/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-j ava/lib/logstash/outputs/elasticsearch/common.rb:172:insafe_bulk'", "/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java /lib/logstash/outputs/elasticsearch/common.rb:101:in submit'", "/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/l ogstash/outputs/elasticsearch/common.rb:86:inretrying_submit'", "/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib

2

I think your LS version is not the same as the elasticsearch one.
Upgrade Logstash.

3

Thanks for replying. I already tried that. My ES, LS and Kibana all three are on same version.

root@ip-10-0-1-113:/etc/logstash-6.0.1# bin/logstash --version
logstash 6.0.1

root@ip-10-0-1-113:/etc/logstash-6.0.1# curl -XGET localhost:9200/_nodes/_all/process?pretty
{
"_nodes" : {
"total" : 1,
"successful" : 1,
"failed" : 0
},
"cluster_name" : "globo-monitoring",
"nodes" : {
"GuiTz9rlTQWprv0kYuBh3Q" : {
"name" : "globo-es01",
"transport_address" : "127.0.0.1:9300",
"host" : "localhost",
"ip" : "127.0.0.1",
"version" : "6.0.1",
"build_hash" : "601be4a",
"roles" : [
"master",
"data",
"ingest"
],
"process" : {
"refresh_interval_in_millis" : 1000,
"id" : 29972,
"mlockall" : false
}
}
}
}

4

Did you upgrade Logstash ?

This look weird to me:

etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/ transport/transport/base.rb:201:in

See the 2.3.4 dir name?

That said I’m moving the question to #logstash

5

I have installed LS plugin and version is 2.3.4. Details are below. Pls let me know if is this not correct way.

Tried below but it won't working :
root@ip-10-0-1-113:/etc/logstash-6.0.1/bin# /etc/logstash-6.0.1/bin/logstash --config /etc/logstash-6.0.1/conf.d/kibana.conf
ERROR: Unrecognised option '--config'

It's working fine :

root@ip-10-0-1-113:/etc/logstash-6.0.1/bin# /etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/bin/logstash --config /etc/logstash-6.0.1/conf.d/kibana.conf
Settings: Default pipeline workers: 4
Pipeline main started

Below are the plugin details
root@ip-10-0-1-113:/etc/logstash-6.0.1/plugin/plugin# ls -ltr
total 140496
-rw-r--r-- 1 root root 143859279 Jul 16 2016 logstash-all-plugins-2.3.4.tar.gz
drwxr-xr-x 6 root root 4096 Dec 12 12:05 logstash-2.3.4

root@ip-10-0-1-113:/etc/logstash-6.0.1/plugin/plugin/logstash-2.3.4/bin# ls -ltr
total 44
-rw-rw-r-- 1 root root 2947 Jun 27 2016 setup.bat
-rw-rw-r-- 1 root root 245 Jun 27 2016 rspec.bat
-rwxrwxr-x 1 root root 322 Jun 27 2016 rspec
-rw-rw-r-- 1 root root 251 Jun 27 2016 logstash-plugin.bat
-rw-rw-r-- 1 root root 203 Jun 27 2016 plugin.bat
-rwxrwxr-x 1 root root 199 Jun 27 2016 plugin
-rwxrwxr-x 1 root root 439 Jun 27 2016 logstash-plugin
-rwxrwxr-x 1 root root 5330 Jun 27 2016 logstash.lib.sh
-rw-rw-r-- 1 root root 689 Jun 27 2016 logstash.bat
-rwxrwxr-x 1 root root 1854 Jun 27 2016 logstash

6

I don't think that this is a valid setting for the elasticsearch output plugin!

Regards,
Nachiket Vartak

7

Yes, got it. Thanks for pointing out this issue. I have started the LS as below and now it's working fine.

/etc/logstash-6.0.1/bin/logstash -f /etc/logstash-6.0.1/conf.d/kibana.conf

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.