X-pack installation with logstash

Elastic Stack Logstash
1

hi... I have been trying to install x pack in my ELK cluster of 3 nodes. I have elasticsearch on all 3 nodes in logstash and kibana in 1 node which I operate physically and rest using ssh server . I have installed x pack in ES of all nodes and in kibana and logstash . ES and kibana is working fine and I am able to use new functionalities on pre-loaded data . When I start logstash to pump new data into ES , it shows error.

[2017-06-13T16:37:44,413][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x7de13e33 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-06-13T16:37:44,596][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}

In spite of this error .. it is pumping data into ES . Config file is -

input {
file {
path => "/home/mywavia/Desktop/original/sample.txt"
start_position => "beginning"
}
}
filter {

grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch { hosts=>["master","slave-1","slave-2"]
user => elastic
password => changeme
}

}
I had followed this document- https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html
Can anyone suggest what's the problem ???

2

Have you configured the logstash.yml file according to these instructions? It seems odd that there is no link to this page though. I will create an issue for this.

3

i have added these lines into logstash.yml
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: ["192.168.2.157:9200","192.168.2.158:9200","192.168.2.156:9200"]
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: changeme

4

@Christian_Dahlqvist I have followed your link to configure Logstash .. and I have appended these lines into logstash.yml =>

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: ["http//master:9200","http//slave-2:9200","http//slave-1:9200"]
xpack.monitoring.elasticsearch.username: logstash_internal
xpack.monitoring.elasticsearch.password: changeme

Since , I have not enabled SSL/TLS so I skipped 3rd point of https://www.elastic.co/guide/en/x-pack/current/monitoring-logstash.html .

I am still getting the same error .

5

can you start logstash with --log.level=debug and send me the logs of the first couple of minutes? you can pm a private pastebin link or a gist if you want

6

Just to double check, the url hosts you've shown above aren't correct urls since they're missing the : character. So instead of:

xpack.monitoring.elasticsearch.url: ["http//master:9200","http//slave-2:9200","http//slave-1:9200"]

should be

xpack.monitoring.elasticsearch.url: ["http://master:9200","http://slave-2:9200","http://slave-1:9200"]
7

Hi .. thanks for your reply.

I have updated my logtash.yml . Now it's working fine . It's pumping data into ES . But it' s still giving some warnings .
Warnings are -

[2017-06-20T11:01:07,501][INFO ][logstash.pipeline ] Pipeline .monitoring-logstash started
[2017-06-20T11:01:07,544][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://master:9200/, http://slave-1:9200/, http://slave-2:9200/]}}
[2017-06-20T11:01:07,546][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://master:9200/, :path=>"/"}
[2017-06-20T11:01:07,632][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x180062b8 URL:http://master:9200/>}
[2017-06-20T11:01:07,632][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://slave-1:9200/, :path=>"/"}
[2017-06-20T11:01:07,655][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x2c86d8f4 URL:http://slave-1:9200/>}
[2017-06-20T11:01:07,656][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://slave-2:9200/, :path=>"/"}
[2017-06-20T11:01:07,709][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x1095b0b4 URL:http://slave-2:9200/>}
[2017-06-20T11:01:07,710][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-06-20T11:01:07,970][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-06-20T11:01:07,982][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>[#<URI::Generic:0x7cc6648c URL://master>, #<URI::Generic:0x20fef014 URL://slave-1>, #<URI::Generic:0x72e7904d URL://slave-2>]}
[2017-06-20T11:01:08,031][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/home/mywavia/Downloads/logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.0.4-java/vendor/GeoLite2-City.mmdb"}
[2017-06-20T11:01:08,106][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-06-20T11:01:08,501][INFO ][logstash.pipeline ] Pipeline main started
[2017-06-20T11:01:08,605][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-06-20T11:01:12,463][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-06-20T11:01:12,468][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x3e9569e6 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-06-20T11:01:12,500][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-06-20T11:01:12,504][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x6f63f27a URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-06-20T11:01:17,471][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-06-20T11:01:17,486][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x542d53e8 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-06-20T11:01:17,506][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-06-20T11:01:17,514][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x2bb4dc7 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-06-20T11:01:17,534][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable:

and So on.
But I can't see logstash instance in monitoring and management section through kibana.
here is the scrrenshot ->

1847×564 29.1 KB

Monitoring section is giving this->

1216×435 17.6 KB

Can you help me with this >

8

It's corresponding kibana trash is -

log [05:31:19.843] [info][status][ui settings] Status changed from uninitialized to green - Ready
request [05:31:55.504] [error][monitoring-ui] TypeError: Cannot read property 'type' of null
at clusters.reduce (/home/mywavia/kibana-5.4.0-linux-x86_64/plugins/x-pack/plugins/monitoring/server/cluster_alerts/alerts_clusters_aggregation.js:95:73)
at Array.reduce (native)
at callWithRequest.then.result (/home/mywavia/kibana-5.4.0-linux-x86_64/plugins/x-pack/plugins/monitoring/server/cluster_alerts/alerts_clusters_aggregation.js:67:21)
at tryCatcher (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/util.js:26:23)
at Promise._settlePromiseFromHandler (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/promise.js:503:31)
at Promise._settlePromiseAt (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/promise.js:577:18)
at Promise._settlePromises (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/promise.js:693:14)
at Async._drainQueue (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/async.js:123:16)
at Async._drainQueues (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/async.js:133:10)
at Immediate.Async.drainQueues (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/async.js:15:14)
at runCallback (timers.js:666:20)
at tryOnImmediate (timers.js:639:5)
at processImmediate [as _immediateCallback] (timers.js:611:5)

9

can you show me the pipeline configuration you're using, or even just the output section of it?

10

yeah ..

Right now , I have disabled x pack security . I have added my hosts file and system recognizes master , slave-1 and slave-2.

11

interesting, I don't see where the localhost is coming from if both your pipeline and xpack settings pointed to master/slave hostnames

12

yeah . i haven't mentioned localhost anywhere . I have used IP instead of localhost . I don't know why it is still going to localhost .

13

everything (es,ls,kibana) is version 5.4.0 right? I suggest that you run with --log.level=debug --debug.config and post the first few minutes of logging. please obscure anything you don't want shown in the logs, or send them privately

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.