X-pack installation with logstash

Elastic Stack Logstash
1

hi... I have been trying to install x pack in my ELK cluster of 3 nodes. I have elasticsearch on all 3 nodes in logstash and kibana in 1 node which I operate physically and rest using ssh server . I have installed x pack in ES of all nodes and in kibana and logstash . ES and kibana is working fine and I am able to use new functionalities on pre-loaded data . When I start logstash to pump new data into ES , it shows error.

[2017-06-13T16:37:44,413][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x7de13e33 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-06-13T16:37:44,596][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}

In spite of this error .. it is pumping data into ES . Config file is -

input {
file {
path => "/home/mywavia/Desktop/original/sample.txt"
start_position => "beginning"
}
}
filter {

grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch { hosts=>["master","slave-1","slave-2"]
user => elastic
password => changeme
}

}
I had followed this document- https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html
Can anyone suggest what's the problem ???

2

Have you configured the logstash.yml file according to these instructions? It seems odd that there is no link to this page though. I will create an issue for this.

3

i have added these lines into logstash.yml
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: ["192.168.2.157:9200","192.168.2.158:9200","192.168.2.156:9200"]
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: changeme

4

@Christian_Dahlqvist I have followed your link to configure Logstash .. and I have appended these lines into logstash.yml =>

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: ["http//master:9200","http//slave-2:9200","http//slave-1:9200"]
xpack.monitoring.elasticsearch.username: logstash_internal
xpack.monitoring.elasticsearch.password: changeme

Since , I have not enabled SSL/TLS so I skipped 3rd point of https://www.elastic.co/guide/en/x-pack/current/monitoring-logstash.html .

I am still getting the same error .

5

can you start logstash with --log.level=debug and send me the logs of the first couple of minutes? you can pm a private pastebin link or a gist if you want

6

Just to double check, the url hosts you've shown above aren't correct urls since they're missing the : character. So instead of:

xpack.monitoring.elasticsearch.url: ["http//master:9200","http//slave-2:9200","http//slave-1:9200"]

should be

xpack.monitoring.elasticsearch.url: ["http://master:9200","http://slave-2:9200","http://slave-1:9200"]
7

Hi .. thanks for your reply.

I have updated my logtash.yml . Now it's working fine . It's pumping data into ES . But it' s still giving some warnings .
Warnings are -

[2017-06-20T11:01:07,501][INFO ][logstash.pipeline ] Pipeline .monitoring-logstash started
[2017-06-20T11:01:07,544][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://master:9200/, http://slave-1:9200/, http://slave-2:9200/]}}
[2017-06-20T11:01:07,546][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://master:9200/, :path=>"/"}
[2017-06-20T11:01:07,632][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x180062b8 URL:http://master:9200/>}
[2017-06-20T11:01:07,632][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://slave-1:9200/, :path=>"/"}
[2017-06-20T11:01:07,655][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x2c86d8f4 URL:http://slave-1:9200/>}
[2017-06-20T11:01:07,656][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://slave-2:9200/, :path=>"/"}
[2017-06-20T11:01:07,709][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x1095b0b4 URL:http://slave-2:9200/>}
[2017-06-20T11:01:07,710][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-06-20T11:01:07,970][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword"}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-06-20T11:01:07,982][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>[#<URI::Generic:0x7cc6648c URL://master>, #<URI::Generic:0x20fef014 URL://slave-1>, #<URI::Generic:0x72e7904d URL://slave-2>]}
[2017-06-20T11:01:08,031][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/home/mywavia/Downloads/logstash-5.4.0/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.0.4-java/vendor/GeoLite2-City.mmdb"}
[2017-06-20T11:01:08,106][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>250}
[2017-06-20T11:01:08,501][INFO ][logstash.pipeline ] Pipeline main started
[2017-06-20T11:01:08,605][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-06-20T11:01:12,463][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-06-20T11:01:12,468][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x3e9569e6 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-06-20T11:01:12,500][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-06-20T11:01:12,504][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x6f63f27a URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-06-20T11:01:17,471][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-06-20T11:01:17,486][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x542d53e8 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-06-20T11:01:17,506][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-06-20T11:01:17,514][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>#<URI::HTTP:0x2bb4dc7 URL:http://logstash_system:xxxxxx@localhost:9200/_xpack/monitoring/?system_id=logstash&system_api_version=2&interval=1s>, :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-06-20T11:01:17,534][WARN ][logstash.outputs.elasticsearch] Marking url as dead. Last error: [LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable:

and So on.
But I can't see logstash instance in monitoring and management section through kibana.
here is the scrrenshot ->

1847×564 29.1 KB

Monitoring section is giving this->

1216×435 17.6 KB

Can you help me with this >

8

It's corresponding kibana trash is -

log [05:31:19.843] [info][status][ui settings] Status changed from uninitialized to green - Ready
request [05:31:55.504] [error][monitoring-ui] TypeError: Cannot read property 'type' of null
at clusters.reduce (/home/mywavia/kibana-5.4.0-linux-x86_64/plugins/x-pack/plugins/monitoring/server/cluster_alerts/alerts_clusters_aggregation.js:95:73)
at Array.reduce (native)
at callWithRequest.then.result (/home/mywavia/kibana-5.4.0-linux-x86_64/plugins/x-pack/plugins/monitoring/server/cluster_alerts/alerts_clusters_aggregation.js:67:21)
at tryCatcher (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/util.js:26:23)
at Promise._settlePromiseFromHandler (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/promise.js:503:31)
at Promise._settlePromiseAt (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/promise.js:577:18)
at Promise._settlePromises (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/promise.js:693:14)
at Async._drainQueue (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/async.js:123:16)
at Async._drainQueues (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/async.js:133:10)
at Immediate.Async.drainQueues (/home/mywavia/kibana-5.4.0-linux-x86_64/node_modules/bluebird/js/main/async.js:15:14)
at runCallback (timers.js:666:20)
at tryOnImmediate (timers.js:639:5)
at processImmediate [as _immediateCallback] (timers.js:611:5)

9

can you show me the pipeline configuration you're using, or even just the output section of it?

10

yeah ..

Right now , I have disabled x pack security . I have added my hosts file and system recognizes master , slave-1 and slave-2.

11

interesting, I don't see where the localhost is coming from if both your pipeline and xpack settings pointed to master/slave hostnames

12

yeah . i haven't mentioned localhost anywhere . I have used IP instead of localhost . I don't know why it is still going to localhost .

13

everything (es,ls,kibana) is version 5.4.0 right? I suggest that you run with --log.level=debug --debug.config and post the first few minutes of logging. please obscure anything you don't want shown in the logs, or send them privately