Logstash + X-Pack with disabled security

Elastic Stack Logstash
1

Hi,

We're setting up an ELK cluster with X-Pack installed, initially with only monitoring enabled for X-Pack. We've got everything working correctly for elasticsearch and kibana, but logstash is failing to send metrics to the elasticsearch cluster, although the elasticsearch output is working correctly and logs are being stored in the elasticsearch cluster.

Is it possible to disable x-pack security for logstash in the same way as for kibana and elasticsearch?

2

Can you share the error you are seeing?

3

This is the logstash output config:

output {
    if "metric" not in [tags] {
        elasticsearch {
            hosts => [ "<es_client_elb>:9200" ]
            template => "/usr/share/logstash/bin/logstash-template-es5.json"
            template_overwrite => true
            idle_flush_time => 10
            flush_size => 3000
        }
    }
    if "metric" in [tags] {
        statsd {
            id => "statsd_events_rate_1m"
            gauge => { "events.rate_1m" => "%{[events][rate_1m]}" }
            port => "8125"
            host => "<set at cfn-init>"
            namespace => "<set at cfn-init>"
            sender => "<set at cfn-init>"
        }
        statsd {
            id => "statsd_events_count"
            count => { "events.count" => "%{[events][count]}" }
            port => "8125"
            host => "<set at cfn-init>"
            namespace => "<set at cfn-init>"
            sender => "<set at cfn-init>"
        }
    }
}

and the errors we're seeing are as follows:
{"level":"WARN","loggerName":"logstash.outputs.elasticsearch","timeMillis":1499159071259,"thread":"Ruby-0-Thread-5: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:224","logEvent":{"message":"Attempted to resurrect connection to dead ES instance, but got an error.","url":{"metaClass":{"metaClass":{"metaClass":{"url":"http://logstash_system:xxxxxx@localhost:9200/","error_type":{"metaClass":{"metaClass":{"error_type":"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError","error":"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}}}}}}}}}
{"level":"INFO","loggerName":"logstash.outputs.elasticsearch","timeMillis":1499159076260,"thread":"Ruby-0-Thread-5: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:224","logEvent":{"message":"Running health check to see if an Elasticsearch connection is working","healthcheck_url":{"metaClass":{"metaClass":{"metaClass":{"healthcheck_url":"http://logstash_system:xxxxxx@localhost:9200/","path":"/"}}}}}}

Part of the issue is that it is trying to connect to elasticsearch on localhost:9200, which i don't understand because the output is configured with a specific host - <es_client_elb>:9200 - which is set to the aws elb at cfn-init, and as mentioned I am seeing logs forwarded from the logstash instance in elasticsearch - the template we're using creates a logstash-YYYY.MM.DD index.

4

Sorry it seems i'm not very smart - I missed this part of the documentation: https://www.elastic.co/guide/en/x-pack/current/monitoring-logstash.html#CO33-1

5

All good, thanks for sharing the solution too :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.