How to authenticate Logstash output to a secure Elasticsearch URL (version 5.6.5)

[loganathan001 (Loganathan Sekaran)]

I am using Logstash 5.6.5. So far used elasticsearch (version 5.6.5) output with HTTP protocol and no authentication. Now Elasticsearch is being secured using basic authentication (user/password) and CA certified HTTPS URL. I don't have any control over the elasticsearch server. I just use it to output to from Logstash.

Now when I try to configure the HTTPS URL of elasticsearch with basic authentication, it fails to create the pipeline.

Output Configuration

output { 
 elasticsearch {
   hosts => ["https://myeslasticsearch.server.io"]
   user => "esusername"
   password => "espassword"
   ssl => true
 }
}

Errors

 1. Error registering plugin {:plugin=>"#<LogStash::OutputDelegator:0x50aa9200
 2. Pipeline aborted due to error {:exception=>#<URI::InvalidComponentError: bad component(expected user component):

How to fix this? I notice that there is a field called cacert which requires some PEM file. But I am not sure what to put there since the Elasticsearch server is using a CA certified SSL not a self-signed one.

Addtional question: I don't have any xpack installed. Is 'xpack' required to be purchased for HTTPS output to Elasticsearch from Logstash?

[tag_v]

try making ssl_certification_verification false as default value is true.

 elasticsearch {
             hosts => ["es:9200"]
              user => "xxxx"
              password => "xxxx"
              ssl => true
              ssl_certificate_verification => false
              index => "indexname"     
        }

if you are using any truststore certs mention paths in output:

         elasticsearch {
                 hosts => ["<es>:9200"]
                  user => "xxxx"
                  password => "xxxx"
                  ssl => true
                  ssl_certificate_verification => false
                  truststore => "<path>"
                  truststore_password => "<password>"
                 index => "<indexname>"     
            }

[loganathan001 (Loganathan Sekaran)]

I found the root cause of the issue. There were three things to fix:

1. The logstash version I tested with was wrong 5.5.0. I downloaded the correct version to match with Elasticsearch Version 5.6.5.

2. The host I used was running on 443 port. When I didn't specify the port as below logstash appends 9200 with it.
   hosts => ['https://my.es.server.com']
   Below configuration corrected the port used by logstash.
   hosts => ['https://my.es.server.com:443']

3. I was missing proxy connection settings.
   proxy => 'http://my.proxy.com:80'

Overall settings that worked.

output {
	elasticsearch {
	   hosts => ['https://my.es.server.com:443']
	   user => 'esusername'
	   password => 'espassword'
	   proxy => 'http://my.proxy:80'
	   index => "my-index-%{+YYYY.MM.dd}"
	}
}

No need for 'ssl' field.

Also NO need for 'xpack' installation for this requirement.