Tough config problem: logstash -> hosted elastic stack, using TLS

[marc-at-qventus (Marc Slayton)]

Hi all --

I'm having a tough configuration problem with I'm hoping the community can help with.

First, the scenario:
= I have a logstash instance (7.6.0) running in a docker container.
= I am using the elastic hosted service.
= I want to confiugre the logstash instance to connect and send metrics to the elastic hosted cluster using TLS.

I've looked through the docs and this seems like a pretty difficult scenario to get working properly. Here is what I have so far:

1.) I created a cert which I've verified can connect to the elastic cluster using TLS:

curl --cacert ./elastic-ca-cert.pem -u elastic:XXXYYXXXZZ https://XXXXXXXX.XXX.aws.found.io:XXXX/_cat/nodes?v

OUTPUT:
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
X.X.X.X 60 99 34 9.19 9.99 10.46 dim * instance-0000000004
X.X.X.X 9 99 9 3.02 2.99 2.91 dim - instance-0000000005

2.) Here's my logstash.yml which I've verified is being parsed within the container.

http.host: "XXXXXXXX.XXXXXX.aws.found.io"
http.port: XXXX
log.level: debug
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: ["https://XXXXXXXX.XXX.aws.found.io:XXXX"]
xpack.monitoring.elasticsearch.hosts: ["https://XXXXXXXX.XXX.aws.found.io:XXXX"]
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: XXXXXX
xpack.monitoring.elasticsearch.ssl.ca: /etc/logstash/conf.d/elastic-ca-cert.pem
xpack.monitoring.elasticsearch.ssl.keystore.path: /etc/pki/ca-trust/extracted/java/cacerts
xpack.monitoring.elasticsearch.ssl.keystore.password: XXXXXX
path.config: /usr/share/logstash/pipeline

3.) My logstash.conf is as minimal as possible, until things are working:

input {
  beats {
    port => 5044
  }
}

output {
    elasticsearch {
        hosts => [ "XXXXXX.XXXXX.aws.found.io:XXXX" ]
        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
        ssl => true
        user => 'elastic'
        password => 'XXXXXXX'
        cacert => '/etc/logstash/conf.d/elastic-ca-cert.pem'
    }
}

THE PROBLEM:
When the docker container runs, it logs the following error, over and over, which
I've read on these boards is related to monitoring:

[2020-02-16T16:59:20,625][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}

[2020-02-16T16:59:21,160][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. 
{:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}

I've read about adding the ca cert to the java/cacerts keystore
and configuring the keystore password. I've tried it, and verified
the ca was added properly. This has no effect.

I've also tried the workarounds from the docs using the trust store
config parameters. No effect.

I've tried generating various other ca certs in hope I've missed something
in the ca generation phase. No effect.

At this point, I'm stumped -- so I'm asking the community for an assist.

There appear to be a lot of moving parts at play here, and I'm wondering
if maybe Elastic's hosted services just aren't configured to support TLS.
Is there any way to check? It's a bit harder having only limited web access
to the elasticsearch side of the config.

Kudos to anyone who can provide assistance. Cheers --

[marc-at-qventus (Marc Slayton)]

Hey all --

As a follow up, I figured out what I was missing.

My logstash.yml was under /etc/logstash/conf.d/logstash.yml, where
it was not being noticed. This may be a bug in the logstash container,
or possibly the container docs.

I changed my Dockerfile to copy to /usr/share/logstash/config/logstash.yml,
and was able to finish configuring my cluster.

Cheers --

Cheers --