How to make Logstash talk to Elasticsearch?

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I have Elasticsearch running in K8s via the operator, and want to talk to it from Logstash.

Everything I've tried so far has failed, mostly resulting in

[2020-04-01T15:25:22,773][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch-es-http:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch-es-http:9200/][Manticore::ClientProtocolException] elasticsearch-es-http:9200 failed to respond"}

I don't want to worry about TLS just yet - I want to make it work first, and worry about security later. I have found hints that it's necessary to turn off TLS via

spec:
  http:
    tls:
      selfSignedCertificate:
        disabled: true

but that doesn't make any difference.

2

Ah, finger trouble, it turns out that that does make a difference - I'm now getting 401 which suggests I just need to find a username and password.

1 Like
3

@TimWardFS did you manage to sort things out?

Regarding TLS, you can also keep TLS enabled and either:

  • configure Logstash in such a way that it ignores server certificates validation
  • mount the TLS certificates secret of Elasticsearch into the Logstash Pod, and configure it to trust those certificates
4

I have, for now,

(1) turned off TLS (disabling selfSignedCertificate did work once I'd got it right)
(2) created the secret for the password for the "elastic" user manually so as not to get a generated one - would be nice to have this as a documented feature rather than something that is known to work just right now but not guaranteed to remain working in future versions.