Logstash connection to elasticsearch

Elastic Stack Logstash
1

I'm using the official docker containers for elasticsearch and logstash and following for the most part the configuration in this website, but I'm having problem with logstash connecting to elasticsearch. The output of logstash is like this:

[2017-08-30T00:28:59,105][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@elasticsearch:9200/, :path=>"/"}
[2017-08-30T00:28:59,109][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@elasticsearch:9200/", :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}

I was looking in the discussions here and for the most part that is a a network issue, but is not in my case:

➜ ~ docker exec -it 6a0b92a7d911 curl -u elastic http://es:9200
Enter host password for user 'elastic':
{
"name" : "iSqmies",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "odHu_VZcRYK90ykh1ywJ-Q",
"version" : {
"number" : "5.5.2",
"build_hash" : "b2f0c09",
"build_date" : "2017-08-14T12:33:14.154Z",
"build_snapshot" : false,
"lucene_version" : "6.6.0"
},
"tagline" : "You Know, for Search"

The previous output is from the logstash container connecting to the elasticsearch.
The configuration of logstash is really basic:

input { stdin { } }
output {
elasticsearch {
hosts => [ "es:9200" ]
user => elastic
password => changeme
}
stdout { codec => rubydebug }
}

Absolutely no change in the config of elasticsearch.
Let me know if you have any idea of what is happening.

2

Why does the log talk about http://logstash_system:xxxxxx@elasticsearch:9200/ when your configuration indicates http://elastic:xxxxxx@es:9200/?

1 Like
3

logs are pointing to http://logstash_system:xxxxxx@elasticsearch:9200/
check your configuration file for your input plugin . for ex: if using filebeat there is filebeat.conf file in which output section is mentioned so comment out the elasticsearch section within it and uncomment the output.logtstash part and mention your logstash url.

4

Maybe I'm missing something here, I haven't got to that point yet, there is nothing sending information at this point the input is stdin I was trying just to establish configuration between both, the only configuration is:

input { stdin { } }
output {
elasticsearch {
hosts => [ “es:9200” ]
user => elastic
password => changeme
}
stdout { codec => rubydebug }
}

And I'm using this as a test to verify communication between the containers/services.
thanks.

5

I'm not entirely sure, I have used this services before but never with the official docker images you provide here, I know that there are some preconfigured options and I think one of those I haven't configured correctly and is creating this log.

6

First time I read your question I dont think I fully understood what you meant to say. So I started playing with two settings.

  1. The name of the elasticsearch container using: "es" or "elasticsearch"
  2. The host configuration on the config file, one more time using: "es" or "elasticsearch"

The combination of using the name "es" for the elasticsearch container and config file, throws the log error:
http://logstash_system:xxxxxx@elasticsearch:9200/
The combination of using the name "elasticsearch" for the container and but using "es" in the config file, the error is:
http://elastic:xxxxxx@es:9200
Only when I use "elasticsearch" on both the container name and the config file everything works just fine.
When I launch logstash I link the container to the elasticsearch running container.

7

I suspect it's the monitoring configuration in the logstash.yml found in the Docker image that's the culprit here.

2 Likes
8

Yes you are right that is the problem, this is the config in logstash.yml

http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline
xpack.monitoring.elasticsearch.url: http://elasticsearch:9200
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: changeme

So that configuration is hardcoded and if the hostname elasticsearch is not resolved that is going to throw and error.
I could copy a new logstash.yml to the correct path during the creation of the container, any other idea of how to deal with this. I would like to do ELK using containers only.
Thanks.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.