Logstash error connecting to ElasticSearch

Elastic Stack Logstash
1

I've tried to created sidecar using beats and logstash on OpenShift. However the logstash always attempted to resurrect connection to dead ES instance (to http://elastisearch:9200), despite configs that I've done.

Logstash configuration:

access-log.conf: |
    input {
      beats {
        port => "5044"
      }
    }
    output {
      elasticsearch {
        hosts => ["http://10.49.141.225:9200"]
      }
    }

Log:

Using bundled JDK: /usr/share/logstash/jdk
2Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
3[2023-05-11T08:36:26,553][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
4[2023-05-11T08:36:26,556][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.7.1", "jruby.version"=>"jruby 9.3.10.0 (2.6.8) 2023-02-01 107b2e6697 OpenJDK 64-Bit Server VM 17.0.7+7 on 17.0.7+7 +indy +jit [x86_64-linux]"}
5[2023-05-11T08:36:26,558][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
6[2023-05-11T08:36:26,562][INFO ][logstash.settings ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
7[2023-05-11T08:36:26,563][INFO ][logstash.settings ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
8[2023-05-11T08:36:26,684][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"283eab08-c6d2-44bb-baed-bffbca3da974", :path=>"/usr/share/logstash/data/uuid"}
9[2023-05-11T08:36:26,976][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
10[2023-05-11T08:36:26,977][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
11Please configure Metricbeat to monitor Logstash. Documentation can be found at:
12https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
13[2023-05-11T08:36:27,142][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
14[2023-05-11T08:36:27,173][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch: Name or service not known", :exception=>Manticore::ResolutionFailure, :cause=>#<Java::JavaNet::UnknownHostException: elasticsearch: Name or service not known>}
15[2023-05-11T08:36:27,174][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
16[2023-05-11T08:36:27,179][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"elasticsearch", :exception=>Manticore::ResolutionFailure, :cause=>#<Java::JavaNet::UnknownHostException: elasticsearch>}
17[2023-05-11T08:36:27,180][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
18[2023-05-11T08:36:27,181][WARN ][logstash.licensechecker.licensereader] Attempt to validate Elasticsearch license failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Elasticsearch Unreachable: [http://elasticsearch:9200/_xpack][Manticore::ResolutionFailure] elasticsearch"}

I've checked the elastic already running:

elastic
elastic1579×200 31.5 KB

2

You are getting the error from the licensechecker, not the elasticsearch output. I think the address this connects to is controlled by xpack.monitoring.elasticsearch.hosts in logstash.yml.

3

update your output section as below, make sure you have /etc/hosts entries if you mention hostname of machines.

output
{
stdout { codec => rubydebug { metadata => true } }
elasticsearch {
hosts => [ "es-prod-01", "es-prod-02", "es-prod-03" ]
index => "filebeats-lsf-%{+YYYY.MM.dd}"
}
}

4

Per what @Badger said, since this is the License Checker, you could also set xpack.monitoring.enabled: false in logstah.yml.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.