Logstash elasticsearch output plugin hangs at health check

[addyk]

I am using a Logstash ver 5.2 docker image and outputting to Elastic Cloud with ver 5.1.2. The appropriate environment variables have been set to correspond with my output.

output {
  stdout { codec => rubydebug }

  elasticsearch {
    hosts => ["${ELASTICSEARCH_HOST}"]
    user => "${ELASTIC_CLOUD_USER}"
    password => "${ELASTIC_CLOUD_PASSWORD}"
    index => "%{level}-%{+YYYY.MM.dd}"
    document_type => "logs"
  }
}

If I comment out the elasticsearch output, then I am able to see stdout. Stdout indicates that my filters and input are coming in fine. However, if I leave the elasticsearch plugin enabled, then I get the following logs:

 Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
 19:21:31.746 [main] INFO  logstash.setting.writabledirectory - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
 19:21:31.796 [LogStash::Runner] INFO  logstash.agent - No persistent UUID file found. Generating new UUID {:uuid=>"3715cecd-0db2-4a64-ac9d-c0c479fb085c", :path=>"/var/lib/logstash/uuid"}
 19:21:36.700 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://username:password@b40c2d46d42bf8d09ffce40c191409fc.us-east-1.aws.found.io:9243/]}}
 19:21:36.719 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://username:password@b40c2d46d42bf8d09ffce40c191409fc.us-east-1.aws.found.io:9243/, :path=>"/"}

On some days, I will actually see stdout with elasticsearch plugin enabled, but on other days without any change, I will get no other output from the "Running health check" line.

Edit: Removed my filters temporarily to confirm that it wasn't anything going on the filters end.

[addyk]

Also want to point out that these two users are having similar issue (not being able to send data from logstash to elastic)

Logstash not sending data to elastic from kafka

Hi, im trying to add data in elastic with logstash and data is input from kafka this is the simple config im using on logstash server in conf file /etc/logstash/conf.d/1-config.conf input{ kafka{ bootstrap_servers => "dummykafka:9092" topics => ["dummy.kafka.topic"] } } output{ elasticsearch{ index => "test.winlogbeat.eventlogs-%{+YYYY.MM.dd}" hosts => ["elasticdatanode:50000"] user => "AdminID" password => "dummypassword" } } i restart the logstash server with this command sud…

I cant put logs in elasticsearch from logstash

Im using logstash 1.4.2 and ... I have the next logstash.conf input { udp { port => 9991 codec => netflow { definitions => "/home/profucom/logstash-1.4.2/lib/logstash/codecs/netflow/netflow.yaml" versions => [5] } } } output { stdout { codec => rubydebug } elasticsearch { index => logstash-%{+YYYY.MM.dd} host => "localhost" } } When i run "./logstash -f logstash.conf works fine, i get all information from my 2 routers, but im not sure if index => logstash-%{+Y…

Edit: To clarify, I do not have the password error like that mentioned in Pipeline aborted due to error in password field logstash

[addyk]

I was able to fix my issue by creating a logstash.yml with the following configuration:

node:
  name: ${HOSTNAME}
xpack:
  monitoring:
    enabled: false
    elasticsearch:
      url: ${ELASTICSEARCH_HOST}
      username: ${ELASTIC_CLOUD_USER}
      password: ${ELASTIC_CLOUD_PASSWORD}

I also upgraded to use docker.elastic.co logstash 5.2.2 image.