Connection refused logstash to elasticsearch

Hi,
I enable trial licence for my 3 node elasticsearch cluster, on docker. And now elasticsearch and kibana is running on https but logstash start giving problem. Earlier ELK was running on http correctly. My all setup elasticsearch cluster, kibana and logstash are on docker.

Now logstash is not making connection to elasticsearch cluster(I guess).
Logstash log is below:

Starting docker-elk-masterxpack_logstash_1 ... done
Attaching to docker-elk-masterxpack_logstash_1
logstash_1          | Using bundled JDK: /usr/share/logstash/jdk
logstash_1          | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash_1          | WARNING: An illegal reflective access operation has occurred
logstash_1          | WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-1/jruby15045012168019468027jopenssl.jar) to field java.security.MessageDigest.provider
logstash_1          | WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
logstash_1          | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
logstash_1          | WARNING: All illegal access operations will be denied in a future release
logstash_1          | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1          | [2021-02-02T11:56:00,557][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.10.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
logstash_1          | [2021-02-02T11:56:01,234][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
logstash_1          | [2021-02-02T11:56:01,237][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
logstash_1          | Please configure Metricbeat to monitor Logstash. Documentation can be found at:
logstash_1          | https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
logstash_1          | [2021-02-02T11:56:01,495][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1          | [2021-02-02T11:56:01,805][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
logstash_1          | [2021-02-02T11:56:01,930][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
logstash_1          | [2021-02-02T11:56:01,965][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash_1          | [2021-02-02T11:56:01,970][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
logstash_1          | [2021-02-02T11:56:01,997][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
logstash_1          | [2021-02-02T11:56:03,088][INFO ][org.reflections.Reflections] Reflections took 37 ms to scan 1 urls, producing 23 keys and 47 values
logstash_1          | [2021-02-02T11:56:03,228][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1          | [2021-02-02T11:56:03,418][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@localhost:9200/]}}
logstash_1          | [2021-02-02T11:56:03,497][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://elastic:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://elastic:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
logstash_1          | [2021-02-02T11:56:03,509][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://localhost:9200"]}
logstash_1          | [2021-02-02T11:56:03,563][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/usr/share/logstash/pipeline/pageactivity.conf"], :thread=>"#<Thread:0x6e86b391 run>"}
logstash_1          | [2021-02-02T11:56:04,265][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.7}
logstash_1          | [2021-02-02T11:56:04,376][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
logstash_1          | [2021-02-02T11:56:04,428][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

And Here is my docker compose file for running logstash:

 logstash:
    image: docker.elastic.co/logstash/logstash:${VERSION}
    volumes:
      - type: bind
        source: ./logstash/pipeline
        target: /usr/share/logstash/pipeline
        read_only: true
      - type: bind
        source: ./logstash/logstash-core/lib/jars/mssql-jdbc-8.4.1.jre8.jar
        target: /usr/share/logstash/logstash-core/lib/jars/mssql-jdbc-8.4.1.jre8.jar
      - certs:$CERTS_DIR
    ports:
      - "5044:5044"
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      SERVERNAME: localhost
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
      ELASTICSEARCH_URL: https://localhost:9200
      ELASTICSEARCH_HOSTS: https://localhost:9200
      ELASTICSEARCH_USERNAME: logstash_system
      ELASTICSEARCH_PASSWORD: XXXXXXXX
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: $CERTS_DIR/logstash/logstash.key
      SERVER_SSL_CERTIFICATE: $CERTS_DIR/logstash/logstash.crt
    networks:
      - elastic

And below is my pipeline file output part of logstash.conf (input part is working correctly for making a db connection)

output{
        elasticsearch {
                hosts => ["https://localhost:9200"]
                index => "myindex"
                user => "elastic"
                password => "XXXXXXXXX"
        }
        stdout { codec => rubydebug }
}

so you are trying to reach elasticsearch:9200 but in reality, you have
hosts => ["https://localhost:9200"]

can you actually resolve it? or do you have any manual DNS entry?

Hi Carmine,
Thanks your reply.
Actually did not find any settings for change [http://elasticsearch:9200/] to [https://localhost:9200/]. I not set [http://elasticsearch:9200/] in any location or file.

I checked logstash.conf, docker compose files.

do you have logstash + elasticsearch running on the same cluster? try to configure it via ip and see if the error changes

Could you post full docker file. We want to see all config if you need help.
But i think ES cluster dont listen on localhost

Hi,
Below is my complete docker file. It include 3 es node,1 kibana and logstash and enterprisearch.

version: '2.2'

services:

  es01:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es01
    environment:
      - node.name=es01
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es02,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=trial 
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true 
      - xpack.security.http.ssl.key=$CERTS_DIR/es01/es01.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.enabled=true 
      - xpack.security.transport.ssl.verification_mode=certificate 
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es01/es01.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es01/es01.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data01:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    ports:
      - 9200:9200
    networks:
      - elastic

    healthcheck:
      test: curl --cacert $CERTS_DIR/ca/ca.crt -s https://localhost:9200 >/dev/null; if [[ $$? == 52 ]]; then echo 0; else echo 1; fi
      interval: 30s
      timeout: 10s
      retries: 5

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es02
    environment:
      - node.name=es02
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es03
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es02/es02.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es02/es02.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es02/es02.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es02/es02.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data02:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    networks:
      - elastic

  es03:
    image: docker.elastic.co/elasticsearch/elasticsearch:${VERSION}
    container_name: es03
    environment:
      - node.name=es03
      - cluster.name=es-docker-cluster
      - discovery.seed_hosts=es01,es02
      - cluster.initial_master_nodes=es01,es02,es03
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - xpack.license.self_generated.type=trial
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=true
      - xpack.security.http.ssl.key=$CERTS_DIR/es03/es03.key
      - xpack.security.http.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.http.ssl.certificate=$CERTS_DIR/es03/es03.crt
      - xpack.security.transport.ssl.enabled=true
      - xpack.security.transport.ssl.verification_mode=certificate
      - xpack.security.transport.ssl.certificate_authorities=$CERTS_DIR/ca/ca.crt
      - xpack.security.transport.ssl.certificate=$CERTS_DIR/es03/es03.crt
      - xpack.security.transport.ssl.key=$CERTS_DIR/es03/es03.key
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - data03:/usr/share/elasticsearch/data
      - certs:$CERTS_DIR
    networks:
      - elastic

  enterprisesearch:
    image: docker.elastic.co/enterprise-search/enterprise-search:$ELK_VERSION
    container_name: es_enterprise
    environment:
      - cluster.name=es-docker-cluster
      - node.name=enterprisesearch
      - ent_search.auth.source=standard
      - elasticsearch.host=https://es01:9200
      - elasticsearch.username=elastic
      - elasticsearch.password=$ELASTIC_PASSWORD
      - allow_es_settings_modification=true
      - ent_search.external_url=https://myservername.com:3002
      - secret_management.encryption_keys=[a2cf0f124f130c3343fc137e8cc06f3617d20a470c06a91146f448e9b97fe0cq]
      - ENT_SEARCH_DEFAULT_PASSWORD=$ELASTIC_PASSWORD
      - elasticsearch.ssl.enabled=true
      - elasticsearch.ssl.verify=false
      - "JAVA_OPTS=-Xms2g -Xmx2g"
      - ent_search.ssl.enabled=true
      - ent_search.ssl.keystore.path=/usr/share/enterprisesearch/keystore.jks
      - ent_search.ssl.keystore.password=changeme
      - ent_search.ssl.keystore.key_password=changeme
    ports:
      - "3002:3002"
    volumes:
      - /home/elasticuser/mytest/docker-elk-masterxpack:/usr/share/enterprisesearch
    links:
      - es01
    depends_on:
      - es01
    networks:
      - elastic

  kib01:
    image: docker.elastic.co/kibana/kibana:${VERSION}
    container_name: kib01
    depends_on: {"es01": {"condition": "service_healthy"}}
    ports:
      - 5601:5601
    environment:
      SERVERNAME: localhost
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: https://es01:9200
      ELASTICSEARCH_USERNAME: kibana_system
      ELASTICSEARCH_PASSWORD: XXXXX
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: $CERTS_DIR/kib01/kib01.key
      SERVER_SSL_CERTIFICATE: $CERTS_DIR/kib01/kib01.crt
    volumes:
      - certs:$CERTS_DIR
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    networks:
      - elastic

  logstash:
    image: docker.elastic.co/logstash/logstash:${VERSION}
    volumes:
      - type: bind
        source: ./logstash/pipeline
        target: /usr/share/logstash/pipeline
        read_only: true
      - type: bind
        source: ./logstash/logstash-core/lib/jars/mssql-jdbc-8.4.1.jre8.jar
        target: /usr/share/logstash/logstash-core/lib/jars/mssql-jdbc-8.4.1.jre8.jar
      - certs:$CERTS_DIR
    ports:
      - "5044:5044"
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      SERVERNAME: localhost
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
      ELASTICSEARCH_URL: https://localhost:9200
      ELASTICSEARCH_HOSTS: https://localhost:9200
      ELASTICSEARCH_USERNAME: logstash_system
      ELASTICSEARCH_PASSWORD: XXXXX
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: $CERTS_DIR/logstash/logstash.key
      SERVER_SSL_CERTIFICATE: $CERTS_DIR/logstash/logstash.crt
    networks:
      - elastic


volumes:
  data01:
    driver: local
  data02:
    driver: local
  data03:
    driver: local
  certs:
    driver: local

networks:
  elastic:
    driver: bridge

As Carmine suggested use ip, I am using vm machine. IP address is not fix. So only server name ex myservername.com is possible to set.

Hi,
I am using VM machine and ip address is dynamic nature. Only server url is static. for example myserver.com

Indeed about the IP since you are running docker, but I did not get something, you are creating 3 ES , 1 kibana and 1 LS, and I can see kibana pointing at the correct endpoint
ELASTICSEARCH_URL: https://es01:9200
and LS to itself,be aware that you are not passing any configuration pipeline to LS, but you are defining the ELASTICSEARCH_URL, also this one one is wrong too cause it should not point to itself but to one of the ES or all of them like
ELASTICSEARCH_URL: ["https://es01:9200","https://es02:9200","https://es02:9200"], the

networks:
  - elastic

will resolve the name for you.

the configuration should be under /opt/logstash/config/logstash.yml

have a look there.

Hi Carmine,
Earlier I tried in that way also. Now I am change again my configuration for LS in that way.
Below is my updated docker file for logstash:

 logstash:
    build:
      context: logstash/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./logstash/pipeline
        target: /usr/share/logstash/pipeline
        read_only: true
      - type: bind
        source: ./logstash/config/logstash.yml
        target: /usr/share/logstash/config/logstash.yml
        read_only: true
      - type: bind
        source: ./logstash/logstash-core/lib/jars/mssql-jdbc-8.4.1.jre8.jar
        target: /usr/share/logstash/logstash-core/lib/jars/mssql-jdbc-8.4.1.jre8.jar
      - certs:$CERTS_DIR
    ports:
      - "5044:5044"
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      SERVERNAME: localhost
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
      ELASTICSEARCH_URL: https://es01:9200
      ELASTICSEARCH_HOSTS: https://es01:9200
      ELASTICSEARCH_USERNAME: logstash_system
      ELASTICSEARCH_PASSWORD: XXXXXXXXXXXXX
      ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES: $CERTS_DIR/ca/ca.crt
      SERVER_SSL_ENABLED: "true"
      SERVER_SSL_KEY: $CERTS_DIR/logstash/logstash.key
      SERVER_SSL_CERTIFICATE: $CERTS_DIR/logstash/logstash.crt
    networks:
      - elastic

Above I define a logstash.yml file ./logstash/config/logstash.yml

And logstash.yml is:

xpack.monitoring.elasticsearch.hosts: [ "https://es01:9200" ]

## X-Pack security credentials
#
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: XXXXXXXX

Now error is:

Attaching to docker-elk-masterxpack_logstash_1
logstash_1          | Using bundled JDK: /usr/share/logstash/jdk
logstash_1          | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash_1          | WARNING: An illegal reflective access operation has occurred
logstash_1          | WARNING: Illegal reflective access by org.jruby.ext.openssl.SecurityHelper (file:/tmp/jruby-1/jruby13847575150117556846jopenssl.jar) to field java.security.MessageDigest.provider
logstash_1          | WARNING: Please consider reporting this to the maintainers of org.jruby.ext.openssl.SecurityHelper
logstash_1          | WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
logstash_1          | WARNING: All illegal access operations will be denied in a future release
logstash_1          | Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
logstash_1          | [2021-02-03T08:10:21,218][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.10.1", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
logstash_1          | [2021-02-03T08:10:21,930][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and targeted for removal in the next major version.
logstash_1          | Please configure Metricbeat to monitor Logstash. Documentation can be found at:
logstash_1          | https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html
logstash_1          | [2021-02-03T08:10:22,182][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
logstash_1          | [2021-02-03T08:10:22,495][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_system:xxxxxx@es01:9200/]}}
logstash_1          | [2021-02-03T08:10:22,789][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://logstash_system:xxxxxx@es01:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@es01:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
logstash_1          | [2021-02-03T08:10:22,878][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [https://logstash_system:xxxxxx@es01:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target {:url=>https://logstash_system:xxxxxx@es01:9200/, :error_message=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@es01:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
logstash_1          | [2021-02-03T08:10:22,885][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [https://logstash_system:xxxxxx@es01:9200/][Manticore::ClientProtocolException] PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"}
logstash_1          | [2021-02-03T08:10:22,913][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
logstash_1          | [2021-02-03T08:10:23,988][INFO ][org.reflections.Reflections] Reflections took 39 ms to scan 1 urls, producing 23 keys and 47 values
logstash_1          | [2021-02-03T08:10:24,146][ERROR][logstash.outputs.elasticsearch] Unknown setting 'username' for elasticsearch
logstash_1          | [2021-02-03T08:10:24,152][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:83)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}
logstash_1          | warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
logstash_1          | LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
logstash_1          |           create at org/logstash/execution/ConvergeResultExt.java:129
logstash_1          |              add at org/logstash/execution/ConvergeResultExt.java:57
logstash_1          |   converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:378
logstash_1          | [2021-02-03T08:10:24,159][ERROR][logstash.agent           ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
logstash_1          | [2021-02-03T08:10:24,176][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in `create'", "org/logstash/execution/ConvergeResultExt.java:57:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:378:in `block in converge_state'"]}
logstash_1          | [2021-02-03T08:10:24,192][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
logstash_1          | warning: thread "Api Webserver" terminated with exception (report_on_exception is true):
logstash_1          | NameError: uninitialized constant Rack::Builder
logstash_1          |     const_missing at org/jruby/RubyModule.java:3760
logstash_1          |               app at /usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:97
logstash_1          |   start_webserver at /usr/share/logstash/logstash-core/lib/logstash/webserver.rb:99
logstash_1          |               run at /usr/share/logstash/logstash-core/lib/logstash/webserver.rb:60
logstash_1          |              each at org/jruby/RubyRange.java:526
logstash_1          |   each_with_index at org/jruby/RubyEnumerable.java:1258
logstash_1          |               run at /usr/share/logstash/logstash-core/lib/logstash/webserver.rb:55
logstash_1          |   start_webserver at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:424
logstash_1          | Exception in thread "Api Webserver" java.lang.NullPointerException
logstash_1          |   at org.jruby.internal.runtime.ThreadService.getMainThread(ThreadService.java:233)
logstash_1          |   at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1822)
logstash_1          |   at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112)
logstash_1          |   at java.base/java.lang.Thread.run(Thread.java:834)
docker-elk-masterxpack_logstash_1 exited with code 1

Earlier I tried with elastic(superuser) now am trying logstash_system user.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.