Pb using logstash Helm chart and X-Pack

Hi,
I'am trying to deploy Logstash on my Kubernetes cluster using this helm chart :

https://hub.kubeapps.com/charts/stable/logstash

I just modified the values.yaml to have the input and output I need like this :

inputs:
  main: |-
    input {
      kinesis {
        kinesis_stream_name => "xxxxx"
        region => "eu-west-1"
        codec => json { }
      }
    }

filters:
  # main: |-
  #   filter {
  #   }

outputs:
  main: |-
    output {
      s3 { 
        access_key_id => "xxxxx"
        secret_access_key => "xxxxx"
        region => "eu-west-1"
        bucket => "xxxxx"
        time_file => 1
      }
    }

But when I deploy the chart I have this errors in logs :

2019-11-29T15:12:50.901803449Z OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2019-11-29T15:12:52.53499802Z WARNING: An illegal reflective access operation has occurred
2019-11-29T15:12:52.535033503Z WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd
2019-11-29T15:12:52.535040078Z WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
2019-11-29T15:12:52.535054769Z WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
2019-11-29T15:12:52.535059405Z WARNING: All illegal access operations will be denied in a future release
2019-11-29T15:13:04.577349287Z Thread.exclusive is deprecated, use Thread::Mutex
2019-11-29T15:13:06.78326088Z Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
2019-11-29T15:13:07.28765579Z [2019-11-29T15:13:07,284][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
2019-11-29T15:13:07.296801742Z [2019-11-29T15:13:07,296][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.4.2"}
2019-11-29T15:13:07.822708794Z [2019-11-29T15:13:07,822][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
2019-11-29T15:13:08.495331757Z [2019-11-29T15:13:08,494][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2019-11-29T15:13:08.631470713Z [2019-11-29T15:13:08,631][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
2019-11-29T15:13:08.664207662Z [2019-11-29T15:13:08,663][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
2019-11-29T15:13:08.671112458Z [2019-11-29T15:13:08,670][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
2019-11-29T15:13:08.709039692Z [2019-11-29T15:13:08,708][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
2019-11-29T15:13:09.798357046Z [2019-11-29T15:13:09,797][INFO ][org.reflections.Reflections] Reflections took 28 ms to scan 1 urls, producing 20 keys and 40 values

Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.

I don't understand why I need an elasticsearch cluster. I just want to read kinesis and push to S3.
I don't need X-Pack too. Is there a way to deactivate elastic and/or xpack with this chart ?

Another thing I don't understand is that just before to try to push to S3, I just test to push to stdout (still reading from kinesis, just to test), and it worked fine. Since I tried to output to S3 I have those errors.

Thank you for your help

Hi @cabrinoob,
From what I quickly see the chart you are using have a default configuration for elasticsearch here. When an Elasticsearch is configured Logstash is enabling Logstash X-Pack monitoring. To disable it you can remove elasticsearch configuration from your values if you don't need it or enforce xpack.monitoring.enabled=false.

Please note also that helm/charts/stable/logstash is not maintained by Elastic, we are just releasing elastic/helm-charts/logstash which will be available on Elastic Helm repo and Helm Hub pretty soon.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.