Pb using logstash Helm chart and X-Pack

Elastic Stack Logstash
1

Hi,
I'am trying to deploy Logstash on my Kubernetes cluster using this helm chart :

I just modified the values.yaml to have the input and output I need like this :

inputs:
  main: |-
    input {
      kinesis {
        kinesis_stream_name => "xxxxx"
        region => "eu-west-1"
        codec => json { }
      }
    }

filters:
  # main: |-
  #   filter {
  #   }

outputs:
  main: |-
    output {
      s3 { 
        access_key_id => "xxxxx"
        secret_access_key => "xxxxx"
        region => "eu-west-1"
        bucket => "xxxxx"
        time_file => 1
      }
    }

But when I deploy the chart I have this errors in logs :

2019-11-29T15:12:50.901803449Z OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
2019-11-29T15:12:52.53499802Z WARNING: An illegal reflective access operation has occurred
2019-11-29T15:12:52.535033503Z WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.8.0.jar) to field java.io.FileDescriptor.fd
2019-11-29T15:12:52.535040078Z WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
2019-11-29T15:12:52.535054769Z WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
2019-11-29T15:12:52.535059405Z WARNING: All illegal access operations will be denied in a future release
2019-11-29T15:13:04.577349287Z Thread.exclusive is deprecated, use Thread::Mutex
2019-11-29T15:13:06.78326088Z Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
2019-11-29T15:13:07.28765579Z [2019-11-29T15:13:07,284][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
2019-11-29T15:13:07.296801742Z [2019-11-29T15:13:07,296][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.4.2"}
2019-11-29T15:13:07.822708794Z [2019-11-29T15:13:07,822][WARN ][logstash.monitoringextension.pipelineregisterhook] xpack.monitoring.enabled has not been defined, but found elasticsearch configuration. Please explicitly set `xpack.monitoring.enabled: true` in logstash.yml
2019-11-29T15:13:08.495331757Z [2019-11-29T15:13:08,494][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://elasticsearch:9200/]}}
2019-11-29T15:13:08.631470713Z [2019-11-29T15:13:08,631][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}
2019-11-29T15:13:08.664207662Z [2019-11-29T15:13:08,663][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
2019-11-29T15:13:08.671112458Z [2019-11-29T15:13:08,670][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch"}
2019-11-29T15:13:08.709039692Z [2019-11-29T15:13:08,708][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
2019-11-29T15:13:09.798357046Z [2019-11-29T15:13:09,797][INFO ][org.reflections.Reflections] Reflections took 28 ms to scan 1 urls, producing 20 keys and 40 values

Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.

I don't understand why I need an elasticsearch cluster. I just want to read kinesis and push to S3.
I don't need X-Pack too. Is there a way to deactivate elastic and/or xpack with this chart ?

Another thing I don't understand is that just before to try to push to S3, I just test to push to stdout (still reading from kinesis, just to test), and it worked fine. Since I tried to output to S3 I have those errors.

Thank you for your help

2

Hi @cabrinoob,
From what I quickly see the chart you are using have a default configuration for elasticsearch here. When an Elasticsearch is configured Logstash is enabling Logstash X-Pack monitoring. To disable it you can remove elasticsearch configuration from your values if you don't need it or enforce xpack.monitoring.enabled=false.

Please note also that helm/charts/stable/logstash is not maintained by Elastic, we are just releasing elastic/helm-charts/logstash which will be available on Elastic Helm repo and Helm Hub pretty soon.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.