Elastic Logstash Helm Chart Doesn't Pass Readiness Probe

I have multiple logstash pods deployed, segregated by function. I am experiencing an issue where, when I have xpack.monitoring configured, the helm chart readiness and liveness probes don't ever show that the pod is ready and running. This is reported on the Helm chart bug tracker in this issue.

The workaround recommended in the bug tracker worked for the logstash container which I hadn't configured already (i.e., it was running using the default logstash.conf file), but not the other, which I configured to listen on port https/5044. I'm using the identical logstash.yml config for each. Logstash will self-report in the log that it has started and is listening on port 5044, but since the readiness probe never passes, it continues to restart every 5 minutes.

Config for the one with the custom logstash.conf is below:

logstash.yml

logstashConfig:
  logstash.yml: |
    http:
      host: "0.0.0.0"
    xpack:
      monitoring:
        enabled: true
        elasticsearch:
          hosts:
            - https://es-master-headless:9200
          username: "${LOGSTASH_USERNAME}"
          password: "${LOGSTASH_PASSWORD}"
          ssl:
            certificate_authority: "/etc/ssl/elk/ca_bundle.pem"
            verification_mode: "none"

logstash.conf

input {
  beats {
    port => 5044
    ssl => true
    ssl_certificate => "/etc/ssl/elk/cert.pem"
    ssl_key => "/etc/ssl/elk/key.pem"
    ssl_certificate_authorities => [
      "/etc/ssl/elk/ca_root.pem",
      "/etc/ssl/elk/ca_int.pem"
    ]
  }
}
  
output {
  elasticsearch {
    # Configure host.
    hosts => [
      "https://es-master-headless:9200"
    ]
    ssl => true
    ssl_certificate_verification => false
    user => "${LOGSTASH_USERNAME}"
    password => "${LOGSTASH_PASSWORD}"

    # Configure index.
    index => "%{[@metadata][beat]}-%{[@metadata][version]}"

    # Configure ILM.
    ilm_enabled => "false"
    # ilm_rollover_alias => "%{[@metadata][beat]}-%{[@metadata][version]}"
    # ilm_pattern => "-000001"
    # ilm_policy => "default-ilm-policy"
  }
}

Logs:

[2020-02-11T17:44:35,805][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2020-02-11T17:44:35,887][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>"main"}
[2020-02-11T17:44:36,082][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2020-02-11T17:44:36,125][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-02-11T17:44:37,395][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch bulk_path=>"/_monitoring/bulk?system_id=logstash&system_api_version=7&interval=1s", ssl_certificate_verification=>false, password=><password>, hosts=>[https://elk-es-master-headless:9200], cacert=>"/etc/ssl/elk/ca_bundle.pem", sniffing=>false, manage_template=>false, id=>"0a04bae9ce401121f243f2697e0b110df46e368877602b46959c0a78d1bb5f13", user=>"logstash_system", ssl=>true, document_type=>"%{[@metadata][document_type]}", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_24f36975-76e5-4efc-81bc-dfc6f0b48f29", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>"auto", ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2020-02-11T17:44:37,409][WARN ][logstash.outputs.elasticsearch] ** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true
[2020-02-11T17:44:37,480][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://logstash_system:xxxxxx@elk-es-master-headless:9200/]}}
[2020-02-11T17:44:37,503][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"https://logstash_system:xxxxxx@elk-es-master-headless:9200/"}
[2020-02-11T17:44:37,507][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>7}
[2020-02-11T17:44:37,508][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-02-11T17:44:37,530][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://elk-es-master-headless:9200"]}
[2020-02-11T17:44:37,534][INFO ][logstash.javapipeline    ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x308d625b run>"}
[2020-02-11T17:44:37,593][INFO ][logstash.javapipeline    ] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2020-02-11T17:44:37,601][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
[2020-02-11T17:44:37,999][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Resolved the issue. While troubleshooting the issue, I updated httpGet.host=127.0.0.1 in the logstash containers that continued to fail. Once I removed this, it passed.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.