Hi @leandrojmp - Thanks for you reply. Below are the complete details.
Deployed Logstash and Elasticsearch- version 8.5.1 in Azure Kubernetes Cluster using helm deployment. Logstash and elasticsearch are deployed in two different cluster, exposing elasticsearch through ingress with trusted certificate and using it in logstash output.
Elastic search is reachable from logstash pod.
curl -v -u"username:password" https://elasticsearch...*
Logstash Config File:
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => "https://elasticsearch...:443"
index => "log4j-%{+YYYY.MM.dd}"
user => ""
password => "*"
}
stdout {
}
}
Logstash Yaml:
pipeline.ordered: auto
path.config: /usr/share/logstash/config/logstash.conf
Now Enabled Info logs only:
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2023-11-19T06:40:09,168][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2023-11-19T06:40:09,238][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.5.1", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.5+8 on 17.0.5+8 +indy +jit [x86_64-linux]"}
[2023-11-19T06:40:09,242][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Xmx1g, -Xms1g, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-11-19T06:40:09,335][INFO ][logstash.settings ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2023-11-19T06:40:09,353][INFO ][logstash.settings ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2023-11-19T06:40:09,948][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-11-19T06:40:10,030][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"957188bc-2005-4a03-8e27-8f125f2ea219", :path=>"/usr/share/logstash/data/uuid"}
[2023-11-19T06:40:13,553][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-11-19T06:40:14,742][INFO ][org.reflections.Reflections] Reflections took 291 ms to scan 1 urls, producing 125 keys and 438 values
[2023-11-19T06:40:16,551][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
[2023-11-19T06:40:16,751][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["https://elasticsearch...:443"]}
[2023-11-19T06:40:17,331][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@elasticsearch...:443/]}}
[2023-11-19T06:40:18,153][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@elasticsearch...*.:443/"}
[2023-11-19T06:40:18,258][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.5.1) {:es_version=>8}
[2023-11-19T06:40:18,262][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2023-11-19T06:40:18,433][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
[2023-11-19T06:40:18,435][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
[2023-11-19T06:40:18,437][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with ecs_compatibility => v8, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2023-11-19T06:40:18,649][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2023-11-19T06:40:18,649][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/usr/share/logstash/config/logstash.conf"], :thread=>"#<Thread:0x17e7d7be run>"}
[2023-11-19T06:40:20,645][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.99}
[2023-11-19T06:40:20,747][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2023-11-19T06:40:20,766][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-11-19T06:40:21,130][INFO ][org.logstash.beats.Server][main][9961797f7d39d1477f01186771a008c1841f9f4e219707d4f538a7c57dc66530] Starting server on port: 5044
[2023-11-19T06:40:21,127][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2023-11-19T06:44:56,903][WARN ][logstash.runner ] SIGTERM received. Shutting down.
[2023-11-19T06:45:03,568][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-11-19T06:45:03,986][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2023-11-19T06:45:04,126][INFO ][logstash.runner ] Logstash shut down.
Once the pipeline is started and running logstash pod is not in ready status and keeps restarting with any other log message.