[2023-11-18T14:35:04,262][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2023-11-18T14:35:04,906][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[2023-11-18T14:35:05,409][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"Copy"}
[2023-11-18T14:35:05,409][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"MarkSweepCompact"}
[2023-11-18T14:35:05,948][WARN ][logstash.runner ] SIGTERM received. Shutting down.
[2023-11-18T14:35:05,961][DEBUG][logstash.agent ] Shutting down all pipelines {:pipelines_count=>1}
[2023-11-18T14:35:05,966][DEBUG][logstash.agent ] Converging pipelines state {:actions_count=>1}
[2023-11-18T14:35:05,970][DEBUG][logstash.agent ] Executing action {:action=>LogStash::PipelineAction::StopAndDelete/pipeline_id:main}
[2023-11-18T14:35:05,980][DEBUG][logstash.javapipeline ] Closing inputs {:pipeline_id=>"main", :thread=>"#<Thread:0x530fc581 sleep>"}
[2023-11-18T14:35:05,982][DEBUG][logstash.inputs.beats ] Stopping {:plugin=>"LogStash::Inputs::Beats"}
[2023-11-18T14:35:05,984][DEBUG][org.logstash.beats.Server] Server shutting down
[2023-11-18T14:35:09,261][DEBUG][org.logstash.execution.PeriodicFlush][main] Pushing flush onto pipeline.
[2023-11-18T14:35:09,909][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
[2023-11-18T14:35:10,415][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"Copy"}
[2023-11-18T14:35:10,416][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"MarkSweepCompact"}
[2023-11-18T14:35:12,002][DEBUG][org.logstash.beats.Server] Server stopped
[2023-11-18T14:35:12,003][DEBUG][logstash.javapipeline ] Closed inputs {:pipeline_id=>"main", :thread=>"#<Thread:0x530fc581 sleep>"}
[2023-11-18T14:35:12,009][DEBUG][logstash.inputs.beats ][main][5a3347c8db8dab80d186f6d7f875ed4b259d3ecf694887b0d0f02217b150b469] Closing {:plugin=>"LogStash::Inputs::Beats"}
[2023-11-18T14:35:12,013][DEBUG][logstash.pluginmetadata ][main][5a3347c8db8dab80d186f6d7f875ed4b259d3ecf694887b0d0f02217b150b469] Removing metadata for plugin 5a3347c8db8dab80d186f6d7f875ed4b259d3ecf694887b0d0f02217b150b469
[2023-11-18T14:35:12,017][DEBUG][logstash.javapipeline ][main] Input plugins stopped! Will shutdown filter/output workers. {:pipeline_id=>"main", :thread=>"#<Thread:0x530fc581 run>"}
[2023-11-18T14:35:12,023][DEBUG][logstash.javapipeline ][main] Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#<LogStash::WorkerLoopThread:0x21a3db4 run>"}
[2023-11-18T14:35:12,110][DEBUG][logstash.outputs.elasticsearch][main] Closing {:plugin=>"LogStash::Outputs::Elasticsearch"}
[2023-11-18T14:35:12,125][DEBUG][logstash.outputs.elasticsearch][main] Stopping sniffer
[2023-11-18T14:35:12,128][DEBUG][logstash.outputs.elasticsearch][main] Stopping resurrectionist
[2023-11-18T14:35:12,829][DEBUG][logstash.outputs.elasticsearch][main] Waiting for in use manticore connections
[2023-11-18T14:35:12,837][DEBUG][logstash.outputs.elasticsearch][main] Closing adapter #LogStash::Outputs::ElasticSearch::HttpClient::ManticoreAdapter:0x11132085
[2023-11-18T14:35:12,849][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager][main] Connection manager is shutting down
[2023-11-18T14:35:12,849][DEBUG][org.apache.http.impl.conn.DefaultManagedHttpClientConnection][main] http-outgoing-0: Close connection
[2023-11-18T14:35:12,852][DEBUG][org.apache.http.impl.conn.PoolingHttpClientConnectionManager][main] Connection manager shut down
[2023-11-18T14:35:12,854][DEBUG][logstash.pluginmetadata ][main] Removing metadata for plugin 56342c1eb75526292660bd0d361a0666f89decca1ce9ddaf6dc0ef7730607566
[2023-11-18T14:35:12,856][DEBUG][logstash.outputs.stdout ][main] Closing {:plugin=>"LogStash::Outputs::Stdout"}
[2023-11-18T14:35:12,857][DEBUG][logstash.pluginmetadata ][main] Removing metadata for plugin d3c500ce86d399abd9deb0ac735fffb0d68af4f2056eec0e366cc1ede2be66e6
[2023-11-18T14:35:12,858][DEBUG][logstash.javapipeline ][main] Pipeline has been shutdown {:pipeline_id=>"main", :thread=>"#<Thread:0x530fc581 run>"}
[2023-11-18T14:35:12,860][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-11-18T14:35:13,050][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2023-11-18T14:35:13,057][DEBUG][logstash.instrument.periodicpoller.os] Stopping
[2023-11-18T14:35:13,133][DEBUG][logstash.instrument.periodicpoller.jvm] Stopping
[2023-11-18T14:35:13,134][DEBUG][logstash.instrument.periodicpoller.persistentqueue] Stopping
[2023-11-18T14:35:13,134][DEBUG][logstash.instrument.periodicpoller.deadletterqueue] Stopping
[2023-11-18T14:35:13,134][DEBUG][logstash.instrument.periodicpoller.flowrate] Stopping
[2023-11-18T14:35:13,163][DEBUG][logstash.agent ] API WebServer has stopped running
[2023-11-18T14:35:13,164][INFO ][logstash.runner ] Logstash shut down.
This means that Logstash received a shutdown command.
You need to provide more context like How are you running Logstash? What does your configuration looks like.
[2023-11-18T14:35:04,906][DEBUG][logstash.instrument.periodicpoller.cgroup] One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
This is unrelated and you can ignore this message, in fact you do not need to have DEBUG level enabled unless you need to debug anything, the INFO log level is enough for most of the troubleshooting cases.
Hi @leandrojmp - Thanks for you reply. Below are the complete details.
Deployed Logstash and Elasticsearch- version 8.5.1 in Azure Kubernetes Cluster using helm deployment. Logstash and elasticsearch are deployed in two different cluster, exposing elasticsearch through ingress with trusted certificate and using it in logstash output.
Elastic search is reachable from logstash pod.
curl -v -u"username:password" https://elasticsearch...*
Logstash Config File:
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => "https://elasticsearch...:443"
index => "log4j-%{+YYYY.MM.dd}"
user => ""
password => "*"
}
stdout {
}
}
Logstash Yaml:
pipeline.ordered: auto
path.config: /usr/share/logstash/config/logstash.conf
Now Enabled Info logs only:
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2023-11-19T06:40:09,168][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2023-11-19T06:40:09,238][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.5.1", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.5+8 on 17.0.5+8 +indy +jit [x86_64-linux]"}
[2023-11-19T06:40:09,242][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Xmx1g, -Xms1g, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-11-19T06:40:09,335][INFO ][logstash.settings ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2023-11-19T06:40:09,353][INFO ][logstash.settings ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2023-11-19T06:40:09,948][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2023-11-19T06:40:10,030][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"957188bc-2005-4a03-8e27-8f125f2ea219", :path=>"/usr/share/logstash/data/uuid"}
[2023-11-19T06:40:13,553][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-11-19T06:40:14,742][INFO ][org.reflections.Reflections] Reflections took 291 ms to scan 1 urls, producing 125 keys and 438 values
[2023-11-19T06:40:16,551][INFO ][logstash.javapipeline ] Pipeline main is configured with pipeline.ecs_compatibility: v8 setting. All plugins in this pipeline will default to ecs_compatibility => v8 unless explicitly configured otherwise.
[2023-11-19T06:40:16,751][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["https://elasticsearch...:443"]}
[2023-11-19T06:40:17,331][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://elastic:xxxxxx@elasticsearch...:443/]}}
[2023-11-19T06:40:18,153][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"https://elastic:xxxxxx@elasticsearch...*.:443/"}
[2023-11-19T06:40:18,258][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.5.1) {:es_version=>8}
[2023-11-19T06:40:18,262][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>8}
[2023-11-19T06:40:18,433][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
[2023-11-19T06:40:18,435][INFO ][logstash.outputs.elasticsearch][main] Config is not compliant with data streams. data_stream => auto resolved to false
[2023-11-19T06:40:18,437][WARN ][logstash.outputs.elasticsearch][main] Elasticsearch Output configured with ecs_compatibility => v8, which resolved to an UNRELEASED preview of version 8.0.0 of the Elastic Common Schema. Once ECS v8 and an updated release of this plugin are publicly available, you will need to update this plugin to resolve this warning.
[2023-11-19T06:40:18,649][INFO ][logstash.outputs.elasticsearch][main] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:v8}
[2023-11-19T06:40:18,649][INFO ][logstash.javapipeline ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>125, "pipeline.sources"=>["/usr/share/logstash/config/logstash.conf"], :thread=>"#<Thread:0x17e7d7be run>"}
[2023-11-19T06:40:20,645][INFO ][logstash.javapipeline ][main] Pipeline Java execution initialization time {"seconds"=>1.99}
[2023-11-19T06:40:20,747][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
[2023-11-19T06:40:20,766][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-11-19T06:40:21,130][INFO ][org.logstash.beats.Server][main][9961797f7d39d1477f01186771a008c1841f9f4e219707d4f538a7c57dc66530] Starting server on port: 5044
[2023-11-19T06:40:21,127][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
[2023-11-19T06:44:56,903][WARN ][logstash.runner ] SIGTERM received. Shutting down.
[2023-11-19T06:45:03,568][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-11-19T06:45:03,986][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
[2023-11-19T06:45:04,126][INFO ][logstash.runner ] Logstash shut down.
Once the pipeline is started and running logstash pod is not in ready status and keeps restarting with any other log message.
There is nothing wrong in the logs you shared.
As you can seee, your Logstash started normally:
[2023-11-19T06:40:20,766][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
[2023-11-19T06:40:21,130][INFO ][org.logstash.beats.Server][main][9961797f7d39d1477f01186771a008c1841f9f4e219707d4f538a7c57dc66530] Starting server on port: 5044
[2023-11-19T06:40:21,127][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}
But after a couple of minutes it received a shutdown command, something in your system told logstash to stop.
[2023-11-19T06:44:56,903][WARN ][logstash.runner ] SIGTERM received. Shutting down.
This doesn't seem to be an issue with Logstash, but with the way you are running it.
I have no experience with Kubernetes, but you need to check if everything is correct, maybe it is trying to see if the logstash pod is healthy, but for some reason it is not working and then it will kill the pod.
Check this similar issue on StackOverflow.
Hi @leandrojmp - OK thanks for your reply I will check further, to know the issue I enabled debug mode and got the below thing, but you are sure that this is not the cause of logstash shut down?
One or more required cgroup files or directories not found: /proc/self/cgroup, /sys/fs/cgroup/cpuacct, /sys/fs/cgroup/cpu
Yes, as already said this is unrelated, it is not an issue, it just appears on DEBUG level you can ignore it.
Check this answer for more context.
Thanks @leandrojmp - I fixed the issue by removing the liveness and readiness probes from my deployment and it started working.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.