# Experiencing Error trying to Launch Logstash **URL:** \ **Category:** Logstash\ **Created:** [February 28, 2025, 4:21pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227 "2025-02-28T16:21:00Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![abbyode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abbyode/32/119171_2.png) [@abbyode](https://discuss.elastic.co/u/abbyode)\ **Post date:** [February 28, 2025, 4:21pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/1 "2025-02-28T16:21:00Z")
I am experiencing some errors trying to launch logstash. I am using logstash plug-in to send json data to Sentinel Log Analytics Workspace. Below is the error i'm getting. ```auto [ERROR] 2025-02-28 15:45:10.466 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create
`"} [FATAL] 2025-02-28 15:45:10.474 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:405:in `block in converge_state'"]} [FATAL] 2025-02-28 15:45:10.481 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?] at usr.share.logstash.lib.bootstrap.environment.
(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?] ``` Below is my input section of the config file and there is no input, just output to sentinel which is confirmed by sentinel to be valid: ```auto input { file { path => "/var/log/verve/alerts.json" start_position => "end" sincedb_path => "/var/log/verve/sincedb.log" codec => "json" } } ``` I will appreciate any help to sort out this issue. regards, Abi. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [February 28, 2025, 4:29pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/2 "2025-02-28T16:29:45Z")
I would hope for two additional errors logged before that. The first of which will tell you the underlying problem. Can you check if any other errors are logged? ---
**Author:** ![abbyode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abbyode/32/119171_2.png) [@abbyode](https://discuss.elastic.co/u/abbyode)\ **Post date:** [February 28, 2025, 4:49pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/3 "2025-02-28T16:49:45Z")
Thank you for your response. Below is the full message: ```auto Using bundled JDK: /usr/share/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO] 2025-02-28 16:48:27.042 [main] runner - Starting Logstash {"logstash.version"=>"7.17.13", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.20+8 on 11.0.20+8 +indy +jit [linux-x86_64]"} [INFO] 2025-02-28 16:48:27.051 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true] [WARN] 2025-02-28 16:48:27.389 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2025-02-28 16:48:29.264 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [INFO] 2025-02-28 16:48:30.579 [Converge PipelineAction::Create
] Reflections - Reflections took 78 ms to scan 1 urls, producing 119 keys and 419 values [WARN] 2025-02-28 16:48:31.551 [Converge PipelineAction::Create
] json - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [WARN] 2025-02-28 16:48:31.658 [Converge PipelineAction::Create
] file - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [ERROR] 2025-02-28 16:48:31.749 [Converge PipelineAction::Create
] registry - Unable to load plugin. {:type=>"output", :name=>"microsoft-sentinel-log-analytics-logstash-output-plugin"} [ERROR] 2025-02-28 16:48:31.763 [Converge PipelineAction::Create
] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'microsoft-sentinel-log-analytics-logstash-output-plugin'. Are you sure this is correct? Trying to load the microsoft-sentinel-log-analytics-logstash-output-plugin output plugin resulted in this error: Unable to load the requested plugin named microsoft-sentinel-log-analytics-logstash-output-plugin of type output. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:120)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$ __VARARGS__ (/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:392)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]} warning: thread "Converge PipelineAction::Create
" terminated with exception (report_on_exception is true): LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create
` create at org/logstash/execution/ConvergeResultExt.java:135 add at org/logstash/execution/ConvergeResultExt.java:60 converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:405 [ERROR] 2025-02-28 16:48:31.770 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create
`"} [FATAL] 2025-02-28 16:48:31.782 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:405:in `block in converge_state'"]} [FATAL] 2025-02-28 16:48:31.804 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit org.jruby.exceptions.SystemExit: (SystemExit) exit at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?] at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?] at usr.share.logstash.lib.bootstrap.environment.
(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?] tadmin@host:~$ ``` Thanks for helping out on this. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [February 28, 2025, 5:05pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/4 "2025-02-28T17:05:23Z")
> [@abbyode](#): > > [ERROR] 2025-02-28 16:48:31.749 [Converge PipelineAction::Create] registry - Unable to load plugin. {:type=\>"output", :name=\>"microsoft-sentinel-log-analytics-logstash-output-plugin"} Did you install the plugin? Logstash is complaining that the plugin does not exist. To install it you need to run: ```auto logstash-plugin install microsoft-sentinel-log-analytics-logstash-output-plugin ``` ---
**Author:** ![abbyode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abbyode/32/119171_2.png) [@abbyode](https://discuss.elastic.co/u/abbyode)\ **Post date:** [February 28, 2025, 6:19pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/5 "2025-02-28T18:19:07Z")
Thank you for that observatrion. I think I removed the plug-in while trying to troubleshoot. It's now installed again. This is the original error I have been getting before now: ```auto tadmin@cgxxxxxx:/usr/share/logstash$ bin/logstash-plugin list --verbose | grep sentinel OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. microsoft-sentinel-log-analytics-logstash-output-plugin (1.1.4) tadmin@cgxxxxxx/usr/share/logstash$ cd\ > tadmin@cxxxx:~$ tadmin@cxxxx:~$ bin/logstash-plugin list --verbose | grep sentinel -bash: bin/logstash-plugin: No such file or directory tadmin@cxxxxx:~$ sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/sentinel.conf Using bundled JDK: /usr/share/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO] 2025-02-28 18:15:37.665 [main] runner - Starting Logstash {"logstash.version"=>"7.17.13", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.20+8 on 11.0.20+8 +indy +jit [linux-x86_64]"} [INFO] 2025-02-28 18:15:37.677 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true] [WARN] 2025-02-28 18:15:38.182 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2025-02-28 18:15:40.225 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [INFO] 2025-02-28 18:15:41.556 [Converge PipelineAction::Create
] Reflections - Reflections took 130 ms to scan 1 urls, producing 119 keys and 419 values [WARN] 2025-02-28 18:15:42.562 [Converge PipelineAction::Create
] json - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [WARN] 2025-02-28 18:15:42.638 [Converge PipelineAction::Create
] file - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [WARN] 2025-02-28 18:15:42.784 [Converge PipelineAction::Create
] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [INFO] 2025-02-28 18:15:42.935 [[main]-pipeline-manager] microsoftsentineloutput - Azure Loganalytics configuration was found valid. [INFO] 2025-02-28 18:15:43.093 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/sentinel.conf"], :thread=>"#"} [INFO] 2025-02-28 18:15:44.311 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.22} [INFO] 2025-02-28 18:15:44.429 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"} [INFO] 2025-02-28 18:15:44.536 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]} [INFO] 2025-02-28 18:15:44.549 [[main] **Author:** ![abbyode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abbyode/32/119171_2.png) [@abbyode](https://discuss.elastic.co/u/abbyode)\ **Post date:** [March 1, 2025, 8:40pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/6 "2025-03-01T20:40:11Z") I am still experiencing this issue. I will appreciate support from anyone here, please. ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [March 1, 2025, 9:46pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/7 "2025-03-01T21:46:29Z")
> [@abbyode](#): > > `java.lang.OutOfMemoryError: Java heap space` This message says you are out of memory, specifically JVM heap space. > [@abbyode](#): > > `-Xms1g, -Xmx1g` Can you change these to larger values, say 2g? How much RAM on your system? > [@abbyode](#): > > ```auto > tadmin@cxxxx:~$ bin/logstash-plugin list --verbose | grep sentinel > -bash: bin/logstash-plugin: No such file or directory > > ``` This means you were not in the right directory when you ran the command. > [@abbyode](#): > > Below is my input section of the config file **and there is no input** , just output to sentinel which is confirmed by sentinel to be valid > > ```auto > input { > file { > path => "/var/log/verve/alerts.json" > start_position => "end" > sincedb_path => "/var/log/verve/sincedb.log" > codec => "json" > } > } > > ``` I am struggling to parse that sentence, can you re-word maybe? The config you attached clearly has _only_ an input section. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [March 1, 2025, 9:50pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/8 "2025-03-01T21:50:44Z")
`microsoft-sentinel-log-analytics-logstash-output-plugin ` is not maintained by Elastic, it is a Microsoft-maintained plugin that exists in this repository: [GitHub - Azure/Azure-Sentinel: Cloud-native SIEM for intelligent security analytics for your entire enterprise.](https://github.com/Azure/Azure-Sentinel) It seems they don't really take bugs in that repo and you are expected to file a support ticket with Microsoft for any issues. You haven't shared the size of alerts you're trying to process from alerts.json, you haven't shared your actual `pipelines.yml` configuration or the actual `pipeline.conf` with the input/filter/output you're using. If you can provide those it'll make helping you easier. It does look like the output plugin has quite verbose debug logs though so enabling `log.level: debug` in logstash.yml will probably reveal something useful. Anyway -- it looks like that output plugin will produce batches of 2000 events or larger and by default Logstash will provision 1 worker for each CPU core. This could end up being quite the number of events you've instructed Logstash to hold in memory at once. Which might explain the `out of memory` error. You're giving Logstash 1gb of memory. So your options are probably (in order of good ideas to bad ideas): 1. [Explicitly set the number of pipeline workers to something smaller than the current value](https://www.elastic.co/guide/en/logstash/current/tuning-logstash.html) (to limit how many events are held in memory at once) 2. [Increase the memory you're providing Logstash](https://www.elastic.co/guide/en/logstash/current/jvm-settings.html) 3. Open a support ticket with Microsoft about their plugin breaking Logstash and tell them they need to update their documentation 4. Use undocumented settings from the output plugin to [set `max_items` lower than `2000` in your output plugin](https://github.com/Azure/Azure-Sentinel/blob/0f2c7faaf107d4708b874d7302d49a5393537b4f/DataConnectors/microsoft-sentinel-log-analytics-logstash-output-plugin/lib/logstash/outputs/microsoft-sentinel-log-analytics-logstash-output-plugin.rb#L48) and [disable automatic batch resizing by setting `amount_resizing: false`](https://github.com/Azure/Azure-Sentinel/blob/0f2c7faaf107d4708b874d7302d49a5393537b4f/DataConnectors/microsoft-sentinel-log-analytics-logstash-output-plugin/lib/logstash/outputs/microsoft-sentinel-log-analytics-logstash-output-plugin.rb#L44C11-L44C26) to reduce how many events are being held in memory by the output. ---
**Author:** ![abbyode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abbyode/32/119171_2.png) [@abbyode](https://discuss.elastic.co/u/abbyode)\ **Post date:** [March 2, 2025, 4:48am UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/9 "2025-03-02T04:48:34Z")
Hello, I did some cleanup and I am no longer getting the memory error. However, when i run logstash, it currently gets stuck here: [INFO] 2025-03-02 04:44:56.992 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50, "pipeline.max\_inflight"=\>500, "pipeline.sources"=\>["/etc/logstash/conf.d/sentinel.conf"], :thread=\>"#\"} [INFO] 2025-03-02 04:44:57.589 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=\>0.59} [INFO] 2025-03-02 04:44:57.632 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=\>"main"} [INFO] 2025-03-02 04:44:57.696 [[main]\1, :running\_pipelines=\>[:main], :non\_running\_pipelines=\>} Any pointer or support will be appreciated. regards, Abi ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 2, 2025, 5:13am UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/10 "2025-03-02T05:13:44Z")
That looks like it started normally. If you have a file input then it may just be waiting for something to be appended to the file it is watching. logstash typically doesn't log anything if it is happily passing events from the input to the output. ---
**Author:** ![abbyode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abbyode/32/119171_2.png) [@abbyode](https://discuss.elastic.co/u/abbyode)\ **Post date:** [March 2, 2025, 5:19am UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/11 "2025-03-02T05:19:32Z")
Yes, I am reading from a json file. The json file reads data continuously via a syslog. I do get the logs being read when I tail the json file. In the past, logstash will be showing how it's parsing the files. Now, it looks like it's stuck. Any thoughts? ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [March 2, 2025, 12:26pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/12 "2025-03-02T12:26:26Z")
what's in the syncedb file: `/var/log/verve/sincedb.log` does it even exist?, and if so is it's content changing as more actual logs appears in the alerts.json file. Unless I'm missing it, you have not shared the output config. ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [March 2, 2025, 3:28pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/13 "2025-03-02T15:28:09Z")
Yes, as @raintown mentions, please share your full pipeline configuration. Logstash does not print anything to the log during normal processing. You would have to include something in your pipeline for it to start logging to the console. ---
**Author:** ![abbyode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abbyode/32/119171_2.png) [@abbyode](https://discuss.elastic.co/u/abbyode)\ **Post date:** [March 2, 2025, 8:58pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/14 "2025-03-02T20:58:23Z")
This is my output configuration: ```auto output { microsoft-sentinel-log-analytics-logstash-output-plugin { client_app_Id => "XXXXXX-5edb-4487-8fb1-235d3dffbf40" client_app_secret => "XXXXXXXGUQN1Z.jQU7u4nvZbnm" tenant_id => "XXXXXXXX2-be283455925a" data_collection_endpoint => "https://lxxxxxxxus-1.ingest.monitor.azure.com" dcr_immutable_id => "xxxxxxxxxxxxece91ac4cb146213e7b" dcr_stream_name => "Custom-verve_CL" #ecs_compatibility => "v8" #compress_data => "true" # stdout { # coded => rubydebug} } } ``` Current stuck at the stage below with no error: ```auto [WARN] 2025-03-02 20:41:14.075 [Converge PipelineAction::Create
] file - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [WARN] 2025-03-02 20:41:14.265 [Converge PipelineAction::Create
] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [INFO] 2025-03-02 20:41:14.467 [[main]-pipeline-manager] microsoftsentineloutput - Azure Loganalytics configuration was found valid. [INFO] 2025-03-02 20:41:14.658 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/sentinel.conf"], :thread=>"#"} [INFO] 2025-03-02 20:41:15.849 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.19} [INFO] 2025-03-02 20:41:15.997 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"} [INFO] 2025-03-02 20:41:16.084 [[main]1, :running_pipelines=>[:main], :non_running_pipelines=>[]} ``` The content of my /var/log/verve/sincedb.log is as below: ```auto 786736 0 64768 3821738966 1740587571.066462 /var/log/verve/alerts.json 789604 0 64768 510141645 1740948076.3117032 /var/log/verve/alerts.json ``` Any more thoughts? ---
**Author:** ![strawgate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/strawgate/32/131008_2.png) [@strawgate](https://discuss.elastic.co/u/strawgate)\ **Post date:** [March 2, 2025, 9:01pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/15 "2025-03-02T21:01:03Z")
```auto # stdout { # coded => rubydebug} ``` it looks like you've commented out the output that writes to stdout, if you want to see things printed to the console you'll want to remove that comment and add the stdout output back in. ---
**Author:** ![abbyode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abbyode/32/119171_2.png) [@abbyode](https://discuss.elastic.co/u/abbyode)\ **Post date:** [March 2, 2025, 9:12pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/16 "2025-03-02T21:12:51Z")
I only added that part during the troubleshooting, but it was giving some errors. Find below what I got when I uncommented that part out. ```auto You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO] 2025-03-02 21:11:36.782 [main] runner - Starting Logstash {"logstash.version"=>"7.17.13", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.20+8 on 11.0.20+8 +indy +jit [linux-x86_64]"} [INFO] 2025-03-02 21:11:36.792 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true] [WARN] 2025-03-02 21:11:37.166 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2025-03-02 21:11:39.163 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [ERROR] 2025-03-02 21:11:40.264 [Converge PipelineAction::Create
] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", \"=>\" at line 22, column 9 (byte 703) after output {\n\nmicrosoft-sentinel-log-analytics-logstash-output-plugin {\n client_app_Id => \"88d1466e-5edb-4487-8fb1-235d3dffbf40\"\n client_app_secret => \"aBs8Q~9kAtFhB1d6lGPTAGUQN1Z.jQU7u4nvZbnm\"\n tenant_id => \"d1cab833-de8d-48f5-a422-be283455925a\"\n data_collection_endpoint => \"https://logsingestion-881y.eastus-1.ingest.monitor.azure.com\"\n dcr_immutable_id => \"dcr-6dccb7d971854ece91ac4cb146213e7b\"\n dcr_stream_name => \"Custom-verve_CL\"\n #ecs_compatibility => \"v8\"\n #compress_data => \"true\"\n\tstdout ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:392:in `block in converge_state'"]} [INFO] 2025-03-02 21:11:40.421 [LogStash::Runner] runner - Logstash shut down. ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 2, 2025, 9:23pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/17 "2025-03-02T21:23:38Z")
Validate your configuration, the `stdout` needs to be outside the configuration of the other plugin, currently it is wrong, also there is a typo, it is `codec` not `coded`, and this is not required as rubydebug is the default codec. Something like this should work: ```auto output { microsoft-sentinel-log-analytics-logstash-output-plugin { client_app_Id => "XXXXXX-5edb-4487-8fb1-235d3dffbf40" client_app_secret => "XXXXXXXGUQN1Z.jQU7u4nvZbnm" tenant_id => "XXXXXXXX2-be283455925a" data_collection_endpoint => "https://lxxxxxxxus-1.ingest.monitor.azure.com" dcr_immutable_id => "xxxxxxxxxxxxece91ac4cb146213e7b" dcr_stream_name => "Custom-verve_CL" #ecs_compatibility => "v8" #compress_data => "true" } stdout {} } ``` ---
**Author:** ![abbyode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abbyode/32/119171_2.png) [@abbyode](https://discuss.elastic.co/u/abbyode)\ **Post date:** [March 2, 2025, 9:42pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/18 "2025-03-02T21:42:45Z")
Thank you for spotting that. Now this is my full configuration: ```auto input { file { path => "/var/log/verve/alerts.json" start_position => "beginning" sincedb_path => "/var/log/verve/sincedb.log" codec => "json" } } output { microsoft-sentinel-log-analytics-logstash-output-plugin { client_app_Id => "xxxxxx487-8fb1-235d3dffbf40" client_app_secret => "xxxxxd6lGPTAGUQN1Z.jQU7u4nvZbnm" tenant_id => "xxxxxxx-a422-be283455925a" data_collection_endpoint => "https://xxxxx1y.eastus-1.ingest.monitor.azure.com" dcr_immutable_id => "xxxxxxxx854ece91ac4cb146213e7b" dcr_stream_name => "Custom-verve_CL" #ecs_compatibility => "v8" #compress_data => "true" } stdout { codec => rubydebug} } ``` The logstash still got stuck as below: ```auto root@cgysenlfp02:~# sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/sentinel.conf Using bundled JDK: /usr/share/logstash/jdk OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release. WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console [INFO] 2025-03-02 21:39:10.006 [main] runner - Starting Logstash {"logstash.version"=>"7.17.13", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.20+8 on 11.0.20+8 +indy +jit [linux-x86_64]"} [INFO] 2025-03-02 21:39:10.015 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true] [WARN] 2025-03-02 21:39:10.349 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified [INFO] 2025-03-02 21:39:12.377 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false} [INFO] 2025-03-02 21:39:13.793 [Converge PipelineAction::Create
] Reflections - Reflections took 96 ms to scan 1 urls, producing 119 keys and 419 values [WARN] 2025-03-02 21:39:14.762 [Converge PipelineAction::Create
] json - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [WARN] 2025-03-02 21:39:14.885 [Converge PipelineAction::Create
] file - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [WARN] 2025-03-02 21:39:14.995 [Converge PipelineAction::Create
] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode. [INFO] 2025-03-02 21:39:15.414 [[main]-pipeline-manager] microsoftsentineloutput - Azure Loganalytics configuration was found valid. [INFO] 2025-03-02 21:39:15.491 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/sentinel.conf"], :thread=>"#"} [INFO] 2025-03-02 21:39:16.626 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.13} [INFO] 2025-03-02 21:39:16.805 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"} [INFO] 2025-03-02 21:39:16.845 [[main]1, :running_pipelines=>[:main], :non_running_pipelines=>[]} ``` ---
**Author:** ![RainTown](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raintown/32/140206_2.png) [@RainTown](https://discuss.elastic.co/u/RainTown)\ **Post date:** [March 2, 2025, 9:59pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/19 "2025-03-02T21:59:41Z")
> [@abbyode](#): > > Now this is my full configuration Just to be sure, those are the full contents of the file `/etc/logstash/conf.d/sentinel.conf` ? Also, whats the current size (in bytes) of the log file, i.e. output from `ls -li /var/log/verve/alerts.json` and current content of the file `/var/log/verve/sincedb.log` btw, maybe important here, likely not, but the inode number of the log file changed ```auto 786736 0 64768 3821738966 1740587571.066462 /var/log/verve/alerts.json 789604 0 64768 510141645 1740948076.3117032 /var/log/verve/alerts.json ``` The inode number is the first field. See > **[File input plugin | Logstash Reference \[8.17\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_tracking_of_current_position_in_watched_files)** ---
**Author:** ![abbyode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abbyode/32/119171_2.png) [@abbyode](https://discuss.elastic.co/u/abbyode)\ **Post date:** [March 2, 2025, 10:08pm UTC](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227/20 "2025-03-02T22:08:46Z")
Thanks for your consideration. Please find below as requested: ```auto root@xxxxxx02:~# ls -li /var/log/verve/alerts.json 789604 -rw-r--r-- 1 root root 873757268 Mar 2 06:07 /var/log/verve/alerts.json root@xxxxxxp02:~# cat /var/log/verve/sincedb.log 786736 0 64768 3821738966 1740587571.066462 /var/log/verve/alerts.json 789604 0 64768 510141645 1740951557.029709 /var/log/verve/alerts.json ``` Thanks for your interest. [Next page](https://discuss.elastic.co/t/experiencing-error-trying-to-launch-logstash/375227.md?page=2)