Experiencing Error trying to Launch Logstash

I am experiencing some errors trying to launch logstash. I am using logstash plug-in to send json data to Sentinel Log Analytics Workspace. Below is the error i'm getting.

[ERROR] 2025-02-28 15:45:10.466 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
[FATAL] 2025-02-28 15:45:10.474 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:405:in `block in converge_state'"]}
[FATAL] 2025-02-28 15:45:10.481 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

Below is my input section of the config file and there is no input, just output to sentinel which is confirmed by sentinel to be valid:

input {
  file {
    path => "/var/log/verve/alerts.json"
    start_position => "end"
  sincedb_path => "/var/log/verve/sincedb.log"
    codec => "json"
  }
}

I will appreciate any help to sort out this issue.

regards,

Abi.

I would hope for two additional errors logged before that. The first of which will tell you the underlying problem. Can you check if any other errors are logged?

Thank you for your response.

Below is the full message:

Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2025-02-28 16:48:27.042 [main] runner - Starting Logstash {"logstash.version"=>"7.17.13", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.20+8 on 11.0.20+8 +indy +jit [linux-x86_64]"}
[INFO ] 2025-02-28 16:48:27.051 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2025-02-28 16:48:27.389 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2025-02-28 16:48:29.264 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2025-02-28 16:48:30.579 [Converge PipelineAction::Create<main>] Reflections - Reflections took 78 ms to scan 1 urls, producing 119 keys and 419 values
[WARN ] 2025-02-28 16:48:31.551 [Converge PipelineAction::Create<main>] json - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2025-02-28 16:48:31.658 [Converge PipelineAction::Create<main>] file - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[ERROR] 2025-02-28 16:48:31.749 [Converge PipelineAction::Create<main>] registry - Unable to load plugin. {:type=>"output", :name=>"microsoft-sentinel-log-analytics-logstash-output-plugin"}
[ERROR] 2025-02-28 16:48:31.763 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'microsoft-sentinel-log-analytics-logstash-output-plugin'. Are you sure this is correct? Trying to load the microsoft-sentinel-log-analytics-logstash-output-plugin output plugin resulted in this error: Unable to load the requested plugin named microsoft-sentinel-log-analytics-logstash-output-plugin of type output. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:392)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
          create at org/logstash/execution/ConvergeResultExt.java:135
             add at org/logstash/execution/ConvergeResultExt.java:60
  converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:405
[ERROR] 2025-02-28 16:48:31.770 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
[FATAL] 2025-02-28 16:48:31.782 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:405:in `block in converge_state'"]}
[FATAL] 2025-02-28 16:48:31.804 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
tadmin@host:~$

Thanks for helping out on this.

Did you install the plugin? Logstash is complaining that the plugin does not exist.

To install it you need to run:

logstash-plugin install microsoft-sentinel-log-analytics-logstash-output-plugin
1 Like

Thank you for that observatrion. I think I removed the plug-in while trying to troubleshoot. It's now installed again.
This is the original error I have been getting before now:

tadmin@cgxxxxxx:/usr/share/logstash$ bin/logstash-plugin list --verbose | grep sentinel
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
microsoft-sentinel-log-analytics-logstash-output-plugin (1.1.4)
tadmin@cgxxxxxx/usr/share/logstash$ cd\
>
tadmin@cxxxx:~$
tadmin@cxxxx:~$ bin/logstash-plugin list --verbose | grep sentinel
-bash: bin/logstash-plugin: No such file or directory
tadmin@cxxxxx:~$ sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/sentinel.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2025-02-28 18:15:37.665 [main] runner - Starting Logstash {"logstash.version"=>"7.17.13", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.20+8 on 11.0.20+8 +indy +jit [linux-x86_64]"}
[INFO ] 2025-02-28 18:15:37.677 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2025-02-28 18:15:38.182 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2025-02-28 18:15:40.225 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2025-02-28 18:15:41.556 [Converge PipelineAction::Create<main>] Reflections - Reflections took 130 ms to scan 1 urls, producing 119 keys and 419 values
[WARN ] 2025-02-28 18:15:42.562 [Converge PipelineAction::Create<main>] json - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2025-02-28 18:15:42.638 [Converge PipelineAction::Create<main>] file - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2025-02-28 18:15:42.784 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2025-02-28 18:15:42.935 [[main]-pipeline-manager] microsoftsentineloutput - Azure Loganalytics configuration was found valid.
[INFO ] 2025-02-28 18:15:43.093 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/sentinel.conf"], :thread=>"#<Thread:0x52573f9e run>"}
[INFO ] 2025-02-28 18:15:44.311 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.22}
[INFO ] 2025-02-28 18:15:44.429 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2025-02-28 18:15:44.536 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2025-02-28 18:15:44.549 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid1068608.hprof ...
Heap dump file created [1121631203 bytes in 2.445 secs]

I am still experiencing this issue. I will appreciate support from anyone here, please.

This message says you are out of memory, specifically JVM heap space.

Can you change these to larger values, say 2g?

How much RAM on your system?

This means you were not in the right directory when you ran the command.

I am struggling to parse that sentence, can you re-word maybe? The config you attached clearly has only an input section.

microsoft-sentinel-log-analytics-logstash-output-plugin is not maintained by Elastic, it is a Microsoft-maintained plugin that exists in this repository: GitHub - Azure/Azure-Sentinel: Cloud-native SIEM for intelligent security analytics for your entire enterprise.

It seems they don't really take bugs in that repo and you are expected to file a support ticket with Microsoft for any issues.

You haven't shared the size of alerts you're trying to process from alerts.json, you haven't shared your actual pipelines.yml configuration or the actual pipeline.conf with the input/filter/output you're using. If you can provide those it'll make helping you easier. It does look like the output plugin has quite verbose debug logs though so enabling log.level: debug in logstash.yml will probably reveal something useful.

Anyway -- it looks like that output plugin will produce batches of 2000 events or larger and by default Logstash will provision 1 worker for each CPU core. This could end up being quite the number of events you've instructed Logstash to hold in memory at once. Which might explain the out of memory error. You're giving Logstash 1gb of memory.

So your options are probably (in order of good ideas to bad ideas):

  1. Explicitly set the number of pipeline workers to something smaller than the current value (to limit how many events are held in memory at once)
  2. Increase the memory you're providing Logstash
  3. Open a support ticket with Microsoft about their plugin breaking Logstash and tell them they need to update their documentation
  4. Use undocumented settings from the output plugin to set max_items lower than 2000 in your output plugin and disable automatic batch resizing by setting amount_resizing: false to reduce how many events are being held in memory by the output.

Hello,

I did some cleanup and I am no longer getting the memory error.
However, when i run logstash, it currently gets stuck here:

[INFO ] 2025-03-02 04:44:56.992 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/sentinel.conf"], :thread=>"#<Thread:0x5d96b722 run>"}
[INFO ] 2025-03-02 04:44:57.589 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.59}
[INFO ] 2025-03-02 04:44:57.632 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2025-03-02 04:44:57.696 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2025-03-02 04:44:57.700 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>}

Any pointer or support will be appreciated.

regards,
Abi

That looks like it started normally. If you have a file input then it may just be waiting for something to be appended to the file it is watching.

logstash typically doesn't log anything if it is happily passing events from the input to the output.

Yes, I am reading from a json file. The json file reads data continuously via a syslog. I do get the logs being read when I tail the json file. In the past, logstash will be showing how it's parsing the files. Now, it looks like it's stuck. Any thoughts?

what's in the syncedb file: /var/log/verve/sincedb.log

does it even exist?, and if so is it's content changing as more actual logs appears in the alerts.json file.

Unless I'm missing it, you have not shared the output config.

1 Like

Yes, as @raintown mentions, please share your full pipeline configuration.

Logstash does not print anything to the log during normal processing. You would have to include something in your pipeline for it to start logging to the console.

This is my output configuration:

output {

microsoft-sentinel-log-analytics-logstash-output-plugin {
      client_app_Id => "XXXXXX-5edb-4487-8fb1-235d3dffbf40"
      client_app_secret => "XXXXXXXGUQN1Z.jQU7u4nvZbnm"
      tenant_id => "XXXXXXXX2-be283455925a"
      data_collection_endpoint => "https://lxxxxxxxus-1.ingest.monitor.azure.com"
      dcr_immutable_id => "xxxxxxxxxxxxece91ac4cb146213e7b"
      dcr_stream_name => "Custom-verve_CL"
      #ecs_compatibility => "v8"
      #compress_data => "true"
#       stdout {
#               coded => rubydebug}



    }
}

Current stuck at the stage below with no error:

[WARN ] 2025-03-02 20:41:14.075 [Converge PipelineAction::Create<main>] file - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2025-03-02 20:41:14.265 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2025-03-02 20:41:14.467 [[main]-pipeline-manager] microsoftsentineloutput - Azure Loganalytics configuration was found valid.
[INFO ] 2025-03-02 20:41:14.658 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/sentinel.conf"], :thread=>"#<Thread:0x32d654c7 run>"}
[INFO ] 2025-03-02 20:41:15.849 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.19}
[INFO ] 2025-03-02 20:41:15.997 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2025-03-02 20:41:16.084 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2025-03-02 20:41:16.171 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

The content of my /var/log/verve/sincedb.log is as below:

786736 0 64768 3821738966 1740587571.066462 /var/log/verve/alerts.json
789604 0 64768 510141645 1740948076.3117032 /var/log/verve/alerts.json

Any more thoughts?

 #       stdout {
 #               coded => rubydebug}

it looks like you've commented out the output that writes to stdout, if you want to see things printed to the console you'll want to remove that comment and add the stdout output back in.

I only added that part during the troubleshooting, but it was giving some errors. Find below what I got when I uncommented that part out.

 You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2025-03-02 21:11:36.782 [main] runner - Starting Logstash {"logstash.version"=>"7.17.13", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.20+8 on 11.0.20+8 +indy +jit [linux-x86_64]"}
[INFO ] 2025-03-02 21:11:36.792 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2025-03-02 21:11:37.166 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2025-03-02 21:11:39.163 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[ERROR] 2025-03-02 21:11:40.264 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"=>\" at line 22, column 9 (byte 703) after output {\n\nmicrosoft-sentinel-log-analytics-logstash-output-plugin {\n      client_app_Id => \"88d1466e-5edb-4487-8fb1-235d3dffbf40\"\n      client_app_secret => \"aBs8Q~9kAtFhB1d6lGPTAGUQN1Z.jQU7u4nvZbnm\"\n      tenant_id => \"d1cab833-de8d-48f5-a422-be283455925a\"\n      data_collection_endpoint => \"https://logsingestion-881y.eastus-1.ingest.monitor.azure.com\"\n      dcr_immutable_id => \"dcr-6dccb7d971854ece91ac4cb146213e7b\"\n      dcr_stream_name => \"Custom-verve_CL\"\n      #ecs_compatibility => \"v8\"\n      #compress_data => \"true\"\n\tstdout ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:392:in `block in converge_state'"]}
[INFO ] 2025-03-02 21:11:40.421 [LogStash::Runner] runner - Logstash shut down.

Validate your configuration, the stdout needs to be outside the configuration of the other plugin, currently it is wrong, also there is a typo, it is codec not coded, and this is not required as rubydebug is the default codec.

Something like this should work:

output {
    microsoft-sentinel-log-analytics-logstash-output-plugin {
      client_app_Id => "XXXXXX-5edb-4487-8fb1-235d3dffbf40"
      client_app_secret => "XXXXXXXGUQN1Z.jQU7u4nvZbnm"
      tenant_id => "XXXXXXXX2-be283455925a"
      data_collection_endpoint => "https://lxxxxxxxus-1.ingest.monitor.azure.com"
      dcr_immutable_id => "xxxxxxxxxxxxece91ac4cb146213e7b"
      dcr_stream_name => "Custom-verve_CL"
      #ecs_compatibility => "v8"
      #compress_data => "true"
    }
    stdout {} 
}
1 Like

Thank you for spotting that.
Now this is my full configuration:

input {
  file {
    path => "/var/log/verve/alerts.json"
    start_position => "beginning"
  sincedb_path => "/var/log/verve/sincedb.log"
    codec => "json"
  }
}


output {

microsoft-sentinel-log-analytics-logstash-output-plugin {
      client_app_Id => "xxxxxx487-8fb1-235d3dffbf40"
      client_app_secret => "xxxxxd6lGPTAGUQN1Z.jQU7u4nvZbnm"
      tenant_id => "xxxxxxx-a422-be283455925a"
      data_collection_endpoint => "https://xxxxx1y.eastus-1.ingest.monitor.azure.com"
      dcr_immutable_id => "xxxxxxxx854ece91ac4cb146213e7b"
      dcr_stream_name => "Custom-verve_CL"
      #ecs_compatibility => "v8"
      #compress_data => "true"

       }
        stdout { codec => rubydebug}



    }

The logstash still got stuck as below:

root@cgysenlfp02:~# sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/sentinel.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2025-03-02 21:39:10.006 [main] runner - Starting Logstash {"logstash.version"=>"7.17.13", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.20+8 on 11.0.20+8 +indy +jit [linux-x86_64]"}
[INFO ] 2025-03-02 21:39:10.015 [main] runner - JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[WARN ] 2025-03-02 21:39:10.349 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2025-03-02 21:39:12.377 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2025-03-02 21:39:13.793 [Converge PipelineAction::Create<main>] Reflections - Reflections took 96 ms to scan 1 urls, producing 119 keys and 419 values
[WARN ] 2025-03-02 21:39:14.762 [Converge PipelineAction::Create<main>] json - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2025-03-02 21:39:14.885 [Converge PipelineAction::Create<main>] file - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2025-03-02 21:39:14.995 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2025-03-02 21:39:15.414 [[main]-pipeline-manager] microsoftsentineloutput - Azure Loganalytics configuration was found valid.
[INFO ] 2025-03-02 21:39:15.491 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/sentinel.conf"], :thread=>"#<Thread:0x15dba87e run>"}
[INFO ] 2025-03-02 21:39:16.626 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.13}
[INFO ] 2025-03-02 21:39:16.805 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2025-03-02 21:39:16.845 [[main]<file] observingtail - START, creating Discoverer, Watch with file and sincedb collections
[INFO ] 2025-03-02 21:39:16.921 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

Just to be sure, those are the full contents of the file /etc/logstash/conf.d/sentinel.conf ?

Also, whats the current size (in bytes) of the log file, i.e. output from ls -li /var/log/verve/alerts.json and current content of the file /var/log/verve/sincedb.log

btw, maybe important here, likely not, but the inode number of the log file changed

786736 0 64768 3821738966 1740587571.066462 /var/log/verve/alerts.json
789604 0 64768 510141645 1740948076.3117032 /var/log/verve/alerts.json

The inode number is the first field. See

Thanks for your consideration. Please find below as requested:

root@xxxxxx02:~# ls -li /var/log/verve/alerts.json
789604 -rw-r--r-- 1 root root 873757268 Mar  2 06:07 /var/log/verve/alerts.json

root@xxxxxxp02:~# cat /var/log/verve/sincedb.log
786736 0 64768 3821738966 1740587571.066462 /var/log/verve/alerts.json
789604 0 64768 510141645 1740951557.029709 /var/log/verve/alerts.json

Thanks for your interest.