Logstash Upgrade and 8.10.1 net snmp error

VamPikmin · September 19, 2023, 3:08am

After the upgrade logstash doesn't start

Any else experience this? Thank you!

org.jruby.exceptions.LoadError: (LoadError) no such file to load -- net/smtp
at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1057) ~[jruby.jar:?]
at RUBY.module:Mail(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/mail-2.6.6/lib/mail.rb:9) ~[?:?]

See log below:

[2023-09-19T12:59:46,045][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-09-19T12:59:46,049][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.10.1", "jruby.version"=>"jruby 9.4.2.0 (3.1.0) 2023-03-08 90d2913fda OpenJDK 64-Bit Server VM 17.0.8+7 on 17.0.8+7 +jit [x86_64-linux]"}
[2023-09-19T12:59:46,052][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms4g, -Xmx4g, -XX:+DisableExplicitGC, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -XX:+HeapDumpOnOutOfMemoryError, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-09-19T12:59:46,730][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2023-09-19T12:59:46,731][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2023-09-19T12:59:46,827][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2023-09-19T12:59:47,101][INFO ][org.reflections.Reflections] Reflections took 105 ms to scan 1 urls, producing 132 keys and 464 values
[2023-09-19T12:59:47,292][INFO ][logstash.javapipeline    ] Pipeline `.monitoring-logstash` is configured with `pipeline.ecs_compatibility: v8` setting. All plugins in this pipeline will default to `ecs_compatibility => v8` unless explicitly configured otherwise.
[2023-09-19T12:59:47,306][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://localhost:9200"]}
[2023-09-19T12:59:47,313][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2023-09-19T12:59:47,327][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2023-09-19T12:59:47,331][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (8.10.1) {:es_version=>8}
[2023-09-19T12:59:47,331][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-09-19T12:59:47,356][WARN ][logstash.javapipeline    ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2023-09-19T12:59:47,366][INFO ][logstash.javapipeline    ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x247e3b77 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}
[2023-09-19T12:59:47,884][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>0.52}
[2023-09-19T12:59:47,915][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2023-09-19T12:59:48,844][INFO ][logstash.javapipeline    ] Pipeline `main` is configured with `pipeline.ecs_compatibility: disabled` setting. All plugins in this pipeline will default to `ecs_compatibility => disabled` unless explicitly configured otherwise.
[2023-09-19T12:59:48,868][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2023-09-19T12:59:48,872][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2023-09-19T12:59:48,879][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2023-09-19T12:59:48,883][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.10.1) {:es_version=>8}
[2023-09-19T12:59:48,883][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-09-19T12:59:48,888][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because ecs_compatibility is not enabled. Elasticsearch data streams require that events adhere to the Elastic Common Schema. While `ecs_compatibility` can be set for this individual Elasticsearch output plugin, doing so will not fix schema conflicts caused by upstream plugins in your pipeline. To avoid mapping conflicts, you will need to use ECS-compatible field names and datatypes throughout your pipeline. Many plugins support an `ecs_compatibility` mode, and the `pipeline.ecs_compatibility` setting can be used to opt-in for all plugins in a pipeline.
[2023-09-19T12:59:48,888][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2023-09-19T12:59:48,889][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2023-09-19T12:59:48,892][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2023-09-19T12:59:48,897][INFO ][logstash.outputs.elasticsearch] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:disabled}
[2023-09-19T12:59:48,906][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2023-09-19T12:59:48,911][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.10.1) {:es_version=>8}
[2023-09-19T12:59:48,911][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-09-19T12:59:48,916][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because ecs_compatibility is not enabled. Elasticsearch data streams require that events adhere to the Elastic Common Schema. While `ecs_compatibility` can be set for this individual Elasticsearch output plugin, doing so will not fix schema conflicts caused by upstream plugins in your pipeline. To avoid mapping conflicts, you will need to use ECS-compatible field names and datatypes throughout your pipeline. Many plugins support an `ecs_compatibility` mode, and the `pipeline.ecs_compatibility` setting can be used to opt-in for all plugins in a pipeline.
[2023-09-19T12:59:48,916][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2023-09-19T12:59:48,929][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2023-09-19T12:59:48,933][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2023-09-19T12:59:48,941][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2023-09-19T12:59:48,941][INFO ][logstash.outputs.elasticsearch] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:disabled}
[2023-09-19T12:59:48,945][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.10.1) {:es_version=>8}
[2023-09-19T12:59:48,945][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-09-19T12:59:48,957][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because ecs_compatibility is not enabled. Elasticsearch data streams require that events adhere to the Elastic Common Schema. While `ecs_compatibility` can be set for this individual Elasticsearch output plugin, doing so will not fix schema conflicts caused by upstream plugins in your pipeline. To avoid mapping conflicts, you will need to use ECS-compatible field names and datatypes throughout your pipeline. Many plugins support an `ecs_compatibility` mode, and the `pipeline.ecs_compatibility` setting can be used to opt-in for all plugins in a pipeline.
[2023-09-19T12:59:48,957][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2023-09-19T12:59:48,975][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2023-09-19T12:59:48,980][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2023-09-19T12:59:48,987][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2023-09-19T12:59:48,990][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.10.1) {:es_version=>8}
[2023-09-19T12:59:48,990][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-09-19T12:59:48,993][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because ecs_compatibility is not enabled. Elasticsearch data streams require that events adhere to the Elastic Common Schema. While `ecs_compatibility` can be set for this individual Elasticsearch output plugin, doing so will not fix schema conflicts caused by upstream plugins in your pipeline. To avoid mapping conflicts, you will need to use ECS-compatible field names and datatypes throughout your pipeline. Many plugins support an `ecs_compatibility` mode, and the `pipeline.ecs_compatibility` setting can be used to opt-in for all plugins in a pipeline.
[2023-09-19T12:59:48,993][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2023-09-19T12:59:48,996][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2023-09-19T12:59:49,000][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2023-09-19T12:59:49,000][INFO ][logstash.outputs.elasticsearch] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:disabled}
[2023-09-19T12:59:49,004][INFO ][logstash.outputs.elasticsearch] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:disabled}
[2023-09-19T12:59:49,006][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2023-09-19T12:59:49,008][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.10.1) {:es_version=>8}
[2023-09-19T12:59:49,008][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-09-19T12:59:49,011][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because ecs_compatibility is not enabled. Elasticsearch data streams require that events adhere to the Elastic Common Schema. While `ecs_compatibility` can be set for this individual Elasticsearch output plugin, doing so will not fix schema conflicts caused by upstream plugins in your pipeline. To avoid mapping conflicts, you will need to use ECS-compatible field names and datatypes throughout your pipeline. Many plugins support an `ecs_compatibility` mode, and the `pipeline.ecs_compatibility` setting can be used to opt-in for all plugins in a pipeline.
[2023-09-19T12:59:49,011][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2023-09-19T12:59:49,016][INFO ][logstash.outputs.elasticsearch] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:disabled}
[2023-09-19T12:59:49,018][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200"]}
[2023-09-19T12:59:49,030][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2023-09-19T12:59:49,039][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2023-09-19T12:59:49,041][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch version determined (8.10.1) {:es_version=>8}
[2023-09-19T12:59:49,042][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>8}
[2023-09-19T12:59:49,047][INFO ][logstash.outputs.elasticsearch][main] Not eligible for data streams because ecs_compatibility is not enabled. Elasticsearch data streams require that events adhere to the Elastic Common Schema. While `ecs_compatibility` can be set for this individual Elasticsearch output plugin, doing so will not fix schema conflicts caused by upstream plugins in your pipeline. To avoid mapping conflicts, you will need to use ECS-compatible field names and datatypes throughout your pipeline. Many plugins support an `ecs_compatibility` mode, and the `pipeline.ecs_compatibility` setting can be used to opt-in for all plugins in a pipeline.
[2023-09-19T12:59:49,047][INFO ][logstash.outputs.elasticsearch][main] Data streams auto configuration (`data_stream => auto` or unset) resolved to `false`
[2023-09-19T12:59:49,058][INFO ][logstash.outputs.elasticsearch] Using a default mapping template {:es_version=>8, :ecs_compatibility=>:disabled}
[2023-09-19T12:59:49,068][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2023-09-19T12:59:49,139][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2023-09-19T12:59:49,352][INFO ][logstash.javapipeline    ][.monitoring-logstash] Pipeline terminated {"pipeline.id"=>".monitoring-logstash"}
[2023-09-19T12:59:49,353][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (LoadError) no such file to load -- net/smtp
org.jruby.exceptions.LoadError: (LoadError) no such file to load -- net/smtp
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1057) ~[jruby.jar:?]
        at RUBY.<module:Mail>(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/mail-2.6.6/lib/mail.rb:9) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/mail-2.6.6/lib/mail.rb:3) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1057) ~[jruby.jar:?]
        at RUBY.register(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-output-email-4.1.2/lib/logstash/outputs/email.rb:101) ~[?:?]
        at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.register(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:69) ~[logstash-core.jar:?]
        at RUBY.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:237) ~[?:?]
        at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1987) ~[jruby.jar:?]
        at RUBY.register_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:236) ~[?:?]
        at RUBY.maybe_setup_out_plugins(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:610) ~[?:?]
        at RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:249) ~[?:?]
        at RUBY.run(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:194) ~[?:?]
        at RUBY.start(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146) ~[?:?]

carly.richmond · September 19, 2023, 9:09am

Hi @VamPikmin,

Welcome back! Just a couple of questions on this one:

How did you install the upgrade version?
Aside from installation have you made any changes to your Logstash config?
Do you have any plugins included in your install, especially custom plugins?

VamPikmin · September 19, 2023, 10:03pm

Hi @carly.richmond

Thank you for your reply.

I upgraded it the usual way with apt (Ubuntu 22.04)

I do have pipeline.ecs_compatibility: disabled

Went through the config and narrowed it down to the logstash-output-email

  else if [winlog][event_id] == "4740" {
        email {
        debug => "true"
        to => "myemail@something.com"
        from => "lulu"
        subject => "%{[event_id_description]} %{[winlog][event_data][TargetUserName]}"
        body => "Event ID: %{[winlog][event_id]}\n\n%{message}"
        via => "sendmail"
        }

        elasticsearch {
        hosts => ["http://localhost:9200"]
        index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
        } }

So far I tried to uninstall and reinstall the plugin but didn't help

/usr/share/logstash/bin/logstash-plugin uninstall logstash-output-email
 /usr/share/logstash/bin/logstash-plugin install logstash-output-email

carly.richmond · September 20, 2023, 10:39am

Thanks for confirming. Can you confirm the version of the plugin that you're using and that you're using the latest version as per the documentation?

I see from the GitHub repo that the latest tagged version of the plugin is from May, so it might be worth raising a GitHub bug issue for this one if you are on the latest as this plugin is a tier 1 supported plugin as per the support matrix.

VamPikmin · September 20, 2023, 10:36pm

Thanks for your help Carly

I'll try to submit an issue on github. I tried to rename the topic here but I can't seem to do it
snmp should say smtp.

lee.clemens · October 2, 2023, 3:17pm

I just wanted to reference the github Issue for future internet travelers, and thanks @VamPikmin for creating it! It looks like there is an open PR as well.

VamPikmin · October 3, 2023, 1:39am

Thank you Lee

system · October 31, 2023, 1:40am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.