Netflow module error

Elastic Stack Logstash
1

(Sorry for any formatting issues, my first post here and didn't want to put in too much data.)

I have ELK 6.2 up and running (single host) and processing data from a variety of sources and can see the data in Kibana. All of the data is received in logstash and then passed to elasticsearch. I'm running it via systemd on a Centos server. I've researched and followed instructions in configuring the netflow module but when I add it to my logstash.yml file, I get an error in the logstash log. If I comment out those lines and restart things, everything works fine.

I see a netflow module in /usr/share/logstash/modules, so I think that was set up correct (also see a configurations sub-directory with kibana and a dashboards directory.

Back to the code, here is the section (highlighted in bold) in my logstash.yml file that triggers the error:

# ------------ Module Settings ---------------
# Define modules here.  Modules definitions must be defined as an array.
# The simple way to see this is to prepend each `name` with a `-`, and keep
# all associated variables under the `name` they are associated with, and
# above the next, like this:
#
# modules:
#   - name: MODULE_NAME
#     var.PLUGINTYPE1.PLUGINNAME1.KEY1: VALUE
#     var.PLUGINTYPE1.PLUGINNAME1.KEY2: VALUE
#     var.PLUGINTYPE2.PLUGINNAME1.KEY1: VALUE
#     var.PLUGINTYPE3.PLUGINNAME3.KEY1: VALUE
#
# Module variable names must be in the format of
#
# var.PLUGIN_TYPE.PLUGIN_NAME.KEY
#
modules:
  - name: netflow
    var.input.udp.port: 9995

With those 3 lines (modules and the 2 lines under it) in the file I get:
[2018-04-03T10:17:30,964][DEBUG][logstash.runner ] --------------- Logstash Settings -------------------
[2018-04-03T10:17:30,968][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-04-03T10:17:31,002][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit

I don't think the warning has anything to do with the error since I also get it once when it works (i.e., gives me data in Kibana).
Any suggestions?

I'm using the following version of Centos and just updated the system.
(Linux elk 3.10.0-693.21.1.el7.x86_64 #1 SMP Wed Mar 7 19:03:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux)

And the following version of java:
openjdk version "1.8.0_161"
OpenJDK Runtime Environment (build 1.8.0_161-b14)
OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode)

2

I have exaclty the same issue. Ubuntu 16.04, clean installation.

If you don't mention module in logstash.yml, netflow goes to elastic without any modification, and Kibana reports are not working.

Still have no idea.

3

And more: I've tried to start logstash like this:

sudo /usr/share/logstash/bin/logstash -f /etc/logstash

I get weird stdout:

ERROR] 2018-04-17 07:06:24.262 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {
:action=>LogStash::PipelineAction::Create/pipeline_id:main,

:exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n",

:backtrace=>[
"/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'",
"/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'",
"/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'",
"org/jruby/RubyArray.java:2486:in `map'",
"/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in `initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'",
"/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'",
"org/jruby/RubyArray.java:1734:in `each'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'",
"/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in `execute'",
"/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in `block in execute'",
"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}
4

I have the solution. It's weird, kind of.

Make sure you don't have path.config in your logstash.yml. This is the moment where everything breaks.

So, afterwards you won't be able to configure netflow module via custom input/output settings somewhere in conf.d. So I've edited /usr/share/logstash/modules/netflow/configuration/logstash/netflow.conf.erb to fit my particular needs.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.