Netflow in Logstash does not find the logstash.yml file

Elastic Stack Logstash
1

Hello everyone,
I am trying to use Netflow on my cluster, so I configured the logstash.yml file as describe on the articules as follows, this directory is located on /etc/logstash:
modules:
-name: netflow
var.input.udp.port: XXXX
var.elasticsearch.host: ["ip1:9200","ip2:9200","ip3:9200"]
var.kibana.host: "kibanaip:5600"

Then I moved to logstash installation directory to /usr/share/logstash and ran:
bin/logstash --modules netflow --setup
in order to start the netflow but, when I run this command I get a warning followed by a lot of errors. "Warning: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
"
Any suggestions?

2

I think you might be missing

bin/logstash --modules netflow --setup --path.settings /path/to/logstash.yml

where "/path/to/logstash.yml" is the path to your logstash.yml

3

Hello @Andrew22, thank you for your response, unfortiunetlly it didn't work, I added the --path/settings /etc/logstash to the end and when run it I got the following error:
Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-09-11T15:18:20,352][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Setting "var.input.udp.port" hasn't been registered>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:36:in get_setting'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:69:inset_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:88:in block in merge'", "org/jruby/RubyHash.java:1419:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:88:in merge'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:137:invalidate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `'"]}
[2019-09-11T15:18:20,368][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

4

I think that might be an indentation error in your logstash.yml file.

5

Thank you @Badger , I did resolve the indentation on the yml file, now I am getting this.

Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2019-09-12T10:13:58,105][INFO ][logstash.config.source.modules] Both command-line and logstash.yml modules configurations detected. Using command-line module configuration to override logstash.yml module configuration.
[2019-09-12T10:13:58,121][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-09-12T10:13:58,133][FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.
[2019-09-12T10:13:58,142][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

6

it looks like you are already running logstash. try stopping the service then running the command again

7

And the other pipelines that I have running will start again?

8

@Andrew22 Seems like now Im getting this error

Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-09-12T10:46:38,341][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Setting "" hasn't been registered>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:36:in get_setting'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:69:inset_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:88:in block in merge'", "org/jruby/RubyHash.java:1419:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:88:in merge'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:137:invalidate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `'"]}
[2019-09-12T10:46:38,356][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

9

@Andrew22 sorry for the late response I was Out of Town, question, yes indeed I have a pipeline already running, so I will have to stop logstash service, then run the above command (bin/logstash --modules netflow --setup --path.settings /path/to/logstash.yml), and this will automatically start the logstash service? I just want to be sure I understand.

10

That will run logstash with the settings in logstash.yml. I dont have much experience with starting logstash this way but I was just aware of how to do it.

11

Thank you @Andrew22, actually it worked, and installed everything on Kibana, but for some reason I am not getting any data in. Not sure what is going on.

12

He @Badger by any chance do you know if after running the netflow Module do you have to manually create a pipeline in Logstash? I ran the command to setup the netflow module and all went through, I have the Index pattern and the visualizations in kibana but no data is getting recieved on the Kibana side. also, after stopping logstash service and running the command

13

I don't know.

14

@badger, do you know how to stop the netflow module?

15

I do not.

16

Elastiflow is worth checking out. Netflow module is based on an earlier version of elastiflow

17

I have exactly same problem. all dashboard/visulization created, netflow-* index pattern created but no index

I think there is no support for this thing

18

@elasticforme is your logstash service running after you ran the Netflow command?

19

yes.
Actually this is my test cluster and I didn't have any pipleline configure. just netflow

as you can see from my last input on my thread. it seems running fine. I can run ps -ef |grep logstash and can see it is running. even logstash log says it is running
netstat -a shows that port 2055 is open and listening on 0.0.0.0
but no input on elasticsearch

21

Oh thats good,
the thing is that when I run the systemctl status logstash i get the following
logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-09-25 14:17:39 PDT; 1min 0s ago
Main PID: 24022 (java)
CGroup: /system.slice/logstash.service
└─24022 /bin/java -Xms16g -Xmx16g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-...

Sep 25 14:17:39 XXXX.com systemd[1]: Started logstash.
Sep 25 14:17:39 XXXX.com systemd[1]: Starting logstash...
Sep 25 14:18:27 XXXX.com logstash[24022]: Thread.exclusive is deprecated, use Thread::Mutex
Sep 25 14:18:30 XXXX.com logstash[24022]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties