Netflow in Logstash does not find the logstash.yml file

And the other pipelines that I have running will start again?

@Andrew22 Seems like now Im getting this error

Thread.exclusive is deprecated, use Thread::Mutex
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2019-09-12T10:46:38,341][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Setting "" hasn't been registered>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:36:in get_setting'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:69:inset_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:88:in block in merge'", "org/jruby/RubyHash.java:1419:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:88:in merge'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:137:invalidate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:242:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in `'"]}
[2019-09-12T10:46:38,356][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

@Andrew22 sorry for the late response I was Out of Town, question, yes indeed I have a pipeline already running, so I will have to stop logstash service, then run the above command (bin/logstash --modules netflow --setup --path.settings /path/to/logstash.yml), and this will automatically start the logstash service? I just want to be sure I understand.

That will run logstash with the settings in logstash.yml. I dont have much experience with starting logstash this way but I was just aware of how to do it.

Thank you @Andrew22, actually it worked, and installed everything on Kibana, but for some reason I am not getting any data in. Not sure what is going on.

He @Badger by any chance do you know if after running the netflow Module do you have to manually create a pipeline in Logstash? I ran the command to setup the netflow module and all went through, I have the Index pattern and the visualizations in kibana but no data is getting recieved on the Kibana side. also, after stopping logstash service and running the command

I don't know.

@badger, do you know how to stop the netflow module?

I do not.

Elastiflow is worth checking out. Netflow module is based on an earlier version of elastiflow

I have exactly same problem. all dashboard/visulization created, netflow-* index pattern created but no index

I think there is no support for this thing

@elasticforme is your logstash service running after you ran the Netflow command?

yes.
Actually this is my test cluster and I didn't have any pipleline configure. just netflow

as you can see from my last input on my thread. it seems running fine. I can run ps -ef |grep logstash and can see it is running. even logstash log says it is running
netstat -a shows that port 2055 is open and listening on 0.0.0.0
but no input on elasticsearch

Oh thats good,
the thing is that when I run the systemctl status logstash i get the following
logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
Active: active (running) since Wed 2019-09-25 14:17:39 PDT; 1min 0s ago
Main PID: 24022 (java)
CGroup: /system.slice/logstash.service
└─24022 /bin/java -Xms16g -Xmx16g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-...

Sep 25 14:17:39 XXXX.com systemd[1]: Started logstash.
Sep 25 14:17:39 XXXX.com systemd[1]: Starting logstash...
Sep 25 14:18:27 XXXX.com logstash[24022]: Thread.exclusive is deprecated, use Thread::Mutex
Sep 25 14:18:30 XXXX.com logstash[24022]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties

that is good sign. it is running. what is your log file says.
mine has this

[2019-09-25T16:17:52,971][INFO ][logstash.javapipeline ] Starting pipeline {:pipeline_id=>"module-netflow", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, :thread=>"#<Thread:0x79801812 run>"}
[2019-09-25T16:17:53,054][INFO ][logstash.javapipeline ] Pipeline started {"pipeline.id"=>"module-netflow"}
[2019-09-25T16:17:53,103][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:2055"}
[2019-09-25T16:17:53,139][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:"module-netflow"], :non_running_pipelines=>}
[2019-09-25T16:17:53,168][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"212992", :queue_size=>"2000"}
[2019-09-25T16:17:53,400][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

I have this on logstash.yml file. I try both ip_address/name and both combination none is working

modules:
  - name: netflow
    var.elasticsearch.hosts: 10.29.111.1:9200
    var.elasticsearch.ssl.enabled: false
    var.kibana.host: elktst10:5601
    var.kibana.scheme: http
    var.kibana.ssl.enabled: false
    var.kibana.ssl.verification_mode: disable

My log says
2019-09-25T00:07:09,781][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2019-09-25T00:08:11,461][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-09-25T00:08:11,486][FATAL][logstash.runner ] Logstash could not be started because there is already another instance using the configured data directory. If you wish to run multiple instances, you must change the "path.data" setting.

And I think this was caused because I stopped the logstash service then ran
bin/logstash --modules netflow --setup --path.settings /etc/logstash

before I started logstash, then all the Netflow thing got installed and setup, same way that happened to you, but on my end the logstash service did not start and I dont have a clue what to do because I have another pipeline on the same logstash server and now is not working

you have to remove that -module thing from logstash.yml file and restart logstash and it will start up.
I just tested it and it works.

but it still does not solve both of our problem that no data is coming in from netflow

Im guessing that you have not yet received a template for the netflow. If you run journalctl -fu logstash you will probably see a message about not having received a template yet. Let it run for a bit and see if it starts indexing into elasticaearch

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.