Logstash issue with Netflow module

juanquy · October 6, 2017, 9:14pm

...
2017-10-06T14:52:47,133][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-10-06T14:52:47,144][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-10-06T14:52:47,305][INFO ][logstash.agent           ] No config files found in path {:path=>"/etc/logstash/conf.d/*"}
[2017-10-06T14:52:55,305][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"codec", :name=>"netflow", :path=>"logstash/codecs/netflow", :error_message=>"NameErr$
[2017-10-06T14:52:55,319][ERROR][logstash.agent           ] Cannot create pipeline {:reason=>"Couldn't find any codec plugin named 'netflow'. Are you sure this is correct? Trying to lo$
[2017-10-06T14:54:26,939][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-10-06T14:54:26,944][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-10-06T14:54:27,128][INFO ][logstash.agent           ] No config files found in path {:path=>"/etc/logstash/conf.d/*"}
[2017-10-06T14:54:35,599][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"codec", :name=>"netflow", :path=>"logstash/codecs/netflow", :error_message=>"NameErr$
[2017-10-06T14:54:35,612][ERROR][logstash.agent           ] Cannot create pipeline {:reason=>"Couldn't find any codec plugin named 'netflow'. Are you sure this is correct? Trying to lo$
[2017-10-06T14:56:07,933][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-10-06T14:56:07,938][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-10-06T14:56:08,158][INFO ][logstash.agent           ] No config files found in path {:path=>"/etc/logstash/conf.d/*"}
[2017-10-06T14:56:15,885][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"codec", :name=>"netflow", :path=>"logstash/codecs/netflow", :error_message=>"NameErr$
[2017-10-06T14:56:15,898][ERROR][logstash.agent           ] Cannot create pipeline {:reason=>"Couldn't find any codec plugin named 'netflow'. Are you sure this is correct? Trying to lo$
[2017-10-06T14:57:08,761][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-10-06T14:57:08,766][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-10-06T14:57:08,922][INFO ][logstash.agent           ] No config files found in path {:path=>"/etc/logstash/conf.d/*"}
[2017-10-06T14:57:17,018][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"codec", :name=>"netflow", :path=>"logstash/codecs/netflow", :error_message=>"NameErr$
[2017-10-06T14:57:17,036][ERROR][logstash.agent           ] Cannot create pipeline {:reason=>"Couldn't find any codec plugin named 'netflow'. Are you sure this is correct? Trying to lo$
[2017-10-06T15:24:56,987][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-10-06T15:24:56,992][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-10-06T15:24:57,208][INFO ][logstash.agent           ] No config files found in path {:path=>"/etc/logstash/conf.d/*"}
[2017-10-06T15:25:05,535][ERROR][logstash.plugins.registry] Problems loading a plugin with {:type=>"codec", :name=>"netflow", :path=>"logstash/codecs/netflow", :error_message=>"NameErr$
[2017-10-06T15:25:05,553][ERROR][logstash.agent           ] Cannot create pipeline {:reason=>"Couldn't find any codec plugin named 'netflow'. Are you sure this is correct? Trying to lo$
...

theuntergeek · October 6, 2017, 9:17pm

What version of Logstash did you install?

juanquy · October 6, 2017, 9:19pm

LS,K,E =v5.6.2

theuntergeek · October 6, 2017, 9:22pm

This is not okay. I don't know how it happened, but I suggest purging the install and re-installing.

juanquy · October 6, 2017, 9:31pm

I saw that and I does not made any sense to me!
Purge only LS or also Kibana and ElasticS?

theuntergeek · October 6, 2017, 9:49pm

You should only need to purge Logstash.

juanquy · October 6, 2017, 9:59pm

before go home this is the new log after purge

...
[2017-10-06T16:57:21,493][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-10-06T16:57:21,498][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-10-06T16:57:21,504][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2017-10-06T16:57:21,506][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2017-10-06T16:57:21,571][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"b3a9cdb6-6065-4845-b859-dad35af5fb1b", :path=>"/var/lib/lo$
[2017-10-06T16:57:21,684][INFO ][logstash.agent           ] No config files found in path {:path=>"/etc/logstash/conf.d/*"}
[2017-10-06T16:57:31,125][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2017-10-06T16:57:31,127][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, $
[2017-10-06T16:57:31,235][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2017-10-06T16:57:31,238][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
...

juanquy · October 6, 2017, 10:02pm

and now is working!, I will keep my eyes during the weekend a mark this issue solve on monday

regards and I appreciated your help a lot!
and have a nice weekend!!
JC

juanquy · October 13, 2017, 2:29pm

Hi:
I hate to say this: New install in a new Ubuntu Server 16.04, and its doing the same, for some reason logstash start without reading .yml config file.
if I start logstash via cli passing parameters manually , it start perfectly, the issue is when it start via systemctl
------------Logstash logs from yesterday when I started via CLI ----------------
...
2017-10-12T16:33:37,384][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-10-12T16:33:37,389][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-10-12T16:33:37,394][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/var/lib/logstash/queue"}
[2017-10-12T16:33:37,395][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/var/lib/logstash/dead_letter_queue"}
[2017-10-12T16:33:39,676][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"73c05d3d-ad8d-4325-bf90-dc8ba92b89d6", :path=>"/var/lib/log$
[2017-10-12T16:33:39,799][INFO ][logstash.agent ] No config files found in path {:path=>"/etc/logstash/conf.d/*"}
[2017-10-12T16:33:48,739][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2017-10-12T16:33:48,741][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/,
[2017-10-12T16:33:48,775][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2017-10-12T16:33:48,782][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[2017-10-12T16:33:49,476][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.3.1-java/vendor/$
[2017-10-12T16:33:49,552][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.3.1-java/vendor/$
[2017-10-12T16:33:49,554][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.3.1-java/vendor/$
[2017-10-12T16:33:49,554][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.3.1-java/vendor/$
[2017-10-12T16:33:49,555][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.3.1-java/vendor/$
[2017-10-12T16:33:49,556][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.3.1-java/vendor/$
[2017-10-12T16:33:49,557][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.3.1-java/vendor/$
[2017-10-12T16:33:49,559][INFO ][logstash.filters.geoip ] Using geoip database {:path=>"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-geoip-4.3.1-java/vendor/$
[2017-10-12T16:33:49,569][INFO ][logstash.pipeline ] Starting pipeline {"id"=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipelin$
[2017-10-12T16:33:49,594][INFO ][logstash.pipeline ] Pipeline main started
[2017-10-12T16:33:49,601][INFO ][logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:2055"}
[2017-10-12T16:33:49,646][INFO ][logstash.inputs.udp ] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2017-10-12T16:33:49,649][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-10-12T16:33:50,262][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 256 from source id 0, because no template to decode it with has been received. This messa$
[2017-10-12T16:33:50,270][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 256 from source id 0, because no template to decode it with has been received. This messa$
[2017-10-12T16:33:50,275][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 256 from source id 0, because no template to decode it with has been received. This messa$
[2017-10-12T16:33:50,283][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 256 from source id 0, because no template to decode it with has been received. This messa$
[2017-10-12T16:33:50,308][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 256 from source id 0, because no template to decode it with has been received. This messa$
[2017-10-12T16:33:50,318][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 256 from source id 0, because no template to decode it with has been received.
[2017-10-12T16:33:51,242][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 256 from source id 0, because no template to decode it with has been received. This messa$
[2017-10-12T16:33:51,276][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 256 from source id 0, because no template to decode it with has been received. This messa$
[2017-10-12T16:33:51,282][WARN ][logstash.codecs.netflow ] Can't (yet) decode flowset id 256 from source id 0,
[2017-10-12T16:38:49,615][INFO ][logstash.filters.translate] refreshing dictionary file
[2017-10-12T16:38:49,691][INFO ][logstash.filters.translate] refreshing dictionary file
[2017-10-12T16:43:42,719][WARN ][logstash.runner ] SIGINT received. Shutting down the agent.
[2017-10-12T16:43:42,735][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}
[2017-10-12T16:43:42,749][WARN ][logstash.inputs.udp ] UDP listener died {:exception=>#<IOError: closed stream>, :backtrace=>["org/jruby/RubyIO.java:3705:in `select'", "/usr/$
...
wierd is that when it start via systemctl is not logging anything I reboot the server look the status of the services , they are green and running.

So unfortunately the issue still here
regards
JC

theuntergeek · October 13, 2017, 3:17pm

Is this a parallel install? I mean, is this in addition to the other Logstash install you already have running?

What are the logs when you launch via systemctl? Does it show similar permission issues? Is the Logstash user able to open port 2055 on this machine? I presume that you've looked similarly to see if port 2055 is open on this one.

There's a problem with starting at the CLI if you do not su to the logstash user first. It can create files with other permissions, which would cause issues when starting again as the logstash user, which might not have permissions to those files.

juanquy · October 13, 2017, 7:20pm

k I will check that, I'm pretty sure I did it as a root user. This a new server we are getting ready to put in production, the one we talk before it was a test server we have in the lab for testing proposes. if everything is ok them we move it to production .
So now we are trying deploy ELK on a production server, do you think the issue could be caz I'm running it on a server runing over LXD? ( Linux containers)

juanquy · October 13, 2017, 7:22pm

.....do you think the issue could be caz I'm running it on a server runing over LXD? ( Linux containers) .....
never mind that is not the issue, I will check privileges and I will back later.

juanquy · October 13, 2017, 9:21pm

ok :
ssh as root and purge ELK, re-install it as root and exactly the same , if I start Logstash via CLI :

...
./logstash --path.settings=/etc/logstash --modules netflow --setup -M netflow.var.input.udp.port=2055
..
It work perfectly.

but if I start it
$ systemctl start logstash.service
It not work at all.

juanquy · October 13, 2017, 9:30pm

This is the file logstash.service.

...
[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash

Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.

Prefixing the path with '-' makes it try to load, but if the file doesn't

exist, it continues onward.

EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target
....

juanquy · October 13, 2017, 9:31pm

no sure if the line :
...
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
...
is OK.

I think it should be like this:
...
ExecStart=/usr/share/logstash/bin/logstash --path.settings "/etc/logstash"
...
but I'm not sure and I don't want to mess with it in a production server.
I will move to the Lab server and try there..

regards
JC

theuntergeek · October 17, 2017, 7:04am

No. Each argument is supposed to be inside its own set of quotes.

system · November 14, 2017, 7:04am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.