Help with identifying cause of Netflow IPFIX UDP service crashing randomly

Elastic Stack Logstash
1

Hi, complete newbie here.

I have setup a Debian 9.1 server running logstash 5.6.0 along with the logstash-codec-netflow (3.5.2) plugin. Input is IPFIX and output to another server running Elasticsearch. (Test environment) I get around 70 000 events a minute into elastic so it is working but what I have noticed is that the the UDP listener dies and restarts randomly. I have set java xms and xmx to 4g just incase it is running out of memory but I suspect it is something else; also have 6 input workers running.

I can see the following in the syslog at the times when it restarts:
Sep 15 07:31:24 TEST logstash[7196]: SystemStackError: stack level too deep
Sep 15 07:31:24 TEST logstash[7196]: module_eval at org/jruby/RubyModule.java:2346
Sep 15 07:31:24 TEST logstash[7196]: define_methods at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/int.rb:47
Sep 15 07:31:24 TEST logstash[7196]: Uint524280be at (eval):2
Sep 15 07:31:24 TEST logstash[7196]: (eval) at (eval):1
Sep 15 07:31:24 TEST logstash[7196]: module_eval at org/jruby/RubyModule.java:2346
Sep 15 07:31:24 TEST logstash[7196]: define_class at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/int.rb:13
Sep 15 07:31:24 TEST logstash[7196]: const_missing at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/int.rb:198
Sep 15 07:31:24 TEST logstash[7196]: each_pair at org/jruby/RubyHash.java:1367
Sep 15 07:31:24 TEST logstash[7196]: const_missing at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/int.rb:194
Sep 15 07:31:24 TEST logstash[7196]: const_get at org/jruby/RubyModule.java:2646
Sep 15 07:31:24 TEST logstash[7196]: register_dynamic_class at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/registry.rb:111
Sep 15 07:31:24 TEST logstash[7196]: registered? at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/registry.rb:102
Sep 15 07:31:24 TEST logstash[7196]: normalize_name at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/registry.rb:71
Sep 15 07:31:24 TEST logstash[7196]: each at org/jruby/RubyArray.java:1613
Sep 15 07:31:24 TEST logstash[7196]: normalize_name at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/registry.rb:63
Sep 15 07:31:24 TEST logstash[7196]: lookup at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/registry.rb:40

/* remove some lines due to topic limit of 7000 characters */

Sep 15 07:31:24 TEST logstash[7196]:      sanitize_parameters! at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/struct.rb:345
Sep 15 07:31:24 TEST logstash[7196]:                 sanitize! at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/sanitize.rb:302
Sep 15 07:31:24 TEST logstash[7196]:                initialize at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/sanitize.rb:210
Sep 15 07:31:24 TEST logstash[7196]:                  sanitize at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/sanitize.rb:192
Sep 15 07:31:24 TEST logstash[7196]:              extract_args at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/base.rb:302
Sep 15 07:31:24 TEST logstash[7196]:              extract_args at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/base.rb:249
Sep 15 07:31:24 TEST logstash[7196]:                initialize at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/base.rb:81
Sep 15 07:31:24 TEST logstash[7196]:   initialize_with_warning at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/warnings.rb:21
Sep 15 07:31:24 TEST logstash[7196]:              decode_ipfix at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-3.5.2/lib/logstash/codecs/netflow.rb:318
Sep 15 07:31:24 TEST logstash[7196]:                     catch at org/jruby/RubyKernel.java:1242
Sep 15 07:31:24 TEST logstash[7196]:              decode_ipfix at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-3.5.2/lib/logstash/codecs/netflow.rb:303
Sep 15 07:31:24 TEST logstash[7196]:                      each at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/array.rb:208
Sep 15 07:31:24 TEST logstash[7196]:                      each at org/jruby/RubyArray.java:1613
Sep 15 07:31:24 TEST logstash[7196]:                      each at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/array.rb:208
Sep 15 07:31:24 TEST logstash[7196]:              decode_ipfix at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-3.5.2/lib/logstash/codecs/netflow.rb:302
Sep 15 07:31:24 TEST logstash[7196]:                    decode at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-3.5.2/lib/logstash/codecs/netflow.rb:124
Sep 15 07:31:24 TEST logstash[7196]:                      each at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/array.rb:208
Sep 15 07:31:24 TEST logstash[7196]:                      each at org/jruby/RubyArray.java:1613
Sep 15 07:31:24 TEST logstash[7196]:                      each at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/bindata-2.4.0/lib/bindata/array.rb:208
Sep 15 07:31:24 TEST logstash[7196]:                    decode at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-codec-netflow-3.5.2/lib/logstash/codecs/netflow.rb:123
Sep 15 07:31:24 TEST logstash[7196]:               inputworker at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.2/lib/logstash/inputs/udp.rb:118
Sep 15 07:31:24 TEST logstash[7196]:              udp_listener at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.1.2/lib/logstash/inputs/udp.rb:89
Sep 15 07:31:25 TEST systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 07:31:25 TEST systemd[1]: logstash.service: Unit entered failed state.
Sep 15 07:31:25 TEST systemd[1]: logstash.service: Failed with result 'exit-code'.
Sep 15 07:31:25 TEST systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
Sep 15 07:31:25 TEST systemd[1]: Stopped logstash.
Sep 15 07:31:25 TEST systemd[1]: Started logstash.

any help would be appreciated.

2

The output that is pasted is not formatted. Could you please format it to make it readable?

Also, what is the config?
On what port is the listener running?
And who is the user running logstash? Privileged or non privileged?

Regards
N

3

It seems my ignorance broke it; I was attemtping to use a custom ipfix_definitions file in my config; once remove all was stable again.

thanks for asking the question :smile:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.