Logstash defaults modules?

Elastic Stack Logstash
1

I just installed ELK and I got it to work for a few minutes then I booted the vm and now Im trying to get back.

logstash doesnt listen to the port I specified in the conf file. I also noticed int the log file

:/etc/logstash$ tail -f /var/log/logstash/logstash-plain.log
[2017-12-19T17:34:03,898][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n"}
[2017-12-19T17:34:18,708][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-19T17:34:18,714][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-19T17:34:18,848][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n"}
[2017-12-19T17:34:33,682][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-19T17:34:33,688][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-19T17:34:33,849][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n"}
[2017-12-19T17:34:48,473][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-19T17:34:48,479][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-19T17:34:48,650][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n"}
[2017-12-19T17:35:03,288][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2017-12-19T17:35:03,294][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2017-12-19T17:35:03,430][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 6, column 1 (byte 132) after ## JVM configuration\n\n# Xms represents the initial size of total heap space\n# Xmx represents the maximum size of total heap space\n\n"}

two modules are spamming netflow and fb_apache ? I didn't install netflow nor fb_apache.
this spamming of the logs keeps me from viewing more important log messages. How can I disable these modules?

2

It looks like you may have put jvm.options into the configuration directory?

1 Like
3

how can I verify and undo that?

btw logstash is taking up lots of cpu and I havent started to push syslog into it yet, only from one Cisco ASA context. Is this normal?
top - 17:50:25 up 3:03, 3 users, load average: 4,16, 4,46, 4,48
Tasks: 242 total, 1 running, 241 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0,1 us, 1,0 sy, 20,3 ni, 78,7 id, 0,0 wa, 0,0 hi, 0,0 si, 0,0 st

USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
44723 logstash 39 19 5121708 290676 17580 S 614,6 1,8 0:28.93 java

16vCPU.

Edit: I want to clarify that I havent moved any files. until now I moved jvm.options out of /etc/logstash .

4

btw logstash is taking up lots of cpu and I havent started to push syslog into it yet, only from one Cisco ASA context. Is this normal?

Is this after you've gotten jvm.options out of the way so that Logstash actually starts up properly?

Edit: I want to clarify that I havent moved any files. until now I moved jvm.options out of /etc/logstash .

So the previous error is gone, yes?

5

Actually there were high cpu util right after I finished installing ELK when I was using 4 vCPUs and rigt now when I have less then 1 incomming logg per second ( using tcpdump udp port 5544 on the server that is running ELK ) cpu can high.

Right know logstash isn't listening to port 5544 any more

6

Except spam from netflow and fb_apache modules in the log

cat /var/log/logstash/logstash-plain-2017-12-19.log | grep -vE 'netflow|fb_apache'

results in:
[2017-12-19T23:57:11,443][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
[2017-12-19T23:57:26,814][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
[2017-12-19T23:57:42,242][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
[2017-12-19T23:57:58,038][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
[2017-12-19T23:58:14,597][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
[2017-12-19T23:58:30,642][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
[2017-12-19T23:58:45,182][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
[2017-12-19T23:58:59,579][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
[2017-12-19T23:59:14,623][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
[2017-12-19T23:59:29,766][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}
[2017-12-19T23:59:44,966][ERROR][logstash.agent ] Cannot create pipeline {:reason=>"Expected one of #, input, filter, output at line 1, column 1 (byte 1) after "}

logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled)
Active: active (running) since ons 2017-12-20 13:59:21 CET; 3s ago
Main PID: 62532 (java)
Tasks: 34
Memory: 208.9M
CPU: 13.278s
CGroup: /system.slice/logstash.service
└─62532 /usr/bin/java -Xmx500m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.hom

cat /etc/systemd/system/logstash.service
[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
Prefixing the path with '-' makes it try to load, but if the file doesn't
exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

:/usr/share/elasticsearch$ cat /etc/logstash/logstash.yml | grep -vE ^#
path.data: /var/lib/logstash
path.config: /etc/logstash

path.logs: /var/log/logstash

7

It seems that logstash cant find logstash.yml , but it is located in the default directory.

:/usr/share/logstash$ sudo bin/logstash --log.level=debug -t -f /etc/logstash/conf.d/logstash.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[DEBUG] 2017-12-20 14:24:48.792 [LogStash::Runner] DateFilter - Date filter with format=MMM dd HH:mm:ss, locale=null, timezone=null built as org.logstash.filters.parser.JodaParser
[DEBUG] 2017-12-20 14:24:48.807 [LogStash::Runner] DateFilter - Date filter with format=MMM dd HH:mm:ss, locale=en-US, timezone=null built as org.logstash.filters.parser.JodaParser
[DEBUG] 2017-12-20 14:24:48.808 [LogStash::Runner] DateFilter - Date filter with format=MMM d HH:mm:ss, locale=null, timezone=null built as org.logstash.filters.parser.JodaParser
[DEBUG] 2017-12-20 14:24:48.808 [LogStash::Runner] DateFilter - Date filter with format=MMM d HH:mm:ss, locale=en-US, timezone=null built as org.logstash.filters.parser.JodaParser
[DEBUG] 2017-12-20 14:24:48.808 [LogStash::Runner] DateFilter - Date filter with format=MMM dd yyyy HH:mm:ss, locale=null, timezone=null built as org.logstash.filters.parser.JodaParser
[DEBUG] 2017-12-20 14:24:48.809 [LogStash::Runner] DateFilter - Date filter with format=MMM dd yyyy HH:mm:ss, locale=en-US, timezone=null built as org.logstash.filters.parser.JodaParser
[DEBUG] 2017-12-20 14:24:48.809 [LogStash::Runner] DateFilter - Date filter with format=MMM d yyyy HH:mm:ss, locale=null, timezone=null built as org.logstash.filters.parser.JodaParser
[DEBUG] 2017-12-20 14:24:48.809 [LogStash::Runner] DateFilter - Date filter with format=MMM d yyyy HH:mm:ss, locale=en-US, timezone=null built as org.logstash.filters.parser.JodaParser
Configuration OK

:/usr/share/logstash$ ls /etc/logstash/
conf.d log4j2.properties logstash.yml startup.options
1:/usr/share/logstash$

also I can't use the configtest

:/usr/share/logstash$ sudo service logstash configtest
logstash: unrecognized service
:/usr/share/logstash$ sudo service logstash status
● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: enabled)
Active: failed (Result: exit-code) since ons 2017-12-20 14:16:34 CET; 16min ago
Process: 1444 ExecStart=/usr/share/logstash/bin/logstash --path.settings /etc/logstash (code=exited, status=143)
Main PID: 1444 (code=exited, status=143)

dec 20 14:16:25 systemd[1]: Started logstash.
dec 20 14:16:33 systemd[1]: Stopping logstash...
dec 20 14:16:34 systemd[1]: logstash.service: Main process exited, code=exited, status=143/n/a
dec 20 14:16:34 systemd[1]: Stopped logstash.
dec 20 14:16:34 systemd[1]: logstash.service: Unit entered failed state.
dec 20 14:16:34 systemd[1]: logstash.service: Failed with result 'exit-code'.

I temporary stopped logstash and trying to run it manually with debugging.

8

The problem is solved if i manually start logstash with

:/usr/share/logstash$ sudo bin/logstash --log.level=debug --path.settings /etc/logstash -f /etc/logstash/conf.d/logstash.conf

how can I config the service to use these default paths?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.