Logstash dont send to output

ahiyaz · May 20, 2020, 5:08pm

its a new installation
im collecting sflow data using sflow plugin
debugging the sflow plugin i can see flows data getting to logstash
but nothing is being sent to the output.
ive tried to logz.io and to local file.
any ideas how to proceed ?
thanks

logstash service status

root@logstash-sapir:/etc/logstash# systemctl status logstash.service
? logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2020-05-20 17:05:38 UTC; 34s ago
Main PID: 28109 (java)
Tasks: 15 (limit: 2318)
CGroup: /system.slice/logstash.service
mq28109 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.he

May 20 17:05:38 logstash-sapir systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
May 20 17:05:38 logstash-sapir systemd[1]: logstash.service: Scheduled restart job, restart counter is at 6.
May 20 17:05:38 logstash-sapir systemd[1]: Stopped logstash.
May 20 17:05:38 logstash-sapir systemd[1]: Started logstash.
May 20 17:05:38 logstash-sapir logstash[28109]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be
May 20 17:05:42 logstash-sapir logstash[28109]: WARNING: An illegal reflective access operation has occurred
May 20 17:05:42 logstash-sapir logstash[28109]: WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstas
May 20 17:05:42 logstash-sapir logstash[28109]: WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
May 20 17:05:42 logstash-sapir logstash[28109]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
May 20 17:05:42 logstash-sapir logstash[28109]: WARNING: All illegal access operations will be denied in a future release

my conf file

input {
udp {
port => 6343
codec => sflow {
}
}
}
filter {

...

mutate {
add_field => { "token" => "token }
}
}

#output {

lumberjack {

hosts => ["listener"]

port => 5006

ssl_certificate => "/usr/share/logstash/keys/TrustExternalCARoot.crt"

codec => "json_lines"

}

#}

output {
file {
path => "/home/logstashadmin/logstash.json"
codec => "json"
}
}

ptamba · May 21, 2020, 6:46am

it appears that you logstash is restarting

what version of java and logstash are running
stop logstash from systemd and run it from CLI and provide the error message. please wrap your log with the </> for easier reading.
if you remove the codec from the input, does it work? if it does, you might want to make sure that your codec plugin is up to date
remove the codec

ahiyaz · May 21, 2020, 7:57am

hello again

Regarding removing the codec, do you mean the file output codec or the sflow which both didnt work.
Openjdk version - 11.0.7

Logstash - 7.7.0-1

Error i got whil running ./logstash
/ ERROR] 2020-05-21 07:44:01.257 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
/
starting log stash with cli: it seems it cant start this way

/systemctl stop logstash
root@logstash-sapir:/usr/share/logstash/bin# bin/logstash
bash: bin/logstash: No such file or directory
root@logstash-sapir:/usr/share/logstash/bin# cd ..
root@logstash-sapir:/usr/share/logstash# bin/logstash
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
ERROR: Failed to read pipelines yaml file. Location: /usr/share/logstash/config/pipelines.yml
usage:
bin/logstash -f CONFIG_PATH [-t] [-r] [-w COUNT] [-l LOG]
bin/logstash --modules MODULE_NAME [-M "MODULE_NAME.var.PLUGIN_TYPE.PLUGIN_NAME.VARIABLE_NAME=VALUE"] [-t] [-w COUNT] [-l LOG]
bin/logstash -e CONFIG_STR [-t] [--log.level fatal|error|warn|info|debug|trace] [-w COUNT] [-l LOG]
bin/logstash -i SHELL [--log.level fatal|error|warn|info|debug|trace]
bin/logstash -V [--log.level fatal|error|warn|info|debug|trace]
bin/logstash --help
[ERROR] 2020-05-21 07:51:41.197 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
root@logstash-sapir:/usr/share/logstash# systemctl status logstash
? logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2020-05-21 07:49:47 UTC; 3min 27s ago
Process: 26958 ExecStart=/usr/share/logstash/bin/logstash --path.settings /etc/logstash (code=exited, status=143)
Main PID: 26958 (code=exited, status=143)

May 21 07:49:24 logstash-sapir logstash[26958]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
May 21 07:49:28 logstash-sapir logstash[26958]: WARNING: An illegal reflective access operation has occurred
May 21 07:49:28 logstash-sapir logstash[26958]: WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9
May 21 07:49:28 logstash-sapir logstash[26958]: WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
May 21 07:49:28 logstash-sapir logstash[26958]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
May 21 07:49:28 logstash-sapir logstash[26958]: WARNING: All illegal access operations will be denied in a future release
May 21 07:49:47 logstash-sapir systemd[1]: Stopping logstash...
May 21 07:49:47 logstash-sapir systemd[1]: logstash.service: Main process exited, code=exited, status=143/n/a
May 21 07:49:47 logstash-sapir systemd[1]: logstash.service: Failed with result 'exit-code'.
May 21 07:49:47 logstash-sapir systemd[1]: Stopped logstash.
lines 1-16/16 (END)/

ahiyaz · May 21, 2020, 7:59am

starting it via systemctl do work:

/root@logstash-sapir:/usr/share/logstash#
root@logstash-sapir:/usr/share/logstash# systemctl start logstash
root@logstash-sapir:/usr/share/logstash# systemctl status logstash
? logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-05-21 07:57:52 UTC; 4s ago
Main PID: 27082 (java)
Tasks: 14 (limit: 2318)
CGroup: /system.slice/logstash.service
mq27082 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-

May 21 07:57:52 logstash-sapir systemd[1]: Started logstash.
May 21 07:57:52 logstash-sapir logstash[27082]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
May 21 07:57:55 logstash-sapir logstash[27082]: WARNING: An illegal reflective access operation has occurred
May 21 07:57:55 logstash-sapir logstash[27082]: WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9
May 21 07:57:55 logstash-sapir logstash[27082]: WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
May 21 07:57:55 logstash-sapir logstash[27082]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
May 21 07:57:55 logstash-sapir logstash[27082]: WARNING: All illegal access operations will be denied in a future release/

ahiyaz · May 21, 2020, 8:03am

one more thing for some reason i have no log file in my logstash
no in:
/root@logstash-sapir:/usr/share/logstash# cd /var/log/logstash/
root@logstash-sapir:/var/log/logstash# ls
root@logstash-sapir:/var/log/logstash#
/

ahiyaz · May 21, 2020, 8:30am

ive removed the codec from input, but still didnt work
/
input {
udp {
port => 6343
}
}
/

/root@logstash-sapir:/etc/logstash# systemctl status logstash
? logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-05-21 08:27:12 UTC; 8s ago
Main PID: 29188 (java)
Tasks: 14 (limit: 2318)
CGroup: /system.slice/logstash.service
mq29188 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-

May 21 08:27:12 logstash-sapir systemd[1]: logstash.service: Main process exited, code=exited, status=143/n/a
May 21 08:27:12 logstash-sapir systemd[1]: logstash.service: Failed with result 'exit-code'.
May 21 08:27:12 logstash-sapir systemd[1]: Stopped logstash.
May 21 08:27:12 logstash-sapir systemd[1]: Started logstash.
May 21 08:27:12 logstash-sapir logstash[29188]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
May 21 08:27:16 logstash-sapir logstash[29188]: WARNING: An illegal reflective access operation has occurred
May 21 08:27:16 logstash-sapir logstash[29188]: WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9
May 21 08:27:16 logstash-sapir logstash[29188]: WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
May 21 08:27:16 logstash-sapir logstash[29188]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
May 21 08:27:16 logstash-sapir logstash[29188]: WARNING: All illegal access operations will be denied in a future release/

ptamba · May 21, 2020, 8:55am

if you install using rpm or deb , logstash binaries should be in /usr/share/logstash/bin/

stop logstash from systemctl
run logstash from cli by :

/usr/share/logstash/bin/logstash -f <path-to-your-config-file>

ahiyaz · May 21, 2020, 9:06am

ahh i didnt specify a config file.
below is output with config file
/
/usr/share/logstash/bin/logstash -f /etc/logstash/logstash.yml
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[WARN ] 2020-05-21 09:01:12.652 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2020-05-21 09:01:12.667 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"7.7.0"}
[ERROR] 2020-05-21 09:01:14.613 [Converge PipelineAction::Create] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "input", "filter", "output" at line 28, column 1 (byte 579) after ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in block in compile_sources'", "org/jruby/RubyArray.java:2577:in map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in block in converge_state'"]}
[INFO ] 2020-05-21 09:01:15.285 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2020-05-21 09:01:19.837 [LogStash::Runner] runner - Logstash shut down.
root@logstash-sapir:/etc/logstash# systemctl status logstash
? logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2020-05-21 08:57:21 UTC; 4min 48s ago
Process: 31329 ExecStart=/usr/share/logstash/bin/logstash --path.settings /etc/logstash (code=exited, status=143)
Main PID: 31329 (code=exited, status=143)

May 21 08:57:19 logstash-sapir systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
May 21 08:57:19 logstash-sapir systemd[1]: logstash.service: Scheduled restart job, restart counter is at 40.
May 21 08:57:19 logstash-sapir systemd[1]: Stopped logstash.
May 21 08:57:19 logstash-sapir systemd[1]: Started logstash.
May 21 08:57:19 logstash-sapir logstash[31329]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
May 21 08:57:21 logstash-sapir systemd[1]: Stopping logstash...
May 21 08:57:21 logstash-sapir systemd[1]: logstash.service: Main process exited, code=exited, status=143/n/a
May 21 08:57:21 logstash-sapir systemd[1]: logstash.service: Failed with result 'exit-code'.
May 21 08:57:21 logstash-sapir systemd[1]: Stopped logstash.
/
it cant find the config file although i put the path

ahiyaz · May 21, 2020, 9:07am

here ive used the --path.settings /etc/logstash/ option
/
/usr/share/logstash/bin/logstash -f /etc/logstash/logstash.yml --path.settings /etc/logstash/
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.headius.backport9.modules.Modules (file:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.11.1.jar) to method sun.nio.ch.NativeThread.signal(long)
WARNING: Please consider reporting this to the maintainers of com.headius.backport9.modules.Modules
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
ERROR: Failed to parse YAML file "/etc/logstash/logstash.yml". Please confirm if the YAML structure is valid (e.g. look for incorrect usage of whitespace or indentation). Aborting... parser_error=>(): could not find expected ':' while scanning a simple key at line 275 column 1
[ERROR] 2020-05-21 09:05:29.216 [main] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
root@logstash-sapir:/etc/logstash# systemctl status logstash
? logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2020-05-21 08:57:21 UTC; 8min ago
Process: 31329 ExecStart=/usr/share/logstash/bin/logstash --path.settings /etc/logstash (code=exited, status=143)
Main PID: 31329 (code=exited, status=143)

May 21 08:57:19 logstash-sapir systemd[1]: logstash.service: Service hold-off time over, scheduling restart.
May 21 08:57:19 logstash-sapir systemd[1]: logstash.service: Scheduled restart job, restart counter is at 40.
May 21 08:57:19 logstash-sapir systemd[1]: Stopped logstash.
May 21 08:57:19 logstash-sapir systemd[1]: Started logstash.
May 21 08:57:19 logstash-sapir logstash[31329]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
May 21 08:57:21 logstash-sapir systemd[1]: Stopping logstash...
May 21 08:57:21 logstash-sapir systemd[1]: logstash.service: Main process exited, code=exited, status=143/n/a
May 21 08:57:21 logstash-sapir systemd[1]: logstash.service: Failed with result 'exit-code'.
May 21 08:57:21 logstash-sapir systemd[1]: Stopped logstash.
/

ahiyaz · May 21, 2020, 9:11am

but why it starting with systemctl ?
it seems like my installation is damaged aah?
i used this guide -- https://docs.logz.io/shipping/shippers/logstash.html

/
root@logstash-sapir:/etc/logstash# systemctl start logstash
root@logstash-sapir:/etc/logstash# systemctl status logstash
? logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-05-21 09:08:55 UTC; 3s ago
Main PID: 31509 (java)
Tasks: 14 (limit: 2318)
CGroup: /system.slice/logstash.service
mq31509 /usr/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-

May 21 09:08:55 logstash-sapir systemd[1]: Started logstash.
May 21 09:08:55 logstash-sapir logstash[31509]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
lines 1-10/10 (END)
/

ahiyaz · May 21, 2020, 9:12am

ill try now to remove my sflow input code

ahiyaz · May 21, 2020, 9:25am

still not starting via cli

ptamba · May 21, 2020, 10:13am

logstash.yml is used for configuring settings of logstash, and it’s not config files. read here for logstash.yml

your config file should be another file. save your config in separate file and reference them with cli (logstash -f). a config contains input, filter and output as specified.

it’s actually not starting but it’s looping.

ahiyaz · May 21, 2020, 10:51am

ok Thanks!!
should were do i point to new configuration yml so it will all ways load with it?

ptamba · May 21, 2020, 11:31am

in deb or rpm systems on a default install and systemd:

logstash will search for logstash.yml and pipelines.yml in /etc/logstash
you can store your config files in /etc/logstash/conf.d . any files with .conf extension in this directory will be loaded automatically. you can confirm this by looking at /etc/logstash/pipelines.yml . there should be a pipeline called main with config path.
you can see further how logstash controls pipelines in the documentation

ahiyaz · May 21, 2020, 11:48am

thanks so much for your help
its working now

system · June 18, 2020, 11:49am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.