Filebeat vs Logstash

Elastic Stack Logstash
21

i run this command from the path folder,
i saved logstash.conf in /bin/logstash
i run from on that folder

22

Pls save your file in conf.d folder instead of bin folder.

23

done,

getting this error in filebeat
2018-04-12T08:18:28.356Z ERROR pipeline/output.go:74 Failed to connect: dial tcp 192.168.2.79:5044: getsockopt: connection refused

24

Please make sure that /etc/logstash/logstash.yml(you have a typo on your replies, /etc/logastash/,so can't be sure if it was a typo or folder is named like this) exists with the paths to data and logs being correct.
After running /bin/logstash -f /path/to/logstash.conf of yours try:
nestat -anutp | grep 5044
and give us the output.

Also any chance you have installed the xpack plugin on logstash?

25

Hi @jawad846 please check that you have mentioned correct Ip in your filebeat.yml. bcoz in your filebeat.yml Ip is 81 and in error its its 79.

can you pls check and confirm the same.

Thanks,
Harsh Bajaj

26

filebeat is @ 192.168.2.223
and

logstash is @ 192.168.2.79

@harshbajaj16

27

root@ip-192-168-2-79:/usr/share/logstash# bin/logstash -f /usr/share/logstash/logs.conf
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-04-12 09:09:10.902 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ] 2018-04-12 09:09:11.227 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ] 2018-04-12 09:09:23.459 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-04-12 09:09:28.562 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.3"}
[INFO ] 2018-04-12 09:09:36.210 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[ERROR] 2018-04-12 09:09:39.862 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 7, column 2 (byte 45) after filter\n\t", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

netstat -anutp | grep 5044
this command is giving any answer,

I main issue i realised that, port is not open

@sancroth

root@ip-192-168-2-79:/usr/share/logstash# netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1331/sshd
tcp6 0 0 :::22 :::* LISTEN 1331/sshd

28

And in filebeat.yml you have give the right IP of logstash o/p which is :

bcoz in earlier file you given 81 IP...?

29

i were working with aws linux, now am testing in ubuntu

@harshbajaj16

30

As you can see there is a typo in your logstash configuration
LogStash::ConfigurationError", :message=>"Expected one of #, { at line 7, column 2 (byte 45) after filter\n\t"
that makes logstash fail.
Thus the port is nevr open since logstash is not actually working.
Could you give me your configurations in a formated way so i can read them?
Try pastebin.com

31

when i run the command
/filebeat -e apache2
the apache2 module is enabled

the o/p

2018-04-12T09:02:11.564Z ERROR pipeline/output.go:74 Failed to connect: dial tcp 192.168.2.79:5044: getsockopt: connection refused
2018-04-12T09:02:13.585Z ERROR pipeline/output.go:74 Failed to connect: dial tcp 192.168.2.79:5044: getsockopt: connection refused
32

u mean logstash.yml file?

@sancroth

33

This is expected behavior since logstash is currently dead.
Please do what i said on my last post.
Copy and paste your logstash.conf(input-filters-output file) on pastebin.com , save it for an hour or more and give us your link.

34

ohk check one more thing that your logstash is working and running up or not???

35

It's not working, he posted his logs from logstash. There is a configuration error that gives the typical logstash start-die loop.

36

@sancroth

file---> /usr/share/logstash/logstash.conf

input {
beats {
port => 5044
}
}
filter
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
output {
stdout { codec => rubydebug }
}

37

input {
beats {
port => 5044
}
}
filter{ # <- here
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
output {
stdout { codec => rubydebug }
}

Your filter is missing the opening curly bracket "{". The above is correct

38

@sancroth here @jawad846 confirm that "done" means logstash is working ....???

39

done, and restarted the logstash

and run this command:
bin/logstash -f /usr/share/logstash/logstash.conf

40

ohk now try to restart the filebeat and check.