Filebeat vs Logstash

Elastic Stack Logstash
1

Please give me the solution for forwarding the log to logstash from filebeat, were the filebeat and logstash are in different VM

am getting the error of :
2018-04-10T11:39:05.750Z ERROR pipeline/output.go:74 Failed to connect: dial tcp 192.168.2.5:5044: getsockopt: connection refused

I just want to use the only filebeat and logstash.

2

Is logstash running?

3

yes, its running

4

Any chance logstash is only listening on localhost?
Can you ping from the filebeat vm to the logstash vm and vise versa?

5

please share your logstash pipeline configuration here..??

6

its doing well

7

it is in /etc/logstash/conf.d/first.conf

The # character at the beginning of a line indicates a comment. Use

comments to describe your configuration.

input {
beats {
port => "5044"
}
}

The filter part of this file is commented out to indicate that it is

optional.

filter {

}

output {
stdout { codec => rubydebug }
}
~

8

Please format logs, configs and terminal input/output using the </>-Button or markdown code fences. This forum uses Markdown to format posts. Without proper formatting, it can be very hard to read your posts. Proper formatting helps us to help you.

The conection refused normally appears because the remote host did refuse the connection. Service not running, port not available, firewall blocking connection.

Running from the filebeat host can you show us the output of:

ping 192.168.2.5

Also check with telnet:

telnet 192.168.2.5 5044
9

Also share the filebeat.yml file here..?

10

Now getting:
018-04-12T06:32:46.770Z ERROR pipeline/output.go:74 Failed to connect: dial tcp 192.168.2.81:5044: getsockopt: connection refused

filebeat.yml

filebeat.prospectors:

  • type: log

    enabled: true

    • /var/log/*.log
    • /var/log//.log

output.logstash:

hosts: ["192.168.2.81:5044"]

file: /etc/logsatash/logstash.yml

path.data: /var/lib/logstash

path.logs: /var/log/logstash

file: /usr/share/logstash/logstash.conf

input {
beats {
hosts => "127.0.0.1"
port => 5044
}
}
filter
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
output {
stdout { codec => rubydebug }
}

11

Hi @jawad846,

Lets go step by step.

In your logstash.conf you have mentioned "hosts". please comment this or remove from the file. it should be like :

beats {
port => 5044
}

Please do this and restart the service and let me know the error if you are getting.

Thanks,
Harsh Bajaj

12

the time of restart:
[logstash@ip-192-168-2-81 logstash]$ sudo service logstash start
logstash: unrecognized service
[logstash@ip-192-168-2-81 logstash]$

13

Hi @jawad846,

please try to start with "Systemctl" or check the status if already start then restart it not start.

14

am using aws linux.

it also not working

and thanks for giving the support

15

were you able to start the service earlier with that command which you mentioned.

16

/bin/logstash -f logstash.conf

17

Please try to start with same command as you were using earlier and check.

18

WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-04-12 07:30:05.634 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[INFO ] 2018-04-12 07:30:05.647 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[WARN ] 2018-04-12 07:30:06.196 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-04-12 07:30:06.390 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.3"}
[INFO ] 2018-04-12 07:30:06.546 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[ERROR] 2018-04-12 07:30:06.597 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, { at line 7, column 4 (byte 54) after filter \n ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:incompile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in block in compile_sources'", "org/jruby/RubyArray.java:2486:inmap'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:inwith_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in block in converge_state'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:inblock in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:90:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

this is RESULT

19

this is the architecture

00%20PM

all the logs need to be save in a single file

20

As i can see your issue is with Logstash is not working your issue is not related to filebeat right?

and in above logs i can see conf file path incorrect.

please run below command with conf file path like below.

/bin/logstash -f /path/logstash.conf

Thanks,
Harsh Bajaj