ELK + Filebeat

Elastic Stack Elasticsearch
1

Hello,

I am a University student and my project is on the ELK Stack and I have followed the implementation on your Website from Elastic Search, Kibana and Logstash.
I understood that I have to send from my servers the Logs to Logstash via Filebeat but when I am testing Filebeat to send Logs on the port 5044 the port is activated but it is not reachable.
Is there any modification should I do so that the ELK stack works properly?

You can see here the pipeline created in /usr/share/logstash:
input {
beats {
port => "5044"
}
}

The filter part of this file is commented out to indicate that it is

optional.

filter {

}

output {
stdout { codec => rubydebug }
}

I just created this pipeline to test if Filebeat was working as mentioned in your tutorial.

Rhe configuration of the filebeat.yml:
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:

Array of hosts to connect to.

#hosts: ["localhost:9200"]

Optional protocol and basic auth credentials.

#protocol: "https"
#username: "elastic"
#password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:

The Logstash hosts

hosts: ["localhost:5044"]

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

I deactivated the Elastic Search output I activated the Logstash output and I put the path of the log file that is in your tutorial and still I don't get any answer .

This is the answer that I am getting:

adel@ubuntu:/usr/share/logstash$ sudo bin/logstash -f first-pipeline.conf --config.reload.automatic
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-02-25 23:38:06.264 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[INFO ] 2018-02-25 23:38:06.674 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[WARN ] 2018-02-25 23:38:09.533 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-02-25 23:38:11.096 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.2"}
[INFO ] 2018-02-25 23:38:13.990 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2018-02-25 23:38:18.013 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2018-02-25 23:38:20.856 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2018-02-25 23:38:21.248 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x16695b4b@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
[INFO ] 2018-02-25 23:38:21.461 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2018-02-25 23:38:21.481 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}

Thank you

2

Please format your code using </> icon as explained in this guide. It will make your post more readable and will help us help you.

Alternatively use markdown style like this:

```
CODE
```
3

You can see here the pipeline created in /usr/share/logstash:
........................................................................................................................................................
input {
beats {
port => "5044"
}
}
The filter part of this file is commented out to indicate that it is
optional.
filter {

}
output {
stdout { codec => rubydebug }
}
........................................................................................................................................................

The configuration of the filebeat.yml:
........................................................................................................................................................

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
Array of hosts to connect to.
#hosts: ["localhost:9200"]
Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
output.logstash:
The Logstash hosts
hosts: ["localhost:5044"]
Optional SSL. By default is off.
List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
........................................................................................................................................................

I deactivated the Elastic Search output I activated the Logstash output and I put the path of the log file that is in your tutorial and still I don't get any answer .

This is the answer that I am getting:
........................................................................................................................................................
adel@ubuntu:/usr/share/logstash$ sudo bin/logstash -f first-pipeline.conf --config.reload.automatic
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2018-02-25 23:38:06.264 [main] scaffold - Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[INFO ] 2018-02-25 23:38:06.674 [main] scaffold - Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[WARN ] 2018-02-25 23:38:09.533 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2018-02-25 23:38:11.096 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.2.2"}
[INFO ] 2018-02-25 23:38:13.990 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}
[INFO ] 2018-02-25 23:38:18.013 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>2, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[INFO ] 2018-02-25 23:38:20.856 [[main]-pipeline-manager] beats - Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2018-02-25 23:38:21.248 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] pipeline - -Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x16695b4b@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:246 sleep>"}
[INFO ] 2018-02-25 23:38:21.461 [[main]<beats] Server - Starting server on port: 5044
[INFO ] 2018-02-25 23:38:21.481 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Pipelines running {:count=>1, :pipelines=>["main"]}
........................................................................................................................................................
Thank you

4

I separated all parts with "........".
If any other information needed to solve my problem will be sent directly..

Thank you

5

Please format your code, logs or configuration files using </> icon as explained in this guide and not the citation button. It will make your post more readable.

Or use markdown style like:

```
CODE
```

There's a live preview panel for exactly this reasons.

Lots of people read these forums, and many of them will simply skip over a post that is difficult to read, because it's just too large an investment of their time to try and follow a wall of badly formatted text.
If your goal is to get an answer to your questions, it's in your interest to make it as easy to read and understand as possible.
Please update your post.

6

You can see here the pipeline created in /usr/share/logstash:

The configuration of the filebeat.yml:

I deactivated the Elastic Search output I activated the Logstash output and I put the path of the log file that is in your tutorial and still I don't get any answer .

This is the answer that I am getting:

Thank you

7

Hello,

Can anyone tell me where the problem is because this problem isn't letting me advance in my University Project.

Thank you

8

What do you mean by this? Can you show filebeat logs?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.