Filebeat system module

Elastic Stack Beats
1

Hello Team,

I am using ELK 6.4.0 and Beats (Filebeat, Metricbeat) 6.4.0. Currently my architecture is Beat->Logstash->Elasticsearch->Kibana.

I am using Filebeat System module and ouput in filebeat.yml is logstash. In this case, logs are coming on kibana dashboard via system module. But logs are not showing in Filebeat syslog and SSH dashboard.

But when i am replacing output logstash with elasticsearch in filebeat.yml file then logs are coming on kibana dashboard as well as showing on Syslog and SSH filebeat dashboard.

is logstash unable to add those required fileds or something else?

I want to use logstash and Filebeat Syslog and SSH dashboard so that we can easily visualize the SSH loging attempt like success and failed with geolocation.

Please help me to troubleshoot the issue.

I am getting one more error when running the below command on filebeat server:

filebeat -e -modules=nginx -setup -E "output.elasticsearch.hosts=["http://localhost:9200"]"

Error:

ERROR	registrar/registrar.go:363	Writing of registry returned error: rename /var/lib/filebeat/registry.new /var/lib/filebeat/registry: no such file or directory. Continuing...

I have checked /var/lib/filebeat/registry file is present on the server.

Sorry, if it is not totally clear. In learning phase of ELK.

Thanks in advance.

2

@Tek_Chand all of the modules inside Filebeat relies on the ingest node which is an Elasticsearch feature to do the transformation on the original data. When you start Filebeat with a module we automatically configure the remote ingest pipeline for the transformation.

When you add Logstash between Filebeat and Elasticsearch you are losing capabilities that you have to either do one of the following:

  • If you don't need Logstash just keep Filebeat -> Elasticsearch
  • Convert the ingest pipeline found in the ingest directory into a Logstash pipeline manually or using the migration tool.
  • Configure the elasticsearch output with condition on your data to send the data to to the ingest pipeline.

It's still a bit more involved to add Logstash in the middle and we aim to provide a better path in the future.

3

For the other error, what distribution are you running?
Can you add the output of which filebeat to this thread?

4

@Pier, Thanks for your response.

I am running Ubuntu 16.04 and Filebeat version is 6.4.0.

admin@xyz:~$ which filebeat
/usr/bin/filebeat

https://www.elastic.co/guide/en/logstash/5.6/filebeat-modules.html

I am following above link so i can use logstash with filebeat module. And i am getting above error when running the command in first step.

But logs are reaching on Kibana dashboard from filebeat only getting single error when running the command in first step.

Thanks.

5

@Pier, Can you please let me know the reason for above mentioned error? i,e

I have tested it on my testing environment but issue is not replicate there. But facing issue in my production environment.
Thanks.

6

I would assume it's either an error related to access rights or the directory indeed does not exist.

7

@Rufflin, Thank you for your response.

Please find the current directory and permission:

root@xyz:/var/lib# ls -lah | grep file
drwxr-x---  2 root    root    4.0K Oct  1 07:05 filebeat
root@xyz:/var/lib/filebeat# ls -lah
total 16K
drwxr-x---  2 root root 4.0K Oct  1 07:05 .
drwxr-xr-x 51 root root 4.0K Sep  5 06:11 ..
-rw-------  1 root root   48 Aug  6 10:30 meta.json
-rw-------  1 root root  717 Oct  1 07:05 registry
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.