Filebeat multi source logs problem

Elastic Stack Beats
1

cat /etc/filebeat/filebeat.yml

filebeat.config.modules:
  path: /etc/filebeat/modules.d/*.yml
  reload.enabled: false
filebeat.inputs:
- type: log
  enabled: false
paths:
    - /var/log/*.log
    - /home/local/example/example.log
filebeat.registry_file: /var/lib/filebeat/registry
output.elasticsearch:
hosts: ["192.168.100.100:9200"]
ssl.certificate_authorities: ["/etc/pki/tls/certs/beats.crt"]
setup.template.settings:
  index:
    number_of_shards: 3
setup.kibana:
    host: "192.168.100.100:5601"
logging.to_files: true
logging.files:
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7

cat /etc/filebeat/modules.d/system.yml

- module: system
  syslog:
    enabled: true

    var.paths: ["/var/log/syslog*"]
    var.convert_timezone: true

  auth:
    enabled: true
    var.paths: ["/var/log/auth.log*"]

    var.convert_timezone: true

filebeat modules list

Enabled:
system

Disabled:
apache2
auditd
elasticsearch
icinga
iis
kafka
kibana
logstash
mongodb
mysql
nginx
osquery
postgresql
redis
traefik

Elk server show only
source: /var/log/syslog* and /var/log/auth.log*

Don't show:
source: /home/local/example/example.log

I need also /home/local/example/example.log source logs and syslog both.

What is my wrong?

2

Looking at your filebeat.yml I see that the inputs that you have defined for /home/local have the enabled flag set to false, you have to enable it. Look at my example below:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /home/local/example/example.log
3

I have some custom grok in logstash conf.d directory. That grok split /home/local/example/example.log path data. I wish - /var/log/*.log path data will be split via filebeat system module and /home/local/example/example.log path data will be split via custom grok.

If I configue filebeat.yum like

output.logstash:
hosts: ["192.168.10.1:5044"]

/home/local/example/example.log path data split field:
"Class" => "c.n.s.s.SmsGatewayService:331"
"source" => "/home/local/example/example.log",
"@timestamp" => 2018-10-03T09:35:57.404Z,
"Thread" => "http-nio-8080-exec-6",
"host" => {
"name" => "vm1"
},

But I configure filebeat like

setup.kibana:
host: "192.168.10.1:5601"

output.elasticsearch:
hosts: ["192.168.10.1:9200"]

/home/local/example/example.log path data will not be split:
message:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.

I need to split module data and custom path data via output elasticsearch and kibana host.

Thanks for help me

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.