Filebeat multi source logs problem

cat /etc/filebeat/filebeat.yml

filebeat.config.modules:
  path: /etc/filebeat/modules.d/*.yml
  reload.enabled: false
filebeat.inputs:
- type: log
  enabled: false
paths:
    - /var/log/*.log
    - /home/local/example/example.log
filebeat.registry_file: /var/lib/filebeat/registry
output.elasticsearch:
hosts: ["192.168.100.100:9200"]
ssl.certificate_authorities: ["/etc/pki/tls/certs/beats.crt"]
setup.template.settings:
  index:
    number_of_shards: 3
setup.kibana:
    host: "192.168.100.100:5601"
logging.to_files: true
logging.files:
rotateeverybytes: 10485760 # = 10MB
keepfiles: 7

cat /etc/filebeat/modules.d/system.yml

- module: system
  syslog:
    enabled: true

    var.paths: ["/var/log/syslog*"]
    var.convert_timezone: true

  auth:
    enabled: true
    var.paths: ["/var/log/auth.log*"]

    var.convert_timezone: true

filebeat modules list

Enabled:
system

Disabled:
apache2
auditd
elasticsearch
icinga
iis
kafka
kibana
logstash
mongodb
mysql
nginx
osquery
postgresql
redis
traefik

Elk server show only
source: /var/log/syslog* and /var/log/auth.log*

Don't show:
source: /home/local/example/example.log

I need also /home/local/example/example.log source logs and syslog both.

What is my wrong?

Looking at your filebeat.yml I see that the inputs that you have defined for /home/local have the enabled flag set to false, you have to enable it. Look at my example below:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /home/local/example/example.log

I have some custom grok in logstash conf.d directory. That grok split /home/local/example/example.log path data. I wish - /var/log/*.log path data will be split via filebeat system module and /home/local/example/example.log path data will be split via custom grok.

If I configue filebeat.yum like

output.logstash:
hosts: ["192.168.10.1:5044"]

/home/local/example/example.log path data split field:
"Class" => "c.n.s.s.SmsGatewayService:331"
"source" => "/home/local/example/example.log",
"@timestamp" => 2018-10-03T09:35:57.404Z,
"Thread" => "http-nio-8080-exec-6",
"host" => {
"name" => "vm1"
},

But I configure filebeat like

setup.kibana:
host: "192.168.10.1:5601"

output.elasticsearch:
hosts: ["192.168.10.1:9200"]

/home/local/example/example.log path data will not be split:
message:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.

I need to split module data and custom path data via output elasticsearch and kibana host.

Thanks for help me

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.