Index is loaded with data from other indexes


(Buek√° Torao) #1

Hi all, I need some help here!

I am reading logs from my FTP server, also from some different servers I am using filebeats + logstash + elasticsearch + kibana.

Mail server -> filebeats -> logstash 5045 index maillog-*
FTP server -> filebeats -> logstash 5046 index pureftp-*

Filebeats sends data to logstash using a separate port 5046 for ftp logs. I created a filter and index for ftp logs. The FTP server is the only one sending logs to that specific port. however, the mapping at kibana shows the maillog index mapping. And, the pureftp index shows maillog data.

Why is that happing?

MAIL server name: relaysrv
Filebeats configuration:
filebeat.inputs:

  • type: log
    enabled: true
    paths:
    • /var/log/*.log
    • /var/log/maillog
      filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: true
      setup.template.settings:
      index.number_of_shards: 3
      setup.kibana:
      output.logstash:
      hosts: ["192.XX.XX.XX:5045"]

Logstash configuration:
input {
beats {
port => 5045
}
}
filter {
grok {
patterns_dir => ["/etc/logstash/patterns.d"]
match => { "message" => ["%{PF}" ] }
}
date {
match => [ "timestamp", "MMM dd HH:mm:ss", "MMM d HH:mm:ss" ]
}
geoip {
source => "relayip"
}
}
output {
elasticsearch {
index => "maillog-%{+YYYY.MM.dd}"
hosts => ["127.0.0.1:9200"]
manage_template => true
template_overwrite => true
template => "/etc/logstash/templates/maillog.json"
template_name => "maillog"
}
}

FTP server name: ftp7srv
Filebeats configuration:

filebeat.inputs:

  • type: log
    enabled: true
    paths:
    • /var/log/pureftpd.log
      filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: true
      setup.template.settings:
      index.number_of_shards: 3
      setup.kibana:
      output.logstash:
      hosts: ["192.xxx.xx.xx:5046"]

Logstash configuration:

input {
beats {
port => 5046
}
}
filter {
grok {
patterns_dir => "/etc/logstash/patterns.d"
match => ["message", "%{PUREFTPTRANSFER}"]
add_tag => ["pureftp_transfer", "pureftp_parse"]
}
grok {
patterns_dir => "/etc/logstash/patterns.d"
match => ["message", "%{PUREFTPAUTHFAIL}"]
add_tag => ["pureftp_auth_fail", "pureftp_parse"]
}
grok {
patterns_dir => "/etc/logstash/patterns.d"
match => ["message", "%{PUREFTPAUTHSUCCESS}"]
add_tag => ["pureftp_auth_success", "pureftp_parse"]
}
if "pureftp_parse" not in [tags] {
grok {
patterns_dir => "/etc/logstash/patterns.d"
match => ["message", "%{PUREFTPSYSLOG}"]
add_tag => ["pureftp_syslog", "pureftp_parse"]
}
}
mutate {
gsub => ["remote", "_", ":"]
}
date {
timezone => "Europe/Amsterdam"
match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]
}
geoip {
source => "remote"
}
}
output {
elasticsearch {
index => "pureftp-%{+YYYY.MM.dd}"
hosts => ["127.0.0.1:9200"]
manage_template => true
template_overwrite => true
template => "/etc/logstash/templates/pureftp.json"
template_name => "pureftp"
}
}

Filebeat Version: 6.4


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.