Index is loaded with data from other indexes

[bueka.torao (Bueká Torao)]

Hi all, I need some help here!

I am reading logs from my FTP server, also from some different servers I am using filebeats + logstash + elasticsearch + kibana.

Mail server -> filebeats -> logstash 5045 index maillog-*
FTP server -> filebeats -> logstash 5046 index pureftp-*

Filebeats sends data to logstash using a separate port 5046 for ftp logs. I created a filter and index for ftp logs. The FTP server is the only one sending logs to that specific port. however, the mapping at kibana shows the maillog index mapping. And, the pureftp index shows maillog data.

Why is that happing?

MAIL server name: relaysrv
Filebeats configuration:
filebeat.inputs:

• type: log
  enabled: true
  paths:
  • /var/log/*.log
  • /var/log/maillog
    filebeat.config.modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true
    setup.template.settings:
    index.number_of_shards: 3
    setup.kibana:
    output.logstash:
    hosts: ["192.XX.XX.XX:5045"]

Logstash configuration:
input {
beats {
port => 5045
}
}
filter {
grok {
patterns_dir => ["/etc/logstash/patterns.d"]
match => { "message" => ["%{PF}" ] }
}
date {
match => [ "timestamp", "MMM dd HH:mm:ss", "MMM d HH:mm:ss" ]
}
geoip {
source => "relayip"
}
}
output {
elasticsearch {
index => "maillog-%{+YYYY.MM.dd}"
hosts => ["127.0.0.1:9200"]
manage_template => true
template_overwrite => true
template => "/etc/logstash/templates/maillog.json"
template_name => "maillog"
}
}

FTP server name: ftp7srv
Filebeats configuration:

filebeat.inputs:

• type: log
  enabled: true
  paths:
  • /var/log/pureftpd.log
    filebeat.config.modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true
    setup.template.settings:
    index.number_of_shards: 3
    setup.kibana:
    output.logstash:
    hosts: ["192.xxx.xx.xx:5046"]

Logstash configuration:

input {
beats {
port => 5046
}
}
filter {
grok {
patterns_dir => "/etc/logstash/patterns.d"
match => ["message", "%{PUREFTPTRANSFER}"]
add_tag => ["pureftp_transfer", "pureftp_parse"]
}
grok {
patterns_dir => "/etc/logstash/patterns.d"
match => ["message", "%{PUREFTPAUTHFAIL}"]
add_tag => ["pureftp_auth_fail", "pureftp_parse"]
}
grok {
patterns_dir => "/etc/logstash/patterns.d"
match => ["message", "%{PUREFTPAUTHSUCCESS}"]
add_tag => ["pureftp_auth_success", "pureftp_parse"]
}
if "pureftp_parse" not in [tags] {
grok {
patterns_dir => "/etc/logstash/patterns.d"
match => ["message", "%{PUREFTPSYSLOG}"]
add_tag => ["pureftp_syslog", "pureftp_parse"]
}
}
mutate {
gsub => ["remote", "_", ":"]
}
date {
timezone => "Europe/Amsterdam"
match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z", "MMM d HH:mm:ss", "MMM dd HH:mm:ss", "ISO8601"]
}
geoip {
source => "remote"
}
}
output {
elasticsearch {
index => "pureftp-%{+YYYY.MM.dd}"
hosts => ["127.0.0.1:9200"]
manage_template => true
template_overwrite => true
template => "/etc/logstash/templates/pureftp.json"
template_name => "pureftp"
}
}

Filebeat Version: 6.4