Hi,
I am using multiline option in filebeat, the issue is its creating new index with the same name for each hits. And also if i use @timestamp as "time filter field name" it will not show any logs, if i use "I dont want to use time filter" option then only it will show the logs".
My filebeat.yml file:
filebeat.prospectors:
type: log
enabled: true
paths:
C:\elk\logs\syslog.log
fields:
document_type: prod
include_lines: ['^TID','^ERR', '^WARN']
multiline.pattern: '^TID'
multiline.negate: true
multiline.match: after
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
setup.template.settings:
index.number_of_shards: 3
setup.kibana:
host: "localhost:5601"
output.logstash:
hosts: ["localhost:5044"]
logging.level: debug
logging.selectors: ["*"]
My logstash.conf file:
input {
beats{
port => 5044
}
}
filter {
grok {
match => [ "message", "TID:%{SPACE}[%{INT:tenant_id}]%{SPACE}[]%{SPACE}[%{TIMESTAMP_ISO8601:event_timestamp}]%{SPACE}%{LOGLEVEL:level}%{SPACE}{%{JAVACLASS:java_class}}%{SPACE}-%{SPACE}%{GREEDYDATA:log_message}" ]
}
date{
match => [ "event_timestamp", "ISO8601","yyyy-MM-dd HH:mm:ss.SSSZ","yyyy-MM-dd'T'HH:mm:ss.SSSZ"]
timezone => "Asia/Kolkata"
}
mutate {
remove_field => [ "message" ]
}
}
output {
if[fields][document_type] == "uit" {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "uit-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} else if[fields][document_type] == "prod" {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "prod-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
}
Please help me to resolve this issue.