Multiline logs are not displaying in kibana


(Vaidehi9039) #1

Hi,

I am using multiline option in filebeat, the issue is its creating new index with the same name for each hits. And also if i use @timestamp as "time filter field name" it will not show any logs, if i use "I dont want to use time filter" option then only it will show the logs".

My filebeat.yml file:

filebeat.prospectors:

type: log

enabled: true
paths:

C:\elk\logs\syslog.log
fields:
document_type: prod
include_lines: ['^TID','^ERR', '^WARN']

multiline.pattern: '^TID'
multiline.negate: true
multiline.match: after

filebeat.config.modules:

path: ${path.config}/modules.d/*.yml

reload.enabled: true

setup.template.settings:
index.number_of_shards: 3

setup.kibana:
host: "localhost:5601"

output.logstash:
hosts: ["localhost:5044"]

logging.level: debug
logging.selectors: ["*"]

My logstash.conf file:

input {
beats{
port => 5044
}
}

filter {
grok {
match => [ "message", "TID:%{SPACE}[%{INT:tenant_id}]%{SPACE}[]%{SPACE}[%{TIMESTAMP_ISO8601:event_timestamp}]%{SPACE}%{LOGLEVEL:level}%{SPACE}{%{JAVACLASS:java_class}}%{SPACE}-%{SPACE}%{GREEDYDATA:log_message}" ]
}
date{
match => [ "event_timestamp", "ISO8601","yyyy-MM-dd HH:mm:ss.SSSZ","yyyy-MM-dd'T'HH:mm:ss.SSSZ"]
timezone => "Asia/Kolkata"
}
mutate {
remove_field => [ "message" ]
}
}

output {
if[fields][document_type] == "uit" {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "uit-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
} else if[fields][document_type] == "prod" {
elasticsearch {
hosts => "localhost:9200"
manage_template => false
index => "prod-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
}

Please help me to resolve this issue.


(CJ Cenizal) #2

Hi there, could you please format your configuration snippets inside of a code block, using triple backticks? This will preserve whitespace, making it easier to read your config and answer your question.

```
Your config here
```

Thanks,
CJ


(CJ Cenizal) #3

Can you take a look at the _source of one of your events to see if @timestamp is being parsed correctly in Logstash?

Thanks,
CJ


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.