Hi,
I have below app logs, like below but they are not in oder in kibana i.e the last log line below is not the first in kibana discover.
2021-10-27 16:51:14.864 | INFO | | | | | [http-nio-8080-exec-32] | next-concapcont - selected account ids : [GB22QNBA0935005402|4303004002]
2021-10-27 16:51:15.138 | INFO | | | | | [http-nio-8080-exec-32] | QueryTranslatorFactoryInitiator - HHH000397: Using ASTQueryTranslatorFactory
2021-10-27 16:51:15.358 | INFO | next-con | bce8dbe3-0f-4365-b906-48b2e794a6 | | | [http-nio-8080-exec-32] | next-concapcont - user_unit_id: LMS
2021-10-27 16:51:15.358 | INFO | next-con | bce8dbe3-0f-4365-b906-48b2e794a6 | | | [http-nio-8080-exec-32] | next-concapcont - corporate_id: null
2021-10-27 16:51:15.358 | INFO | next-con | bce8dbe3-0f-4365-b906-48b2e794a6 | | | [http-nio-8080-exec-32] | next-concapcont - consentStatus: accept
This is /etc/filebeat/filebeat.yml
capturing logs
.
- type: log
fields_under_root: true
fields:
log_type: nextgen-api_app_server1
app_id: app
multiline.pattern: ^[[:space:]]+(at|\.{3})\b|^Caused by:|^java|^...|^-
multiline.negate: true
multiline.match: after
paths:
- /opt/tomcat/logs/catalina.out
below is the pattern file /etc/logstash/patterns/custom.txt
.
MY_DATE_PATTERN %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:%{MINUTE}(?::?%{SECOND})
custom.txt
is used in /etc/logstash/conf.d/logstash.conf
file,
if [log_type] == "nextgen-api_app_server1" and [app_id] == "app"
{
mutate { gsub => ["message","\|"," "] } grok { patterns_dir => ["/etc/logstash/patterns"] match => { "message" => "%{MY_DATE_PATTERN:timestamp}%{SPACE}%{LOGLEVEL:level}%{SPACE}%{UUID:ConsentID}%{SPACE}%{WORD:TraceID}%{SPACE}%{WORD:TransactionID}%{SPACE}%{GREEDYDATA:messagetext}" } }
mutate {
replace => {
"[type]" => "nextgen-api_app_server1"
}
}
}
I want to order the logs as per the timestamp
in message.
I know above MY_DATE_PATTERN
Is not correct, I have changed it like below, but this is not working.
MY_DATE_PATTERN %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}.%{SSS}
there is no date
filter used in logstash
so I assume currently the logs are getting ordered in with the time they are coming in the pipeline and hence they not appearing in order and hence I want to oder them based on the timestamp in message.