Hi there,
I'm trying to translate my logstash configuration for using filebeat and the ingest pipeline feature.
While using logstash, I had the following configuration:
---- LOGSTASH -----
input:
codec => multiline { pattern => "%{SYSLOG5424SD}:%{DATESTAMP}].*" negate => "true" what => "previous"
filter:
grok { match => { 'message' => '%{SYSLOG5424SD:thread}:%{DATESTAMP:datetime}] %{LOGLEVEL:level}\s{1,}\[id:%{GREEDYDATA:solid_rid}\] %{JAVACLASS:class}: %{GREEDYDATA:log_message}
---- FILEBEAT + INGEST NODE ----
Now, for filebeat, I'm using the following prospector (I tried to translate 1 to 1 the pattern from logstash)
multiline.pattern: '\[.*?\]+:(?:(?:1[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]).(?:0?[1-9]|1[0-2]).(20)\d{2}\s*(?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:(?:[0-5][0-9]|60)(?:[:.,][0-9]+)?)\].*' multiline.negate: true multiline.match: before
And the following pipeline has been configured:
"processors" : [
{
"grok": {
"field" : "message",
"patterns" : [
"%{SYSLOG5424SD:thread}:%{DATESTAMP:timestamp}\] %{LOGLEVEL:level}\s{1,}\[id:%{GREEDYDATA:solid_rid}\] %{JAVACLASS:class}: %{GREEDYDATA:log_message}"
]
},
{
"date" : {
"field" : "timestamp",
"formats" : ["dd.MM.YYYY HH:mm:ss.SSS"]
} } ]
But the "mutliline" logs are not treated into elasticsearch... I get the following error message
IllegalArgumentException: field [timestamp] not present as part of path [timestamp]
Which I assume comes from the "at ..." line that doesn't contains the [timestamp] field...
---- MULTILINE ERROR MESSAGE ----
[DispatchThread: [com.ibm.mq.jmqi.remote.impl.RemoteSession@555659779[connectionId=414D51434D51484F505141202020202008C5AB59265D7F01;conversationId=3]]]:01.01.2017 01:57:57.243] WARN [id:]
commons.service.mq.MQMessageListenerContainer: Execution of JMS message listener failed, and no ErrorHandler has been set.
org.springframework.jms.listener.adapter.ListenerExecutionFailedException: Listener method 'handleMessage' threw exception; nested exception is java.lang.NumberFormatException: null
at org.springframework.jms.listener.adapter.MessageListenerAdapter.invokeListenerMethod(MessageListenerAdapter.java:309)
Any help is welcome!
Thanks in advance,
Best try with the simulate API.