How do I get the first 2 lines of a multiline log using logstash 2.4.0?

Elastic Stack Logstash
1

Hi...

I figured out how to get the first line using this post: How do I get first line from multiline message by using logstash 1.5?

using this config:

filter {
  if [type] == "java" {
    grok {
      # Do multiline matching with (?m)
      match => [ "message", "(?m)^%{TIMESTAMP_ISO8601:java_timestamp}%{SPACE}%{LOGLEVEL:loglevel}%{SPACE}%{NUMBER:pid}%{SPACE}(\[%{TRANSID:transactionId}\]%{SPACE})?---%{SPACE}%{SYSLOG5424SD:thread}%{SPACE}%{JAVACLASSSHORT:class}%{SPACE}:%{SPACE}%{GREEDYDATA:logmessage}" ]
      patterns_dir => "/etc/logstash/conf.d/patterns"
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    mutate {
      add_field => { "message_first_line" => "%{message}" }
    }
    mutate {
      gsub => [
        "message_first_line", "\n.*", ""
      ]
    }
    date {
      match => [ "java_timestamp", "ISO8601" ]
    }
  }
}

The gsub section replaces everything after and including the first \n with "".

Which I thot was quite clever but what I really want is to see the first line of the stack trace as well - log might look like:

2017-11-02 20:49:54.859 ERROR 14703 [CoreProcess] --- [pool-7-thread-1] e.u.i.e.b.integration.PollerIntegration  : Connection failure for client drs, recovering with blank message.

org.springframework.messaging.MessagingException: Failed to execute on session; nested exception is org.springframework.core.NestedIOException: Failed to list files; nested exception is 2: No such file
	at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:419)
	at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.doLs(AbstractRemoteFileOutboundGateway.java:500)
	at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.handleRequestMessage(AbstractRemoteFileOutboundGateway.java:469)
	at org.springframework.integration.handler.AbstractReplyProducingMessageHandler$AdvisedRequestHandler.handleRequestMessage(AbstractReplyProducingMessageHandler.java:144)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)

What I really want is the first line plus the first line of the stack trace IF it's a multiline log otherwise just the first line.

Can anyone help?

I found this which looked promising: https://stackoverflow.com/questions/5422949/how-to-find-the-3rd-occurrence-of-a-pattern-on-a-line

I tried this config and some similar permutations:

mutate {
  add_field => { "message_first_2_lines" => "%{message}" }
}
mutate {
  gsub => [
    "message_first_2_lines", "^((?:.*?\n){2}).*$", "\1"
  ]
}

Without good results - the .*$ didn't seem to match the entire multiline string.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.