Message not splitting into events on new line

[kbrisso (Kevin)]

Hello-

I have a message that looks like below:

[INFO] 2020-09-21 09:40:21,476[ajp-nio-9031-exec-1] com.gts.ci.jws.filter.ValidationFilter - Validation Filter URL /jsp_web/access/app_logout.jsp\n[DEBUG] 2020-09-21 09:40:21,476[ajp-nio-9031-exec-1] com.gts.ci.jws.filter.ValidationFilter - Validation Filter called\n[ERROR] 2020-09-21 09:40:21,477[ajp-nio-9031-exec-1] com.gts.ci.jws.filter.ValidationFilter - req.getParameterNames().hasMoreElements(): false\n[DEBUG] 2020-09-21 09:40:21,477[ajp-nio-9031-exec-1] com.gts.ci.jws.framework.web.servlet.SessionEventListener - Session created ID:3C4D96F599D8F9A665CD5379053C88F8.gttaqaapp63\n[DEBUG] 2020-09-21 09:40:21,477[ajp-nio-9031-exec-1] com.gts.ci.jws.framework.web.servlet.SessionBindingListener - Session binding listener invoked ::::false\n

Each log messages has a new line \n at the end

My log stash config is a as follows

filter {
   split { terminator => "\n" } 
 grok {
    match => { "message" => ['(?m)\[%{LOGLEVEL:logLevel}]\ %{TIMESTAMP_ISO8601:time}\[%{DATA:thread}\]%{GREEDYDATA:logMessage}'] }
  }
}

My issue is the message is not getting parsed at \n new line, what am I missing? For each message that has a new line, I want that to be its own event. Its parsing some of it because the level and thread are getting into the fields. I would appreciate any help, this is my first setup of this. I'm obviously missing something..

Thanks,
Kevin

[Badger]

@kbrisso

split { terminator => "\n" }

Try changing this to

split {}

or even

split { terminator => "
" }

In the ruby source code "\n" refers to a newline character. In a logstash configuration file it refers to the character backslash followed by the character n.

[kbrisso (Kevin)]

Hello Badger-

This worked perfect, thank you so much. I was close.

One more problem, this is an very old system and some of the log messages actually have \n in the log output itself along with other random characters like XML tags. Is there a way to encode \n in file beat before shipping to log stash so the grok process doesn't split this into multiple messages?

I was looking at this setting in file beat:

  # Configure the file encoding for reading files with international characters
  # following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).
  # Some sample encodings:
  #   plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
  #    hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
  #encoding: plain

[DEBUG] 2020-09-23 08:50:36,248[ajp-nio-9031-exec-9]  CUSTOMIZATION - [ThingWatchListTag][fetchWatchListDetailsForUser][ThingWatchListTag getWatchListDetailsForUser() ] olio key : \n com.grk.id.wsc.olio.value.olioKeyVO@2d51e371\n
[
  actorID=<null>\n
  actorType=FTE\n
  asn=<null>\n
  filterType=DealerNo\n
  giin=<null>\n
  cIPSearch=<null>\n
  pagination=<null>
  thingIDType=<null>\n
  strtIndx=<null>\n
  realTimeFlag=true\n
  requestParams={}\n
  uid=<null>\n
  uidType=<null>\n
  userLoginID=jjones\n
  referenceIDTable={}\n
  callerName=<null>\n
  callerPhone=<null>\n
  wirn=<null>\n
  pega=<null>\n
  rdChoice=<null>\n
  reqTypeSAT=<null>\n
  reqDateSAT=<null>\n
]

Thanks,
Kevin

[Badger]

@kbrisso

Is there a way to encode \n in file beat

That's a filebeat question and I do not run filebeat. You might want to ask in the filebeat forum.