Message not splitting into events on new line

Elastic Stack Logstash
1

Hello-

I have a message that looks like below:

[INFO] 2020-09-21 09:40:21,476[ajp-nio-9031-exec-1] com.gts.ci.jws.filter.ValidationFilter - Validation Filter URL /jsp_web/access/app_logout.jsp\n[DEBUG] 2020-09-21 09:40:21,476[ajp-nio-9031-exec-1] com.gts.ci.jws.filter.ValidationFilter - Validation Filter called\n[ERROR] 2020-09-21 09:40:21,477[ajp-nio-9031-exec-1] com.gts.ci.jws.filter.ValidationFilter - req.getParameterNames().hasMoreElements(): false\n[DEBUG] 2020-09-21 09:40:21,477[ajp-nio-9031-exec-1] com.gts.ci.jws.framework.web.servlet.SessionEventListener - Session created ID:3C4D96F599D8F9A665CD5379053C88F8.gttaqaapp63\n[DEBUG] 2020-09-21 09:40:21,477[ajp-nio-9031-exec-1] com.gts.ci.jws.framework.web.servlet.SessionBindingListener - Session binding listener invoked ::::false\n

Each log messages has a new line \n at the end

My log stash config is a as follows

filter {
   split { terminator => "\n" } 
 grok {
    match => { "message" => ['(?m)\[%{LOGLEVEL:logLevel}]\ %{TIMESTAMP_ISO8601:time}\[%{DATA:thread}\]%{GREEDYDATA:logMessage}'] }
  }
}

My issue is the message is not getting parsed at \n new line, what am I missing? For each message that has a new line, I want that to be its own event. Its parsing some of it because the level and thread are getting into the fields. I would appreciate any help, this is my first setup of this. I'm obviously missing something..

Thanks,
Kevin

2

Try changing this to

split {}

or even

split { terminator => "
" }

In the ruby source code "\n" refers to a newline character. In a logstash configuration file it refers to the character backslash followed by the character n.

3

Hello Badger-

This worked perfect, thank you so much. I was close. :slight_smile:

One more problem, this is an very old system and some of the log messages actually have \n in the log output itself along with other random characters like XML tags. Is there a way to encode \n in file beat before shipping to log stash so the grok process doesn't split this into multiple messages?

I was looking at this setting in file beat:

  # Configure the file encoding for reading files with international characters
  # following the W3C recommendation for HTML5 (http://www.w3.org/TR/encoding).
  # Some sample encodings:
  #   plain, utf-8, utf-16be-bom, utf-16be, utf-16le, big5, gb18030, gbk,
  #    hz-gb-2312, euc-kr, euc-jp, iso-2022-jp, shift-jis, ...
  #encoding: plain

[DEBUG] 2020-09-23 08:50:36,248[ajp-nio-9031-exec-9]  CUSTOMIZATION - [ThingWatchListTag][fetchWatchListDetailsForUser][ThingWatchListTag getWatchListDetailsForUser() ] olio key : \n com.grk.id.wsc.olio.value.olioKeyVO@2d51e371\n
[
  actorID=<null>\n
  actorType=FTE\n
  asn=<null>\n
  filterType=DealerNo\n
  giin=<null>\n
  cIPSearch=<null>\n
  pagination=<null>
  thingIDType=<null>\n
  strtIndx=<null>\n
  realTimeFlag=true\n
  requestParams={}\n
  uid=<null>\n
  uidType=<null>\n
  userLoginID=jjones\n
  referenceIDTable={}\n
  callerName=<null>\n
  callerPhone=<null>\n
  wirn=<null>\n
  pega=<null>\n
  rdChoice=<null>\n
  reqTypeSAT=<null>\n
  reqDateSAT=<null>\n
]

Thanks,
Kevin

4

That's a filebeat question and I do not run filebeat. You might want to ask in the filebeat forum.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.