Handle exceptions

[Sripal]

Hi all,
I have sysout log with many exception. how to read that exceptions. Is there any better way to read the exception. I'm using gork filter to read the pattern(am successfully able to read the pattern). But I can't able to handle the exception. In the same log file i have XML response also. Kindly suggest the best way to read the log file.

[guyboertje]

No one can help you with so little information.

You need to post examples of the log lines that you are having troubles with and the config you have so far.

[Sripal]

Hi @guyboertje,

Thanks for the reply.

I can't upload my log file. it's 98kb. Kinldy help me.

Sample logs

[9/10/18 13:00:45:978 IST] 000002bd SystemOut     O response   : {
  "@count": 122,
  "@start": 1,
  "@totalcount": 122,
  "Messages": [],
  "ResourceName": "Test",
  "ReturnCode": 0,
  "content": [
    {"Test": {"header": {"TestID": "C1002660"}}},
    {"Test": {"header": {"TestID": "C1011849"}}},
    {"Test": {"header": {"TestID": "C1031425"}}},
[9/10/18 15:40:04:441 IST] 000002be SystemOut     O isUserPresentAndValid:   true
[9/10/18 15:40:04:449 IST] 000002be SystemOut     O <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<CountList>
    <ProCode>MA2014</ProCode>
    <Actcode>000</Actcode>
    <ApplicationName>Test1</ApplicationName>
    <UserId>15497845</UserId>
    <NoOfDays>30</NoOfDays>
    <Count></Count>
</CountList>

[9/10/18 15:40:04:467 IST] 000002be SystemOut     O 
[9/10/18 15:40:04:474 IST] 000002be SystemOut     O <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<CountList>
    <ProCode>MA2014</ProCode>
    <Actcode>000</Actcode>
    <ApplicationName>Test2</ApplicationName>
    <UserId>15497845</UserId>
    <NoOfDays>30</NoOfDays>
    <Count></Count>
</CountList>

[Sripal]

[Sripal]

Anybody can help me??

[balumurari1]

once try this and create separate file at the output to identify the exception

output {
file { path => ...
codec => line {
format => "custom format: %{message}"}
}
}

[guyboertje]

Before you can begin parsing, you need to collect the multiline sections into one logical line using the LS multiline codec or the multiline feature in Filebeat. For the multiline codec use:

  pattern => "^\["
  negate => true
  what => "previous"

All the lines start with [9/10/18 13:00:45:978 IST] 000002bd SystemOut O
You can use dissect to extract the data between the delimiters. [, ] and space(s)

[9/10/18 13:00:45:978 IST] 000002bd SystemOut     O response   : {
  "@count": 122,
  "@start": 1,
  "@totalcount": 122,
  "Messages": [],
  "ResourceName": "Test",
  "ReturnCode": 0,
  "content": [
    {"Test": {"header": {"TestID": "C1002660"}}},
    {"Test": {"header": {"TestID": "C1011849"}}},
    {"Test": {"header": {"TestID": "C1031425"}}},

The above line looks like it contains incomplete JSON - it will not be parseable by the JSON filter. Are all such lines truncated?

[9/10/18 15:40:04:441 IST] 000002be SystemOut     O isUserPresentAndValid:   true

The above two lines have a KV pair key: value but the other lines don't. You will need a conditional section to apply a KV filter to those lines only.

The two lines with XML snippets in them may be able to be parsed by the XML filter.

Use this config to start the develop test cycle. Replace the message string in the generator input with more complicated source strings as you add filters and conditionals.

input {
  generator {
    message => "[9/10/18 15:40:04:441 IST] 000002be SystemOut     O isUserPresentAndValid:   true"
    count => 1
  }
}

filter {
  dissect {
    mapping => {
      message => '[%{time}] %{code} %{channel->} %{zero} %{msg}'
    }
  }
}

output { stdout { codec => rubydebug } }

[Sripal]

Thank you for your reply, I'll try and let you know the result.
However the JSON also in a completed form only. I uploaded the sample output.