Logstash Grok Parsing Issue

Elastic Stack Logstash
1

I am using Logstash to read some log file.
Here are some data sources records

<2016-07-07 00:31:01>  Start
<2016-07-07 00:31:59>  Warning - Export_Sysem 6 (1) => No records to be exported
<2016-07-07 00:32:22>  Export2CICAP (04) => Export PO : 34 record(s)
<2016-07-07 00:32:22>  Export2CICAP (04) => Export CO : 87 record(s)
<2016-07-07 00:32:22>  Export2CICAP (04) => Export FC

This is my conf file

grok{
    match => {"message" => [
    '<%{TIMESTAMP_ISO8601:Timestamp}> (%{WORD:Level} - )%{NOTSPACE:Job_Code} => %{GREEDYDATA:Message}',	    
    '<%{TIMESTAMP_ISO8601:Timestamp}>  %{WORD:Parameter} - %{GREEDYDATA:Message}',
    '<%{TIMESTAMP_ISO8601:Timestamp}>  %{WORD:Status}',
    ]}
}

This is part of my output

   {
       "message" => "??2016-07-07 00:31:01>  Start\r?",
      "@version" => "1",
    "@timestamp" => "2016-07-08T03:22:01.076Z",
          "path" => "C:/CIGNA/Export.log",
          "host" => "SIMSPad",
          "type" => "txt",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}
{
       "message" => "<2016-07-07 00:31:59>  Warning - Export_Sysem 6 (1) => No records to be exported\r?",
      "@version" => "1",
    "@timestamp" => "2016-07-06T16:31:59.000Z",
          "path" => "C:/CIGNA/Export.log",
          "host" => "SIMSPad",
          "type" => "txt",
     "Timestamp" => "2016-07-07 00:31:59",
     "Parameter" => "Warning",
       "Message" => "Export_Sysem 6 (1) => No records to be exported\r?"
}
{
       "message" => "<2016-07-07 00:32:22>  Export2CICAP (04) => Export CO : 87 record(s)\r?",
      "@version" => "1",
    "@timestamp" => "2016-07-06T16:32:22.000Z",
          "path" => "C:/CIGNA/Export.log",
          "host" => "SIMSPad",
          "type" => "txt",
     "Timestamp" => "2016-07-07 00:32:22",
        "Status" => "Export2CICAP"
}

As seen from the output, the outcome did not fully parse the message. How should I modify my grok statement so it can fully parse the message?

2

What does "fully parse the message" mean, exactly? Are there additional fields you want to extract?

3

As you seen from the first output, <2016-07-07 00:31:01> Start becomes ??2016-07-07 00:31:01> Start\r? and it cannot be successfully parsed.

Then as seen from the 3rd output<2016-07-07 00:32:22> Export2CICAP (04) => Export CO : 87 record(s)\r? has not fully parsed. Here is the expected output

{
       "message" => "<2016-07-07 00:32:22>  Export2CICAP (04) => Export CO : 87 record(s)\r?",
      "@version" => "1",
    "@timestamp" => "2016-07-06T16:32:22.000Z",
          "path" => "C:/CIGNA/Export.log",
          "host" => "SIMSPad",
          "type" => "txt",
     "Timestamp" => "2016-07-07 00:32:22",
      "Job_Code" => "Export2CICAP (04)"
       "Message" => "Export CO : 87 record(s)"
}

which is different from the current one.

4

As you seen from the first output, <2016-07-07 00:31:01> Start becomes ??2016-07-07 00:31:01> Start\r? and it cannot be successfully parsed.

Perhaps there's a byte-order mark or something at the beginning of the file. Check in a hex editor.

Then as seen from the 3rd output<2016-07-07 00:32:22> Export2CICAP (04) => Export CO : 87 record(s)\r? has not fully parsed. Here is the expected output

Well, you don't have any grok expressions for matching that kind of message. Your first expression is close but still doesn't match.

5

Does that mean I can only graps the whole part of data, i.e. Export2CICAP (04) => Export CO : 87 record(s)\r? into a field?

6

You can extract any parts of the data you like but you must write grok expressions for doing so.

7

Sorry, I don't quite get it. What does it mean by you don't have any grok expressions for matching that kind of message.?

8

You have three grok expressions in your configuration. None of them work for the message you're asking about. The first expression almost matches but not quite. You need to adjust it or add another expression. I don't think I can explain this in any other way. Good luck.

9

Thanks for your replies.

And as for [quote="magnusbaeck, post:4, topic:55010"]
Perhaps there's a byte-order mark or something at the beginning of the file. Check in a hex editor.
[/quote]

How should I get rid of a BOM?