Logstash grok match pattern for message field

Elastic Stack Logstash
1

My log data is like,

2015-01-31 15:58:56,851 [9] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - Setting offline returninvoice: c0000000-2144-04e2-f409-ffff08d20b85
2015-01-31 15:58:56,860 [9] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - InvoiceCart: Overwrite subtotal, taxtotal and total of invoice by dispensed total
2015-01-31 15:58:57,400 [9] ERROR NCR.AKPOS.Enterprise_Comm.EventSender - EventSender.SendInvoice() Generate Message Error: System.ArgumentNullException: Value cannot be null.Parameter name: value at System.Xml.Linq.XAttribute..ctor(XName name, Object value)
 at NCR.AKPOS.Enterprise_Comm.MessageHandlerObjecttoXMLHelper.CreateXMLFromInvoice(Invoice invoice, Customer_ID customer_id, List`1 parentInvoices, Boolean isParentInvoice) in D:\Satish\Work\Vidbox Source\branches\ver1.2.0.0\XE\Kiosk Solutions\AKPOS\AKPOS\Enterprise_Comm\MessageHandlerObjecttoXMLHelper.cs:line 53

In this logs, logs are separated by DATE with TIME. my intention is 3 lines(documents) are there.

My Pattern is:

grok { 
match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg} " } 
     }

i have a problem with GREEDYDATA:msg pattern, msg filed data is trimmed (total message is not come to the msg filed).

I am not understand what the problem is, I tried with so many examples of patterns like (multiline) examples.So can you please provide the solution for this...thanks in advance...

2

I don't quite get what you're asking, but you have a trailing space in your pattern (right after %{GREEDYDATA:msg}) that you should remove.

So you're using a multiline filter (or codec)? How has that been configured? If you can supply a complete example that exhibits that problem you're seeing it will be easier to help.

3

Thanks @magnusbaeck, am using that configuration....

 filter{

multiline{
    pattern => "^\["
    what => "previous"
    negate=> true
}

mutate {
    gsub => ['message', "\n", " "]
}

grok { 
    match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg} " } 
  }

}
4

Okay. And what's the problem you're seeing?

5 
at NCR.AKPOS.Enterprise_Comm.MessageHandlerObjecttoXMLHelper.CreateXMLFromInvoice(Invoice invoice, Customer_ID customer_id, List`1 parentInvoices, Boolean isParentInvoice) in D:\Satish\Work\Vidbox Source\branches\ver1.2.0.0\XE\Kiosk Solutions\AKPOS\AKPOS\Enterprise_Comm\MessageHandlerObjecttoXMLHelper.cs:line 53

this line of data not coming when a 3rd line parsing

6

I'm surprised you're getting anything at all given that your multiline config uses the pattern ^\[ and none of your messages begin with a left square bracket.

7

Ok it's a wrong sorry, then what's the way to represent for this type of data in the filter

8 
July 10th 2015, 12:28:08.965 	MessageProcessorWS.ProcessMessage() MessageHandler.ProcessMessage
July 10th 2015, 12:28:08.960 	Changing piece AC0000002 siteid from b49cfa1a-c8ef-46d1-a675-663c66f957bf to
July 10th 2015, 12:28:08.954 	Updating stock
July 10th 2015, 12:28:08.950 	post invoice result:
July 10th 2015, 12:28:08.947 	Cart getCartPayment():
July 10th 2015, 12:28:08.942 	Cart.SendHistoryEvents() Sending Product: id: Product10 transType:

this is in kibana, left side filed is time right side filed is msg . So in this msg filed some data is trimmed

9

Well, it obviously needs to match the first line of each message so try the prefix of your grok expression:

multiline{
    pattern => "^%{TIMESTAMP_ISO8601}"
    what => "previous"
    negate=> true
}

Note that Logstash has a bug (still not fixed in 1.5 I think) where the last multiline message in a stream is never passed on since Logstash keeps on waiting for the beginning of the next message.

10

No @magnusbaeck, It's not working. Again it's taking a new line....

11 
input {

file {
  path => [ "\\logfilepath.*_bak" ]
  start_position => "beginning"         
 }
 }

filter {

multiline{
        pattern => "^%{TIMESTAMP_ISO8601}"
        what => "previous"
        negate=> true
    }

mutate {
    gsub => ['message', "\n", " "]
}

grok { 
    match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg} " } 
  }

}
output {

	elasticsearch {
            bind_host => "127.0.0.1"
            port => "9200"
            protocol => http
       }
	stdout { codec => rubydebug }
    }

This is the logstash.conf, i have in my application.Can you check my configuration file once..

12

The following works on my machine with Logstash 1.4.2. Well, the grok filter doesn't work for multiline messages but all messages (except the last one, as explained above) are being picked up.

$ cat data
2015-01-31 15:58:56,851 [9] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - Setting offline returninvoice: c0000000-2144-04e2-f409-ffff08d20b85
2015-01-31 15:58:56,860 [9] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - InvoiceCart: Overwrite subtotal, taxtotal and total of invoice by dispensed total
2015-01-31 15:58:57,400 [9] ERROR NCR.AKPOS.Enterprise_Comm.EventSender - EventSender.SendInvoice() Generate Message Error: System.ArgumentNullException: Value cannot be null.Parameter name: value at System.Xml.Linq.XAttribute..ctor(XName name, Object value)
 at NCR.AKPOS.Enterprise_Comm.MessageHandlerObjecttoXMLHelper.CreateXMLFromInvoice(Invoice invoice, Customer_ID customer_id, List`1 parentInvoices, Boolean isParentInvoice) in D:\Satish\Work\Vidbox Source\branches\ver1.2.0.0\XE\Kiosk Solutions\AKPOS\AKPOS\Enterprise_Comm\MessageHandlerObjecttoXMLHelper.cs:line 53
2015-01-31 15:58:56,851 [9] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - Setting offline returninvoice: c0000000-2144-04e2-f409-ffff08d20b85
$ cat test.config 
input { stdin { codec => plain } }
output { stdout { codec => rubydebug } }
filter {
  multiline{
    pattern => "^%{TIMESTAMP_ISO8601}"
    what => "previous"
    negate=> true
  }
  mutate {
    gsub => ['message', "\n", " "]
  }
  grok { 
    match => { "message" => "%{TIMESTAMP_ISO8601:time} \[%{NUMBER:thread}\] %{LOGLEVEL:loglevel} %{JAVACLASS:class} - %{GREEDYDATA:msg}" } 
  }
}
$ /opt/logstash/bin/logstash -f test.config < data
{
       "message" => "2015-01-31 15:58:56,851 [9] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - Setting offline returninvoice: c0000000-2144-04e2-f409-ffff08d20b85",
      "@version" => "1",
    "@timestamp" => "2015-07-10T13:19:55.235Z",
          "host" => "seldlx20533",
          "time" => "2015-01-31 15:58:56,851",
        "thread" => "9",
      "loglevel" => "DEBUG",
         "class" => "NCR.AKPOS.ShoppingCart.CartInvoicer",
           "msg" => "Setting offline returninvoice: c0000000-2144-04e2-f409-ffff08d20b85"
}
{
       "message" => "2015-01-31 15:58:56,860 [9] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - InvoiceCart: Overwrite subtotal, taxtotal and total of invoice by dispensed total",
      "@version" => "1",
    "@timestamp" => "2015-07-10T13:19:55.235Z",
          "host" => "seldlx20533",
          "time" => "2015-01-31 15:58:56,860",
        "thread" => "9",
      "loglevel" => "DEBUG",
         "class" => "NCR.AKPOS.ShoppingCart.CartInvoicer",
           "msg" => "InvoiceCart: Overwrite subtotal, taxtotal and total of invoice by dispensed total"
}
{
       "message" => "2015-01-31 15:58:57,400 [9] ERROR NCR.AKPOS.Enterprise_Comm.EventSender - EventSender.SendInvoice() Generate Message Error: System.ArgumentNullException: Value cannot be null.Parameter name: value at System.Xml.Linq.XAttribute..ctor(XName name, Object value)  at NCR.AKPOS.Enterprise_Comm.MessageHandlerObjecttoXMLHelper.CreateXMLFromInvoice(Invoice invoice, Customer_ID customer_id, List`1 parentInvoices, Boolean isParentInvoice) in D:\\Satish\\Work\\Vidbox Source\\branches\\ver1.2.0.0\\XE\\Kiosk Solutions\\AKPOS\\AKPOS\\Enterprise_Comm\\MessageHandlerObjecttoXMLHelper.cs:line 53",
      "@version" => "1",
    "@timestamp" => "2015-07-10T13:19:55.235Z",
          "host" => "seldlx20533",
          "tags" => [
        [0] "multiline",
        [1] "_grokparsefailure"
    ]
}
1 Like
13

As same as Work on my machine also, But, below line also taking separate document.

at NCR.AKPOS.Enterprise_Comm.MessageHandlerObjecttoXMLHelper.CreateXMLFromInvoice(Invoice invoice, Customer_ID customer_id, List`1 parentInvoices, Boolean isParentInvoice) in Solutions\AKPOS\AKPOS\Enterprise_Comm\MessageHandlerObjecttoXMLHelper.cs:line 53


   at System.Xml.XmlDownloadManager.GetStream(Uri uri, ICredentials credentials)
   at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn)

My Machine loading:

 {
       "message" => "   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)\r",
      "@version" => "1",
    "@timestamp" => "2015-07-10T13:26:11.885Z",
          "host" => "",
          "path" => "",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

    {
           "message" => "   at System.Xml.XmlDownloadManager.GetStream(Uri uri, ICredentials credentials)\r",
          "@version" => "1",
        "@timestamp" => "2015-07-10T13:26:11.886Z",
              "host" => 
              "path" => 
              "tags" => [
            [0] "_grokparsefailure"
        ]
    }
    {
           "message" => "   at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn)\r",
          "@version" => "1",
        "@timestamp" => "2015-07-10T13:26:11.887Z",
              "host" => "",
              "path" => ",
              "tags" => [
            [0] "_grokparsefailure"
        ]
    }
14

Which version of Logstash?

15

logstash 1.5.1

16

Okay. That might at least explain the difference. I don't have time to install 1.5.1 and debug this.

17

I ran this example on Logstash 1.5.2, and it seems to work just fine.

    $ cat test.data | ../bin/logstash -f ./test.config 
Logstash startup completed{
       "message" => "2015-01-31 15:58:56,851 [9] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - Setting offline returninvoice: c0000000-2144-04e2-f409-ffff08d20b85",
      "@version" => "1",
    "@timestamp" => "2015-07-10T13:45:01.625Z",
          "host" => "Christians-MacBook-Air-3.local",
          "time" => "2015-01-31 15:58:56,851",
        "thread" => "9",
      "loglevel" => "DEBUG",
         "class" => "NCR.AKPOS.ShoppingCart.CartInvoicer",
           "msg" => "Setting offline returninvoice: c0000000-2144-04e2-f409-ffff08d20b85"
}

{
       "message" => "2015-01-31 15:58:56,860 [9] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - InvoiceCart: Overwrite subtotal, taxtotal and total of invoice by dispensed total",
      "@version" => "1",
    "@timestamp" => "2015-07-10T13:45:01.626Z",
          "host" => "Christians-MacBook-Air-3.local",
          "time" => "2015-01-31 15:58:56,860",
        "thread" => "9",
      "loglevel" => "DEBUG",
         "class" => "NCR.AKPOS.ShoppingCart.CartInvoicer",
           "msg" => "InvoiceCart: Overwrite subtotal, taxtotal and total of invoice by dispensed total"
}
{
       "message" => "2015-01-31 15:58:57,400 [9] ERROR NCR.AKPOS.Enterprise_Comm.EventSender - EventSender.SendInvoice() Generate Message Error: System.ArgumentNullException: Value cannot be null.Parameter name: value at System.Xml.Linq.XAttribute..ctor(XName name, Object value)  at NCR.AKPOS.Enterprise_Comm.MessageHandlerObjecttoXMLHelper.CreateXMLFromInvoice(Invoice invoice, Customer_ID customer_id, List`1 parentInvoices, Boolean isParentInvoice) in D:\\Satish\\Work\\Vidbox Source\\branches\\ver1.2.0.0\\XE\\Kiosk Solutions\\AKPOS\\AKPOS\\Enterprise_Comm\\MessageHandlerObjecttoXMLHelper.cs:line 53",
      "@version" => "1",
    "@timestamp" => "2015-07-10T13:45:01.627Z",
          "host" => "Christians-MacBook-Air-3.local",
          "tags" => [
        [0] "multiline"
    ],
          "time" => "2015-01-31 15:58:57,400",
        "thread" => "9",
      "loglevel" => "ERROR",
         "class" => "NCR.AKPOS.Enterprise_Comm.EventSender",
           "msg" => "EventSender.SendInvoice() Generate Message Error: System.ArgumentNullException: Value cannot be null.Parameter name: value at System.Xml.Linq.XAttribute..ctor(XName name, Object value)  at NCR.AKPOS.Enterprise_Comm.MessageHandlerObjecttoXMLHelper.CreateXMLFromInvoice(Invoice invoice, Customer_ID customer_id, List`1 parentInvoices, Boolean isParentInvoice) in D:\\Satish\\Work\\Vidbox Source\\branches\\ver1.2.0.0\\XE\\Kiosk Solutions\\AKPOS\\AKPOS\\Enterprise_Comm\\MessageHandlerObjecttoXMLHelper.cs:line 53"
}
{
       "message" => "2015-01-31 15:58:56,851 [9] DEBUG NCR.AKPOS.ShoppingCart.CartInvoicer - Setting offline returninvoice: c0000000-2144-04e2-f409-ffff08d20b85 ",
      "@version" => "1",
    "@timestamp" => "2015-07-10T13:45:01.627Z",
          "host" => "Christians-MacBook-Air-3.local",
          "tags" => [
        [0] "multiline"
    ],
          "time" => "2015-01-31 15:58:56,851",
        "thread" => "9",
      "loglevel" => "DEBUG",
         "class" => "NCR.AKPOS.ShoppingCart.CartInvoicer",
           "msg" => "Setting offline returninvoice: c0000000-2144-04e2-f409-ffff08d20b85 "
}
1 Like
18

@Christian_Dahlqvist Thanks, are you change any configuration in logstash.conf ????

19

I copied the config and data from the example Magnus provided and ran it as shown in the example.

20

OK OK @Christian_Dahlqvist, Thanks for information....