Multiline pattern & flush pattern in filebeat


#1

Hi All,

I am using multiline pattern within filebeat.yml to format the logs as follows,

filebeat.inputs:

  • document_type: webapp
    enabled: true
    paths:
    • /opt/sample/app.log
      multiline.pattern: '^., [[^]]+]'
      multiline.negate: true
      multiline.match: after
      tags: ["shop"]
      exclude_lines: ["^$"]
      fields: {application: shop}
      fields_under_root: true
      tail_files: true

My logs are having some unique ID to identify the block of logs to combine multiline logs. So I tried with flush pattern to do this in filebeat. But my logs are getting collapsed with another blocks since I am not able to use the same group value ( Unique-ID) from multiline.pattern to flush pattern.

for example, ( Only for example and this patterns are not valid one for my logs )
multiline.pattern: 'Started'
multiline.negate: true
multiline.match: after
multiline.flush_pattern: 'Completed'

My sample log,

I, [2018-07-24T05:40:19.326479 #27608]  INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6] Processing by CartController#account_provision as JSON
I, [2018-07-24T05:40:19.326567 #27608]  INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6]   Parameters: {"speed"=>"8.99", "tracking"=>{"google"=>{"clientId"=>"1274719564.1532410753"}, "offer"=>{"offer_code"=>"STREAM_HUB", "purchase_code"=>"", "referral_code"=>"", "buyer_pk"=>"", "order_id"=>""}, "web"=>{"user_agent"=>"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0", "referrer"=>"https://shop-stream-qa.ooma.com/cart?", "view"=>"desktop", "ip"=>"127.0.0.1", "tenant"=>"stream", "tracking_session_id"=>"ioiwo2j58ep2uu9tum3mo6zfcsas69am", "url"=>"https://shop-stream-qa.ooma.com/cart/checkout#review", "page"=>"https://shop-stream-qa.ooma.com/cart/checkout#review"}}}
I, [2018-07-24T05:40:20.916634 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Started GET "/cart/confirm" for 127.0.0.1 at 2018-07-24 05:40:21 +0000
I, [2018-07-24T05:40:21.813837 #27608]  INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6] Completed 200 OK in 1487ms (Views: 0.2ms)
I, [2018-07-24T05:40:21.918109 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Processing by CartController#confirm as HTML
I, [2018-07-24T05:40:21.924357 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd]   Rendered cart/_templates.html.erb (3.3ms)
I, [2018-07-24T05:40:21.927207 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd]   Rendered cart/_review_items.html.erb (2.2ms)
I, [2018-07-24T05:40:21.927628 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd]   Rendered cart/confirm.html.erb within layouts/application (8.4ms)
I, [2018-07-24T05:40:21.929867 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd]   Rendered layouts/_head.html.erb (2.0ms)
I, [2018-07-24T05:40:21.930372 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd]   Rendered layouts/_gtm.html.erb (0.3ms)
I, [2018-07-24T05:40:21.930732 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd]   Rendered layouts/_header.html.erb (0.2ms)
I, [2018-07-24T05:40:21.932368 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Completed 200 OK in 14ms (Views: 13.7ms)

Here, filebeat treating the followings are multiline events where actually it's not.

I, [2018-07-24T05:40:20.916634 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Started GET "/cart/confirm" for 127.0.0.1 at 2018-07-24 05:40:21 +0000
I, [2018-07-24T05:40:21.813837 #27608]  INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6] Completed 200 OK in 1487ms (Views: 0.2ms)

It should decide the multiline block with 'Request ID'. Is this possible using multiline pattern ?

For example I want to group my logs using request ID. Here Req IDs are,
343ca189-6b84-4657-8dc0-1a93879482c6 & 43df0991-650e-4bcb-a3b2-a20625515fdd

The blocks I need is,

I, [2018-07-24T05:40:19.326479 #27608]  INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6] Processing by CartController#account_provision as JSON
I, [2018-07-24T05:40:19.326567 #27608]  INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6]   Parameters: {"speed"=>"8.99", "tracking"=>{"google"=>{"clientId"=>"1274719564.1532410753"}, "offer"=>{"offer_code"=>"STREAM_HUB", "purchase_code"=>"", "referral_code"=>"", "buyer_pk"=>"", "order_id"=>""}, "web"=>{"user_agent"=>"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0", "referrer"=>"https://shop-stream-qa.ooma.com/cart?", "view"=>"desktop", "ip"=>"127.0.0.1", "tenant"=>"stream", "tracking_session_id"=>"ioiwo2j58ep2uu9tum3mo6zfcsas69am", "url"=>"https://shop-stream-qa.ooma.com/cart/checkout#review", "page"=>"https://shop-stream-qa.ooma.com/cart/checkout#review"}}}
I, [2018-07-24T05:40:21.813837 #27608]  INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6] Completed 200 OK in 1487ms (Views: 0.2ms)

and

I, [2018-07-24T05:40:20.916634 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Started GET "/cart/confirm" for 127.0.0.1 at 2018-07-24 05:40:21 +0000
I, [2018-07-24T05:40:21.918109 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Processing by CartController#confirm as HTML
I, [2018-07-24T05:40:21.924357 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd]   Rendered cart/_templates.html.erb (3.3ms)
I, [2018-07-24T05:40:21.927207 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd]   Rendered cart/_review_items.html.erb (2.2ms)
I, [2018-07-24T05:40:21.927628 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd]   Rendered cart/confirm.html.erb within layouts/application (8.4ms)
I, [2018-07-24T05:40:21.929867 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd]   Rendered layouts/_head.html.erb (2.0ms)
I, [2018-07-24T05:40:21.930372 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd]   Rendered layouts/_gtm.html.erb (0.3ms)
I, [2018-07-24T05:40:21.930732 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd]   Rendered layouts/_header.html.erb (0.2ms)
I, [2018-07-24T05:40:21.932368 #27608]  INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Completed 200 OK in 14ms (Views: 13.7ms)

Is this possible in filebeat ? Can anyone please suggest.

Thanks,
Vennila K


#2

Hi all,

shall I get any update/idea on this ?

Regards,
Vennila K