Hi All,
I am using multiline pattern within filebeat.yml to format the logs as follows,
filebeat.inputs:
- document_type: webapp
enabled: true
paths:- /opt/sample/app.log
multiline.pattern: '^., [[^]]+]'
multiline.negate: true
multiline.match: after
tags: ["shop"]
exclude_lines: ["^$"]
fields: {application: shop}
fields_under_root: true
tail_files: true
- /opt/sample/app.log
My logs are having some unique ID to identify the block of logs to combine multiline logs. So I tried with flush pattern to do this in filebeat. But my logs are getting collapsed with another blocks since I am not able to use the same group value ( Unique-ID) from multiline.pattern to flush pattern.
for example, ( Only for example and this patterns are not valid one for my logs )
multiline.pattern: 'Started'
multiline.negate: true
multiline.match: after
multiline.flush_pattern: 'Completed'
My sample log,
I, [2018-07-24T05:40:19.326479 #27608] INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6] Processing by CartController#account_provision as JSON
I, [2018-07-24T05:40:19.326567 #27608] INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6] Parameters: {"speed"=>"8.99", "tracking"=>{"google"=>{"clientId"=>"1274719564.1532410753"}, "offer"=>{"offer_code"=>"STREAM_HUB", "purchase_code"=>"", "referral_code"=>"", "buyer_pk"=>"", "order_id"=>""}, "web"=>{"user_agent"=>"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0", "referrer"=>"https://shop-stream-qa.ooma.com/cart?", "view"=>"desktop", "ip"=>"127.0.0.1", "tenant"=>"stream", "tracking_session_id"=>"ioiwo2j58ep2uu9tum3mo6zfcsas69am", "url"=>"https://shop-stream-qa.ooma.com/cart/checkout#review", "page"=>"https://shop-stream-qa.ooma.com/cart/checkout#review"}}}
I, [2018-07-24T05:40:20.916634 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Started GET "/cart/confirm" for 127.0.0.1 at 2018-07-24 05:40:21 +0000
I, [2018-07-24T05:40:21.813837 #27608] INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6] Completed 200 OK in 1487ms (Views: 0.2ms)
I, [2018-07-24T05:40:21.918109 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Processing by CartController#confirm as HTML
I, [2018-07-24T05:40:21.924357 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Rendered cart/_templates.html.erb (3.3ms)
I, [2018-07-24T05:40:21.927207 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Rendered cart/_review_items.html.erb (2.2ms)
I, [2018-07-24T05:40:21.927628 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Rendered cart/confirm.html.erb within layouts/application (8.4ms)
I, [2018-07-24T05:40:21.929867 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Rendered layouts/_head.html.erb (2.0ms)
I, [2018-07-24T05:40:21.930372 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Rendered layouts/_gtm.html.erb (0.3ms)
I, [2018-07-24T05:40:21.930732 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Rendered layouts/_header.html.erb (0.2ms)
I, [2018-07-24T05:40:21.932368 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Completed 200 OK in 14ms (Views: 13.7ms)
Here, filebeat treating the followings are multiline events where actually it's not.
I, [2018-07-24T05:40:20.916634 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Started GET "/cart/confirm" for 127.0.0.1 at 2018-07-24 05:40:21 +0000
I, [2018-07-24T05:40:21.813837 #27608] INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6] Completed 200 OK in 1487ms (Views: 0.2ms)
It should decide the multiline block with 'Request ID'. Is this possible using multiline pattern ?
For example I want to group my logs using request ID. Here Req IDs are,
343ca189-6b84-4657-8dc0-1a93879482c6 & 43df0991-650e-4bcb-a3b2-a20625515fdd
The blocks I need is,
I, [2018-07-24T05:40:19.326479 #27608] INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6] Processing by CartController#account_provision as JSON
I, [2018-07-24T05:40:19.326567 #27608] INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6] Parameters: {"speed"=>"8.99", "tracking"=>{"google"=>{"clientId"=>"1274719564.1532410753"}, "offer"=>{"offer_code"=>"STREAM_HUB", "purchase_code"=>"", "referral_code"=>"", "buyer_pk"=>"", "order_id"=>""}, "web"=>{"user_agent"=>"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0", "referrer"=>"https://shop-stream-qa.ooma.com/cart?", "view"=>"desktop", "ip"=>"127.0.0.1", "tenant"=>"stream", "tracking_session_id"=>"ioiwo2j58ep2uu9tum3mo6zfcsas69am", "url"=>"https://shop-stream-qa.ooma.com/cart/checkout#review", "page"=>"https://shop-stream-qa.ooma.com/cart/checkout#review"}}}
I, [2018-07-24T05:40:21.813837 #27608] INFO -- : [343ca189-6b84-4657-8dc0-1a93879482c6] Completed 200 OK in 1487ms (Views: 0.2ms)
and
I, [2018-07-24T05:40:20.916634 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Started GET "/cart/confirm" for 127.0.0.1 at 2018-07-24 05:40:21 +0000
I, [2018-07-24T05:40:21.918109 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Processing by CartController#confirm as HTML
I, [2018-07-24T05:40:21.924357 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Rendered cart/_templates.html.erb (3.3ms)
I, [2018-07-24T05:40:21.927207 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Rendered cart/_review_items.html.erb (2.2ms)
I, [2018-07-24T05:40:21.927628 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Rendered cart/confirm.html.erb within layouts/application (8.4ms)
I, [2018-07-24T05:40:21.929867 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Rendered layouts/_head.html.erb (2.0ms)
I, [2018-07-24T05:40:21.930372 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Rendered layouts/_gtm.html.erb (0.3ms)
I, [2018-07-24T05:40:21.930732 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Rendered layouts/_header.html.erb (0.2ms)
I, [2018-07-24T05:40:21.932368 #27608] INFO -- : [43df0991-650e-4bcb-a3b2-a20625515fdd] Completed 200 OK in 14ms (Views: 13.7ms)
Is this possible in filebeat ? Can anyone please suggest.
Thanks,
Vennila K