Filebeat - Multiline events

Elastic Stack Beats
1

Hello,

In my case : logs are generated by Ruby On Rails.

I would like to generate a message by request using multiline patterns :

multiline.pattern: 'Started'
multiline.negate: true
multiline.match: after
multiline.flush_pattern: 'Completed'

It's working well but i have overlapping events.

An extract of logs :

Oct 21 16:48:11 HOST [519]: Started POST "/bss/CountObjects/domain.fr" for 10.113.1.240 at 2019-10-21 16:48:11 +0200
Oct 21 16:48:11 HOST [527]: Started POST "/bss/CountObjects/domain.fr" for 10.113.1.240 at 2019-10-21 16:48:11 +0200
Oct 21 16:48:11 HOST [603]: Started POST "/bss/CountObjects/domain.fr" for 10.113.1.240 at 2019-10-21 16:48:11 +0200
Oct 21 16:48:11 HOST [527]: Processing by BssController#count_objects as JSON
Oct 21 16:48:11 HOST [527]: Parameters: {"type"=>"dl", "domain_name"=>"domain.fr"}
Oct 21 16:48:11 HOST [519]: Processing by BssController#count_objects as JSON
Oct 21 16:48:11 HOST [519]: Parameters: {"type"=>"userAccount", "domain_name"=>"domain.fr"}
Oct 21 16:48:11 HOST [603]: Processing by BssController#count_objects as JSON
Oct 21 16:48:11 HOST [603]: Parameters: {"type"=>"worker", "domain_name"=>"domain.fr"}
Oct 21 16:48:11 HOST [603]: Completed 200 OK in 91ms (Views: 0.3ms | ActiveRecord: 21.2ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_advancedsearch_mod.html.erb (1.2ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_account_mod_fidems.html.erb (0.1ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_account_mod_alias.html.erb (0.0ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_account_mod_change_password.html.erb (0.0ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_account_mod_move_account.html.erb (0.1ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_account_mod_inwebo_modify.html.erb (0.1ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_account_mod_information.html.erb (0.0ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_accounts_list.html.erb (5.0ms)
Oct 21 16:48:11 HOST [24257]: Completed 200 OK in 840ms (Views: 5.5ms | ActiveRecord: 10.7ms)
Oct 21 16:48:11 HOST [527]: Completed 200 OK in 521ms (Views: 0.4ms | ActiveRecord: 17.7ms)
Oct 21 16:48:12 HOST [519]: Completed 200 OK in 954ms (Views: 0.4ms | ActiveRecord: 17.5ms)
Oct 21 16:48:16 HOST [24257]: Started GET "/SsoRole?account=user@domain.fr&ts=1571669296075&preauth=a2850ade8d7c8b94894aa0691a495c91222c1ec9" for 172.22.2.75 at 2019-10-21 16:48:16 +0200
Oct 21 16:48:16 HOST [24257]: Processing by SessionController#acquire_role as HTML
Oct 21 16:48:16 HOST [24257]: Parameters: {"account"=>"user@domain.fr", "ts"=>"1571669296075", "preauth"=>"a2850ade8d7c8b94894aa0691a495c91222c1ec9"}
Oct 21 16:48:16 HOST [24257]: Completed 200 OK in 34ms (Views: 0.2ms | ActiveRecord: 1.0ms)

Thanks for your help.

Sincerely

2

Hi @jmilot,

can you share your input configuration or make sure you are not using a pattern that might read that file multiple times?

3

Hi @pmercado,

Which configuration ?

The problem is that i have in one message :

Oct 21 16:48:11 HOST [519]: Started POST "/bss/CountObjects/domain.fr" for 10.113.1.240 at 2019-10-21 16:48:11 +0200
Oct 21 16:48:11 HOST [527]: Started POST "/bss/CountObjects/domain.fr" for 10.113.1.240 at 2019-10-21 16:48:11 +0200
Oct 21 16:48:11 HOST [603]: Started POST "/bss/CountObjects/domain.fr" for 10.113.1.240 at 2019-10-21 16:48:11 +0200
Oct 21 16:48:11 HOST [527]: Processing by BssController#count_objects as JSON
Oct 21 16:48:11 HOST [527]: Parameters: {"type"=>"dl", "domain_name"=>"domain.fr"}
Oct 21 16:48:11 HOST [519]: Processing by BssController#count_objects as JSON
Oct 21 16:48:11 HOST [519]: Parameters: {"type"=>"userAccount", "domain_name"=>"domain.fr"}
Oct 21 16:48:11 HOST [603]: Processing by BssController#count_objects as JSON
Oct 21 16:48:11 HOST [603]: Parameters: {"type"=>"worker", "domain_name"=>"domain.fr"}
Oct 21 16:48:11 HOST [603]: Completed 200 OK in 91ms (Views: 0.3ms | ActiveRecord:

Then in the following :

Oct 21 16:48:11 HOST [24257]: Rendered bss/_advancedsearch_mod.html.erb (1.2ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_account_mod_fidems.html.erb (0.1ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_account_mod_alias.html.erb (0.0ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_account_mod_change_password.html.erb (0.0ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_account_mod_move_account.html.erb (0.1ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_account_mod_inwebo_modify.html.erb (0.1ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_account_mod_information.html.erb (0.0ms)
Oct 21 16:48:11 HOST [24257]: Rendered bss/_accounts_list.html.erb (5.0ms)
Oct 21 16:48:11 HOST [24257]: Completed 200 OK in 840ms (Views: 5.5ms | ActiveRecord: 10.7ms)
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.