Parse Logs in Logstash

[SerSSH (Serjeif)]

Hi guys, I need help with the parsing of some logs. I have tried to do them with the grok plugin, however I need to optimize this for a large number of logs that have different formats and different fields or variables. The variables are currently separated by spaces between them, so the kv plugin has not worked for me, this only brings me the first word that is close to the colon (:). A log is as follows:

Jan 20 17:40:04 btasdbora04 Oracle Audit[7687]: [ID 748625 local1.warning] LENGTH : '422' ACTION :[97] ' SELECT :"SYS_B_0", status, :"SYS_B_1",archiver, database_status, active_state FROM v$instance ' DATABASE USER:[3] 'SYS' PRIVILEGE :[6] 'SYSDBA' CLIENT USER:[6] 'oracle' CLIENT TERMINAL:[7] 'unknown' STATUS:[1] '0' DBID:[9] '365542621' SESSIONID:[10] '4294967295' USERHOST:[22] 'Domain' CLIENT ADDRESS:[58]

I try to parse this log with kv

filter {
kv {
field_split => ","
value_split => ":"
}
}
but the space between variable is the problem: CLIENT USER:[6] CLIENT TERMINAL:[7]

Thanks for you time.

[Badger]

Hard to see how that is a kv format. Without dozens of more examples it is hard to say how it is any format.

[pastechecker (Charlie)]

You will not get the expected results.
Look on your log and you will see that the field for example does not have a key:value pair:

", status, : 

Can you please parse your example manually into fields you would like to have?

[SerSSH (Serjeif)]

Of course, in this example I need to get the fields

CLIENT USER=oracle
PRYVILEGE= SYSDBA
DATABASE USER = SYS
CLIENT TERMINAL = unknown

But i need to get this fiedls without use grok.

There are many logs with different structure and this is a problem to make groks for each. I have made grok for this log structure, however there are several that do not parse.

I want to know if is possible appplicate any filter for all the logs.

[pastechecker (Charlie)]

You would have to adjust your data structure to get it right, otherwise you will never going to stop parsing it.

One possible regex pattern you could use to specify the delimiter is:

[A_Z]{1,}"[

So take the part with grok:

"message_header": "Jan 20 17:40:04 btasdbora04 Oracle Audit[7687]: [ID 748625 local1.warning]"

"message_for_kv":"LENGTH : '422' ACTION :[97] ' SELECT :"SYS_B_0", status, :"SYS_B_1",archiver, database_status, active_state FROM v$instance ' DATABASE USER:[3] 'SYS' PRIVILEGE :[6] 'SYSDBA' CLIENT USER:[6] 'oracle' CLIENT TERMINAL:[7] 'unknown' STATUS:[1] '0' DBID:[9] '365542621' SESSIONID:[10] '4294967295' USERHOST:[22] 'Domain' CLIENT ADDRESS:[58]" 

and try to work from this point...
But I would rather focus on fixing the input rather than parsing it.