Parsing logfiles

[SIRAJEDDINE-HAMZA]

I'm new to using ElasticStack and I'm having trouble parsing a log file using Logstash. Specifically, I want to split the file using the timestamp as a separator and extract data from each block, but I'm not sure how to go about it.

Here's a sample of the file i'm working with :

2019-05-31T14:54:50.186759-04:00
Errors in file /u01/app/oracle/diag/rdbms/orclcdb/orclcdb/trace/orclcdb_ora_3384.trc:
ORA-00313: open failed for members of log group 1 of thread 1
ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/ORCLCDB/redo01.log'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 7
2019-05-31T14:54:50.186870-04:00
Errors in file /u01/app/oracle/diag/rdbms/orclcdb/orclcdb/trace/orclcdb_ora_3384.trc:
2019-05-31T15:18:50.433792-04:00
ALTER SYSTEM SET parallel_max_servers=1 SCOPE=BOTH;
2019-05-31T15:18:51.872284-04:00
alter pluggable database orcl open
ORA-65019 signalled during: alter pluggable database orcl open...
2019-05-31T15:18:52.976230-04:00
alter pluggable database orcl save state
Completed: alter pluggable database orcl save state
2019-05-31T14:54:50.186870-04:00
Errors in file /u01/app/oracle/diag/rdbms/orclcdb/orclcdb/trace/orclcdb_ora_3384.trc:
2019-05-31T15:18:50.433792-04:00
ALTER SYSTEM SET parallel_max_servers=1 SCOPE=BOTH;
2019-05-31T15:18:51.872284-04:00
alter pluggable database orcl open
ORA-65019 : signalled during: alter pluggable database orcl open...
ORA-65017 : signalled during: alter pluggable database orcl open...
2019-05-31T15:18:52.976230-04:00
alter pluggable database orcl save state
Completed: alter pluggable database orcl save state
2019-05-31T15:18:51.872284-04:00
alter pluggable database orcl open
ORA-65019 : signalled during: alter pluggable database orcl 

And here's the output I'm looking to extract :

{
  "@timestamp" => "2019-05-31T18:54:50.186759Z",
  "ora_errors" => [
            [0] "ORA-00313: open failed for members of log group 1 of thread 1",
            [1] "ORA-00312: online log 1 thread 1: '/u01/app/oracle/oradata/ORCLCDB/redo01.log'",
            [2] "ORA-27037: unable to obtain file status"
  ]
},
{
  "@timestamp" => "2019-05-31T15:18:51.872284-04:00",
  "ora_errors" => [
            [0] "ORA-65019 signalled during: alter pluggable database orcl open..."
  ]
},
{
  "@timestamp" => "2019-05-31T15:18:51.872284-04:00",
  "ora_errors" => [
            [0] "ORA-65019 : signalled during: alter pluggable database orcl open...",
			[1] "ORA-65017 : signalled during: alter pluggable database orcl open..."
  ]
},
{
  "@timestamp" => "2019-05-31T15:18:51.872284-04:00",
  "ora_errors" => [
            [0] "ORA-65019 : signalled during: alter pluggable database orcl"
  ]
}

If you have any tips or links to share I would really appreciate it.

[Badger]

Assuming you are reading the file with a file input, then use a multiline codec

codec => multiline {
    pattern => "^\d{4}-\d{2}-\d{2}"
    negate => true
    what => previous
    auto_flush_interval => 2
}

Then you could extract the errors using this. Note that .scan returns an array of arrays which contain each of the capture groups. A second capture group is needed to match both end-of-line and end-of-message, so we also extract just the first member of each inner array.

    mutate { add_field => { "[@metadata][ts]" => "%{message}" } }
    mutate { gsub => [ "[@metadata][ts]", "\n.*", "" ] }
    date { match => [ "[@metadata][ts]", "YYYY-MM-dd'T'HH:mm:ss.SSSSSSZZ" ] }

    ruby {
        code => '
            matches = event.get("message").scan(/^(ORA-\d{5}[^\n]+)(\n|$)/)
            a = []
            matches.each { |x| a << x[0] }
            if a != []; event.set("ora_errors", a); end
        '
    }
    if ! [ora_errors] { drop {} }

[SIRAJEDDINE-HAMZA]

Thank you very much for the help