How to skip some lines from a log file without skipping the complete message

[Daniel_Gonzalez1 (Daniel Gonzalez)]

Hello Everyone!

I'm working with a file that has some lines that I want to skip when they come to logstash.

These are a sample of the log lines:

> 15Jan20 11:34:10.39 PRHTGRCE MXP1 RACF CONNECT success for PRHTGRCE: CONNECT C067684
>    Jobname + id: PRHTGRCE
>    RACF command: CONNECT C067684 AUTHORITY(USE) GROUP(ATRBO01) NOADSP NOAUDITOR NOGRPACC NOOPERATIONS NOSPECIAL OWNER(ATRBO01) RESUME UACC(NONE)
>    Name        : USER PARA GRUPO CER    Instdata    : USUARIO PARA LA CONEXION DE GRUPOS RACF REP. OSI
> 
>  15Jan20 11:34:14.61 PRHTGRCE MXP1 RACF ALTUSER success for PRHTGRCE: ALTUSER C110250
>    Jobname + id: PRHTGRCE
>    RACF command: ALTUSER C110250 NOUAUDIT
>    Name        : USER PARA GRUPO CER    Instdata    : USUARIO PARA LA CONEXION DE GRUPOS RACF REP. OSI
> 1zSecure Audit for RACF user events  15Jan20 00:31 to 15Jan20 18:36                                                         list   51
>  SMF records for all users with successful commandswq
> 
>  Date/time           User     Sys  Description
> 
>  15Jan20 11:34:15.14 PRHTGRCE MXP1 RACF CONNECT success for PRHTGRCE: CONNECT C110250
>    Jobname + id: PRHTGRCE
>    RACF command: CONNECT C110250 AUTHORITY(USE) GROUP(ATRBO01) NOADSP NOAUDITOR NOGRPACC NOOPERATIONS NOSPECIAL OWNER(ATRBO01) RESUME UACC(NONE)
>    Name        : USER PARA GRUPO CER    Instdata    : USUARIO PARA LA CONEXION DE GRUPOS RACF REP. OSI

15Jan20 11:51:28.40 PRHTGRCE MXP1 RACF ALTUSER success for PRHTGRCE: ALTUSER C067734
1zSecure Audit for RACF user events  15Jan20 00:31 to 15Jan20 18:36                                                         list   52
 SMF records for all users with successful commands

 Date/time           User     Sys  Description
   Jobname + id: PRHTGRCE
   RACF command: ALTUSER C067734 NOUAUDIT
   Name        : USER PARA GRUPO CER    Instdata    : USUARIO PARA LA CONEXION DE GRUPOS RACF REP. OSI

The thing that I'm looking for is to omit the next lines:

1zSecure Audit for RACF user events  15Jan20 00:31 to 15Jan20 18:36                                                         list   51
>  SMF records for all users with successful commandswq
> 
>  Date/time           User     Sys  Description

without skipping the above or the continue of the log data. It suppose that the info that I want to omit are header of a report, this headers comes paste with the logs

Each log line starts with the date

15Jan20 11:34:15.14

I used the next config to try to skip all the lines that came with those variables, but it skips the whole message.

input {
    file {
        path => [ "/home/linux/Downloads/reporte_mf" ]
        start_position => "beginning"
        sincedb_path => "/dev/null"
        codec => multiline {
        pattern => "(?<date>\s\d+\d+\w+\d\d\s\d+:\d+:\d+.\d+)"
        negate => true
        what => previous

}
    }
}
# The filter part of this file is commented out to indicate that it is
# optional.
 filter {
      grok{
              match => { "message" => "\s(?<date>\d\d\w+\d\d\s\d+:\d+:\d+.\d+)\s" }

}

      if [message] =~ /(1zSecure.+| SMF records for all users with successful commands| Date\/time.+)/ {
         drop {} }
}

output {

    stdout { codec => rubydebug }
}

For example, if in the message the confing finds some of the lines I want to skip, it omit that message, not only the lines I want to skip

Is it possible to erase or omit those lines of the log, without skipping the complete message?

[Badger]

If there are specific lines you want to omit from events you could use mutate+gsub to remove them.