How to skip second row in log file while processing it through CSV filter

[Ashish_kapoor]

Hi

I want to skip second line in my logs while processing these logs with CSV filter.

second line of my logs looks like this:

#Remark Values: ComponentType="OAM" ReleaseVersion="11.1.1.9.0"

and logstash conf for processing this file is

input {
    pipeline { 
        address => OAMAudit
    }  
}
filter {
    mutate {
        gsub => ["message","\\\"","'"]
    }
    csv {
        columns => ["Date", "Time", "Initiator", "EventType", "EventStatus", "MessageText", "AuditUser", "AdditionalInfo", "AdminRoleName", "AgentID", "AgentType", "ApplicationDomainName", "ApplicationName", "AuthenticationMethod", "AuthenticationPolicyID", "AuthenticationSchemeID", "AuthorizationPolicyID", "AuthorizationScheme", "ClientIPAddress", "ConstraintType", "ContextFields", "DataSourceName", "DataSourceType", "DomainName", "ECID", "EventCategory", "FailureCode", "GenericAttribute1", "GenericAttribute2", "GenericAttribute3", "GenericAttribute4", "GenericAttribute5", "HomeInstance", "HostId", "HostIdentifierName", "HostNwaddr", "IdentityDomain", "Impersonator", "InstanceName", "NewAttributes", "NewSettings", "OldAttributes", "OldSettings", "PolicyAdminContext", "PolicyName", "PolicyObjectID", "PolicyType", "ProtectionLevel", "RID", "ReadOnly", "RemoteIP", "RequestID", "Resource", "ResourceHost", "ResourceHostName", "ResourceID", "ResourceOperations", "ResourceTemplateName", "ResourceType", "ResourceURI", "ResponseType", "Roles", "SSOSessionID", "SchemeName", "ServerName", "ServiceIdentifier", "ServiceOperation", "ServiceURI", "SessionCreationTime", "SessionExpirationTime", "SessionID", "SessionLastAccessTime", "SessionLastUpdateTime", "Target", "TargetComponentType", "TenantId", "ThreadId", "TransactionId", "UserDN", "UserID", "UserTenantId"]
        separator => " "
        skip_empty_columns => "true"
        skip_empty_rows => "true"
        skip_header => "true"
    }
     if [Date] == "#Remark" {
        drop { }
    }
    mutate {
      	add_field => {
   		"timestamp2" => "%{Date} %{Time}"
       }
    }
    date {
          match => ["timestamp2", "yyyy-MM-dd HH:mm:ss.SSS", "dd-MM-yyyy HH:mm:ss"]
                    target => "@timestamp"
         }
    mutate {
      remove_field => ["@version","path","host" ]
    }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "oam_auditlogs"
  }
}

[Badger]

@Ashish_kapoor

if [Date] == "#Remark" { drop { } }

That looks like a good way to do it. What is your question?

[Ashish_kapoor]

Hi

I am not able to skip this line with below code

if [Date] == "#Remark" {
        drop { }
    }

[Badger]

You could try

if [message] =~ /^#Remark/ { drop {} }

before the csv filter.

Are the fields in the CSV wrapped in quotes?

[Ashish_kapoor]

Thanks for help. I able to resolve this issue, But I need to delete header and second line in message Manually.