Drop filter erros

Elastic Stack Logstash
1

I am using the dissect filter to ingest csv files into elastic
I used the drop filter to eliminate the headers.

filter {
       if [Email] == "Email" { drop{} }
}

but when I look at the index there where still documents with the Email as keyword.

that should not be possible since it they should be dropped(which most where).

when I locked at the json ths is twhat it shows

  "_source": {
    "Email": "Email\r",

I tried

filter {
       if [Email] == "Email\r" { drop{} }
}

but no luck.

also

filter {
       if [Email] == "Email\r" { drop{} }
}

but there is one doc that still shows up.
any help would be great

2

That would match the string "Email+backslash+r". You do not have that, you have "Email+CR" (carriage return). You have options

  1. Set config.support_escapes in logstash.yml, but beware this is global so it might affect other parts of your pipeline. \r will then be treated as a literal CR

  2. Do a regexp match, where I think \r will match a CR.

    if [Email] =~ "Email\r?" { drop {} }

  3. I had a third option but it slips my mind right now....

3

Perhaps the 3rd option is

mutate
    {
        gsub => [ 'message',"\r", '']
        gsub => [ 'message',"\r\n", '']
    }

if [Email] == "Email" { drop {} }

However 2nd is the most suitable = regex.

4

that worked.
also that last section should have been

filter {
       if [tags] == "_dissectfailure" { drop{} }
}

it should drop all rows that generate this tag but there is still one in the index.

5

also is there a way to mutate all the CR's

I noticed that all multiple rows have them

also i noticed one doc with

image

that got indext. do you know what the dimond ? is and how to get rid of it?

I tried to copy and paste it but it got rid of the question mark

6

Change to:
if "_dissectfailure" in [tags] { drop{} }

Try with:

mutate
    { gsub => [ 'email',"<put your spec>", ""] }

If is not working, then use regex someting like: ^. - for any 1st character or perhaps: ^\W , not sure what is a content of the email field.