Logstash if message is empty, drop the entire row

[katara]

Hello,
I have a scenario where my Log messages are empty in a few cases:

So what I want to do is, If message is empty, then drop the whole row.

I tried

 filter {
      if [Message] == "" {
        drop { }
      }
    }

which eliminated all the message field , which is not what i want.

Please help me out.

Katara

[Fabio-sama (Fabio)]

Hi Katara,

are you sure the field you want to make the check on is Message? Can you post here an example of a two outputs (without the drop filter applied), one of a document with something in that Message field, and one with a void Message field (so that should be discarded)?

Thank you

[katara]

Hi @Fabio-sama,
The image attached in my question is when I use no Message filter. This includes the blank messages as well.

The below attached is where i applied the filter,

it simply did not have a field called message.

I want to ignore the entire row and not the field, when the message is blank.

Thanks!

Katara

[Fabio-sama (Fabio)]

I'm sorry Katara. I didn't really get what you want.

In your first pic I can see some lines with an empty field Message and you said you didn't apply the filter there, so it's ok.

In your last pic you said you have applied the filter, in fact I don't see the rows with blank Message field anymore.

What am I missing?

[katara]

@Fabio-sama,
In the second picture, there is no message field at all.

What I want to do is not remove the field "message"
But rather remove the entire row where there is no message.
So I'd have no blank messages in my index.
I basically want to eliminate all my empty messages.

Thanks,
Katara

[Fabio-sama (Fabio)]

Ok so, if I got it right, the second pic is the one you want, but you would like it to include also the Message field, which now is not shown (not even scrolling horizontally), right?

Though, I don't get why in your first pic you don't have some fields you do have in your second pic (like SourceModuleName, SourceModuleType, @version). You sure the only difference between the two pipelines generating the outputs of the two pics is only in the drop{} part?

Can you post here the logstash pipeline?

And sorry if I insist, but can you post also the output (not the pic of the landing db, but the output in the standard output) of the pipeline? I'd like to see what Logstash spits out in both case (with a void Message field and with a populated Message field).

Thanks

[katara]

Hi @Fabio-sama,
Sure, Sorry , I had an image posted, understand it isnt very clear.

Here's my current Logstash configuration:

input {
        tcp {
                port => 8443
                codec => json_lines { charset => CP1252 }
                }
}
filter {
      if [Message] == "" {
        drop { }
      }
    }
output {
if [ApplicationName] == "OASIS"
{
elasticsearch {
    hosts => ["10.56.5.266:9200"]
    index => "nxlogappsoasis"
}
}
else
{
elasticsearch {
    hosts => ["10.56.5.266:9200"]
    index => "nxlogapps"
}
}

For this my output is,

{

* "_index": "nxlogappsoasis",
* "_type": "_doc",
* "_id": "oeDCpHABxa029eIHyPuj",
* "_version": 1,
* "_score": 1,
* "_source": {
  * "OSType": "Windows",
  * "@timestamp": "2020-03-04T08:56:33.592Z",
  * "tag_filepath": "D:\OrdersWCF\*",
  * "port": 53642,
  * "host": "ICDWP",
  * "SourceModuleName": "oasisdig",
  * "SourceModuleType": "im_file",
  * "@version": "1",
  * "EventReceivedTime": "2020-03-04 03:56:33.682089-05:00",
  * "ApplicationName": "OASIS",
  * "Message": "Timestamp: 3/4/2020 3:56:32 AM: Message: Core: ValidateBillTo Method Start - 0002309223 ---------------------------------------- ----------------------------------------",
  * "Technology": ".NET",
  * "EventName": "Oasis Event",
  * "Severity": "Info",
  * "Hostname": "ICDWP",
  * "tag_nxlog": "nxlog"}

}
{

* "_index": "nxlogappsoasis",
* "_type": "_doc",
* "_id": "oeDCpHABxa029eIHyPuj",
* "_version": 1,
* "_score": 1,
* "_source": {
  * "OSType": "Windows",
  * "@timestamp": "2020-03-04T08:56:33.592Z",
  * "tag_filepath": "D:\Orders\*",
  * "port": 53642,
  * "host": "ICDWP",
  * "SourceModuleName": "oasisdig",
  * "SourceModuleType": "im_file",
  * "@version": "1",
  * "EventReceivedTime": "2020-03-04 04:46:03.682089-05:00",
  * "ApplicationName": "OASIS",
  * "Technology": ".NET",
  * "EventName": "Oasis Event",
  * "Severity": "Info",
  * "Hostname": "ICDWP",
  * "tag_nxlog": "nxlog"}

}

What i want for is if there is no message field, the second set of Jason should not come at all, instead of skipping only the Message field.

Thanks!

Katara

[Fabio-sama (Fabio)]

And when there is no message field, does it appear as a empty string or doesn't it appear at all?
What if you run this pipeline?

input {
  tcp {
    port => 8443
    codec => json_lines { charset => CP1252 }
  }
}

filter {
  if ![Message] {
    drop { }
  }
}

output {
  if [ApplicationName] == "OASIS" {
    elasticsearch {
      hosts => ["10.56.5.266:9200"]
      index => "nxlogappsoasis"
    }
  } else {
    elasticsearch {
      hosts => ["10.56.5.266:9200"]
      index => "nxlogapps"
    }
  }
}

[katara]

@Fabio-sama,
when there is no Message field, the field simply doesnt exist.
But since it gets upserted with the ones that have actual message as well, they show up as blank since there is no value.

The ignore statement also gives me a no message field. But doesn't drop the row entirely.

I used a regex in my nxlog like

> if ($raw_event =~ /^\s*$/) {drop ();}

Which seems to do the job!!
Thanks very much for your time and help

[Fabio-sama (Fabio)]

Ok that is what I wanted to know.

If the field doesn't exist, you cannot compare it to a void string.

So it'll never enter the if [Message] == "" statement.

The right condition would be something like if ![Message], which is the syntax for if there's no field Message in this event.