Logstash is not sending data to the output

Elastic Stack Logstash
1

Hi,
This is the message source is sending to the logstash

<132>Mar 13 23:50:32 k8s-93f066ee-9b76 {\"projectid\":\"JHAR_BV\",\"_kmd\":{\"lmt\":\"2022-03-13T23:50:26:388Z\",\"ecf\":\"2022-03-13T23:50:26:388Z\"},\type\":\"warn\",\"createdOn\",\"2022-03-13T23:50:26:388Z\",\"deviceid\":\"web\",\"userid\":\:\"65678g54445s666544\,\"appVersion\":\"0.0\",\"platform\":\"web\",\"message\":\[\\\"\\\n It looks like you're using disabled attributes with reactive from directive.\\\\n\\\"]\",\_id\":\"65677vg65566\",\"app_key\":\"kid\"}"

I want to filter on the basis of projectid, the project id which has value "JHAR_BV" should go to output, all other should drop.
I created two grok filter for this which look like

filter {
    grok {
        match => { "message" => "%{SYSLOG5424PRI}%{MONTH:month} +%{NONNEGONT:day:int} +%{TIME:time} +%{DATA:kis} %{GREEDYDATA:json_payload}"}
match => { "message" => "%{SYSLOG5424PRI:number}%{CISCOTIMESTAMP:timestmp} %{DATA:id} %{GREEDYDATA:json_payload}"}

}
json {
  source => "json_payload"
}
if [projectid] != "JHAR_BV" {
    drop { }
}
}

Now the problem is I am not getting any error while starting the logstash, also data is not flowing to the output. I am not sure if the issue is with my grok or my applied logic is not correct?

Please help me.
Thanks in advance

2

Can anyone please help me as I am struggling to get the output. If required qny additional info please let me know.

Than you

3

You need to share your full pipeline and also the output you are getting, there is not enough information to know what is happening.

First you need to check if your grok and json filters are working and you have the field projectid, if you do not have this field, logstash will drop everything because your conditional will always be true.

Also, there is a type in one of your grok patterns, NONNEGONT, it is NONNEGINT.

4

Hi @leandrojmp

This is my pipeline config and I want to drop all messages which don't have projectid=JHAR_BV. Updated the config but still getting nothing in the output.

input { 
        tcp {
               port => 1027
}
}

filter {
    grok {
        match => { "message" => "%{SYSLOG5424PRI}%{MONTH:month} +%{NONNEGINT:day:int} +%{TIME:time} +%{DATA:kis} %{GREEDYDATA:json_payload}"}
        match => { "message" => "%{SYSLOG5424PRI:number}%{CISCOTIMESTAMP:timestmp} %{DATA:id} %{GREEDYDATA:json_payload}"}

}
json {
  source => "json_payload"
}
if [projectid] != "JHAR_BV" {
    drop { }
}
}
output {
          stdout { codec => rubydebug }
}

I am trying with both grok one by one but not getting output in any case. Also not getting any error while starting the logstash.

Thank you

5

You need to remove the drop part of your pipeline to be able to troubleshoot it, if there is something wrong in your grok, logstash will drop everything.

Remove this part to see what logstash is receiving and outputing.

if [projectid] != "JHAR_BV" {
    drop { }
}
6

Hi @leandrojmp Thank you

After removing that part logstash is sending to the output. The issue is with drop part.

Now how the messages that doesn't contain the projectid=JHAR_BV would drop?

I only need the message that contains projectid=JHAR_BV in the output.

Please help

Thank you

7

As I said, without seeing what is your real output there is not much to help.

Your conditional would drop any message where the value for field projectid is not JHAR_BV, does your message have this field with this exact name and with this exact value?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.