How can I check if an object is empty?

Elastic Stack Logstash
1

I manipulate http requests.
my event structure looks like

{
 "http_body":{
   "raw":"..."
  }
}

I detect the content type of the requests and when it's application/json, I parse the http_body as json and endup with

{
 "http_body":{
   "json":{...}
  }
}

I then move known information from [http_body][json] to predetermined locations
The json doesn't always contain only "known information" and I don't want to lose this unknown information but I would like to cleanup the other events

at the moment when all the information is known, i have events which endup with :

{
 "http_body":{
   "json":{}
  }
}

and some with

{
 "http_body":{}
}

when the json or http_body fields are "empty" I would like to be able to remove them but I cant figure out how to detect it without resorting to a ruby filter.

if ![http_body][json] is false since the field is defined but is empty.
if ![http_body][json] or [http_body][json] == {} fails with The given configuration is invalid. Reason: Expected one of #, ", ', -, [, / at line
if ![http_body][json] or ([http_body][json] == {}) fails with The given configuration is invalid. Reason: Expected one of #, ", ', -, [, / at line

the only way I found for now is

if ![http_body][json]{
    mutate {
      remove_field => [
        "[http_body][json]"
      ]
    }
  }else{
    ruby{
      code =>'if(event.get("[http_body][json]").empty?) then
                event.remove("[http_body][json]")
              end'
    }
  }

which feels quite awkward .... what have I missed ?

thanks.

2

Hi Jean_Helou,

what is if you try to add a tag if the filter failed?

        grok {
            match          => ["message", "HTTP Filter"]
            tag_on_failure => [ "_no_http" ]
        }

        grok {
            match          => ["message", "JSON Filter"]
            tag_on_failure => [ "_no_json" ]
        }

then use the tag_on_failure to remove it.

filter {
  if "_no_http" in [tags] {
    drop { }
  }
}

filter {
  if "_no_json" in [tags] {
    drop { }
  }
}

for more information, take a look here

3

My explanation wasn't clear : the json parsing and groking does work just fine.

Imagine I have the following events, which both parsed just fine

{
 "http_body":{
   "json":{
    "knownKey":"value",
    "unknownKey":"value"
   }
  }
}
{
 "http_body":{
   "json":{
    "knownKey":"value",
   }
  }
}

now I extract the information i am interested in with for further processing and specific indexing

mutate {
  rename => {["http_body"]["json"]["knownKey"] => "_knownKey"}
}

I am left with

{
 "_knownKey":"value",
 "http_body":{
   "json":{
    "unknownKey":"value"
   }
  }
}
{
 "_knownKey":"value",
 "http_body":{
   "json":{}
  }
}

I want to keep the "unknownKey":"value" in case we can identify a pattern later on and reprocess it, but in the second event I was able to process all of the http_body so I would like to clean up.

the processing is applied to many different payloads which share some common elements and many different elements. I don't really know in advance which payloads will have unknownKeys and which won't...(otherwise it wouldn't be an unknown key :slight_smile: )

4

ah ok, sorry :frowning:

did you try something like that ->

IF !([http_body][json] =~ /.+/).

link for more information here

1 Like
5

Thanks for the suggestion, I ended up using

if ![http_body][json] or !([http_body][json] =~ ".+")

which worked.

1 Like
6

Hi Jean_Helou,

nice to hear that its working now for you :slight_smile:

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.