Help Setting Up JSON Filter

Hello, I have a JSON array in the following format that I need to create a set of filters for to get the desired output. The JSON is using the SenML format if anyone is curious.

[
    {"bname":"urn:dev:ow:10e2073a0108006:","btime":1276020076,"bunit":"A","bversion":5,"name":"voltage","unit":"V","value":120.1},
    {"name":"current","time":-5,"value":1.2},
    {"name":"current","time":-4,"value":1.3},
    {"name":"current","time":-3,"value":1.4}
]

The first item in the JSON array contains base name (bname), base time (btime), base unit (bunit) and base version (bversion). These values set the default value for "non b" values in the later items.

For example, the desired event output for the first 2 items in the JSON array should be:

{
          "name" => "voltage",
         "bunit" => "A",
         "value" => 120.1,
         "bname" => "urn:dev:ow:10e2073a0108006:",
        "bttime" => 1276020076,
          "unit" => "V",
       "bverion" => 5,
      "@version" => 5,
    "@timestamp" => 2010-06-08T18:01:16.000Z
}
{
          "name" => "current",
         "bunit" => "A",
         "value" => 1.2,
         "bname" => "urn:dev:ow:10e2073a0108006:",
         "btime" => 1276020076,
          "unit" => "A",
       "bverion" => 5,
      "@version" => 5,
    "@timestamp" => 2010-06-08T18:01:11.000Z
}

But using the default JSON codec I get the following output for the first 2 items in the array:

{
          "name" => "voltage",
         "bunit" => "A",
         "value" => 120.1,
         "bname" => "urn:dev:ow:10e2073a0108006:",
        "bttime" => 1276020076,
          "unit" => "V",
       "bverion" => 5,
      "@version" => 5,
    "@timestamp" => 2010-06-08T18:01:16.000Z
}
{
          "name" => "current",
          "time" => -5,
         "value" => 1.2,
      "@version" => "1",
    "@timestamp" => 2018-03-09T23:01:06.951Z
}

This is my pipeline configuration:

input {
  stdin {}
}

filter {  
  # Parse "bt" timestamp as UNIX timestamp
  date {
    match => [ "btime", "UNIX" ]
  }
  
  # Parse "bver" version
  mutate {
     copy => { "bver" => "@version" }
  }
}

output {
  stdout { codec => "rubydebug" }
}

Any help with this would be greatly appreciated. I'm having a lot of trouble figuring out how to insert/append the base values into other array items that do not have them.

Thanks

Because of the typical shapes of input Logstash handles, the json codec makes some assumptions that don't seem to fit your data; when it receives a top-level array, it assumes that each object inside that array is a stand-alone event (which isn't the case here).

I've taken a look at the SenML working draft, and I don't think it would be terribly hard to write a codec to handle this input, but it would end up being custom code; it may be as easy as wrapping the MIT-licensed senml-parser gem. (caveat: if you end up using the CBOR representation, that gem may not work on jruby)

From the working draft of SenML, it appears as though each "stream" of measurement readings must be a valid JSON array (e.g., it doesn't support a continuous stream of newline-delimited objects); depending on the size of your input, it may be helpful to use a streaming parser which is capable of emitting top-level elements before it has read all the way to the end.

Thank you for the reply Ry. I'll look into your suggestions.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.