# Help Setting Up JSON Filter

**URL:** <https://discuss.elastic.co/t/help-setting-up-json-filter/123363>\
**Category:** Logstash\
**Created:** [March 9, 2018, 11:50pm UTC](https://discuss.elastic.co/t/help-setting-up-json-filter/123363 "2018-03-09T23:50:02Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kurorion](https://avatars.discourse-cdn.com/v4/letter/k/8edcca/32.png) [@Kurorion](https://discuss.elastic.co/u/Kurorion)\
**Post date:** [March 9, 2018, 11:50pm UTC](https://discuss.elastic.co/t/help-setting-up-json-filter/123363/1 "2018-03-09T23:50:02Z")

</div>

Hello, I have a JSON array in the following format that I need to create a set of filters for to get the desired output. The JSON is using the SenML format if anyone is curious.

```
[
    {"bname":"urn:dev:ow:10e2073a0108006:","btime":1276020076,"bunit":"A","bversion":5,"name":"voltage","unit":"V","value":120.1},
    {"name":"current","time":-5,"value":1.2},
    {"name":"current","time":-4,"value":1.3},
    {"name":"current","time":-3,"value":1.4}
]

```

The first item in the JSON array contains **base name** (bname), **base time** (btime), **base unit** (bunit) and **base version** (bversion). These values set the **default value** for "non b" values in the later items.

For example, the desired event output for the first 2 items in the JSON array should be:

```
{
          "name" => "voltage",
         "bunit" => "A",
         "value" => 120.1,
         "bname" => "urn:dev:ow:10e2073a0108006:",
        "bttime" => 1276020076,
          "unit" => "V",
       "bverion" => 5,
      "@version" => 5,
    "@timestamp" => 2010-06-08T18:01:16.000Z
}
{
          "name" => "current",
         "bunit" => "A",
         "value" => 1.2,
         "bname" => "urn:dev:ow:10e2073a0108006:",
         "btime" => 1276020076,
          "unit" => "A",
       "bverion" => 5,
      "@version" => 5,
    "@timestamp" => 2010-06-08T18:01:11.000Z
}

```

But using the default JSON codec I get the following output for the first 2 items in the array:

```
{
          "name" => "voltage",
         "bunit" => "A",
         "value" => 120.1,
         "bname" => "urn:dev:ow:10e2073a0108006:",
        "bttime" => 1276020076,
          "unit" => "V",
       "bverion" => 5,
      "@version" => 5,
    "@timestamp" => 2010-06-08T18:01:16.000Z
}
{
          "name" => "current",
          "time" => -5,
         "value" => 1.2,
      "@version" => "1",
    "@timestamp" => 2018-03-09T23:01:06.951Z
}

```

This is my pipeline configuration:

```
input {
  stdin {}
}

filter {  
  # Parse "bt" timestamp as UNIX timestamp
  date {
    match => ["btime", "UNIX"]
  }
  
  # Parse "bver" version
  mutate {
     copy => { "bver" => "@version" }
  }
}

output {
  stdout { codec => "rubydebug" }
}

```

Any help with this would be greatly appreciated. I'm having a lot of trouble figuring out how to insert/append the base values into other array items that do not have them.

Thanks

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [March 10, 2018, 12:10am UTC](https://discuss.elastic.co/t/help-setting-up-json-filter/123363/2 "2018-03-10T00:10:19Z")

</div>

Because of the typical shapes of input Logstash handles, the `json` codec makes some assumptions that don't seem to fit your data; when it receives a top-level _array_, it assumes that each object inside that array is a stand-alone event (which isn't the case here).

I've taken a look at the [SenML](https://tools.ietf.org/html/draft-ietf-core-senml-13) working draft, and I don't think it would be terribly hard to write a codec to handle this input, but it would end up being custom code; it may be as easy as wrapping the MIT-licensed [senml-parser](https://github.com/gobi-project/senml-parser/blob/master/lib/senml_parser.rb) gem. (caveat: if you end up using the CBOR representation, that gem may not work on jruby)

From the working draft of SenML, it appears as though each "stream" of measurement readings must be a valid JSON array (e.g., it doesn't support a continuous stream of newline-delimited objects); depending on the size of your input, it may be helpful to use a streaming parser which is capable of emitting top-level elements before it has read all the way to the end.

---

<div class="post-metadata">

**Author:** ![Kurorion](https://avatars.discourse-cdn.com/v4/letter/k/8edcca/32.png) [@Kurorion](https://discuss.elastic.co/u/Kurorion)\
**Post date:** [March 12, 2018, 3:57pm UTC](https://discuss.elastic.co/t/help-setting-up-json-filter/123363/3 "2018-03-12T15:57:31Z")

</div>

Thank you for the reply Ry. I'll look into your suggestions.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 9, 2018, 3:57pm UTC](https://discuss.elastic.co/t/help-setting-up-json-filter/123363/4 "2018-04-09T15:57:32Z")

</div>

Closed on April 9, 2018, 3:57pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.