# Another Logstash Json Filter

**URL:** <https://discuss.elastic.co/t/another-logstash-json-filter/176586>\
**Category:** Logstash\
**Created:** [April 12, 2019, 9:08am UTC](https://discuss.elastic.co/t/another-logstash-json-filter/176586 "2019-04-12T09:08:07Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mrde](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@Mrde](https://discuss.elastic.co/u/Mrde)\
**Post date:** [April 12, 2019, 9:08am UTC](https://discuss.elastic.co/t/another-logstash-json-filter/176586/1 "2019-04-12T09:08:07Z")

</div>

Hello,

Sorry to add another topic on this subject but even by reading many posts about this, i didn't succeed to solve my issue : Time to learn from community 🙂

Here is an exemple of json input that i don't succeed to parse :

```auto
{
    "Meta Data": {
        "1. Information": "Informations (label1, label2, label3, label4)",
        "2. Parameter1": "Value1",
        "3. Parameter2": "Value2",
         "5. Last Refreshed": "2019-04-12 16:45:00",
        "6. Time Zone": "GMT+8"
    },
    "Time Series": {
        "2019-04-12": {
            "1. label1": "1.0000",
            "2. label2": "1.1111",
            "3. label3": "1.2222",
        },
        "2019-04-11": {
            "1. label1": "1.0000",
            "2. label2": "1.1111",
            "3. label3": "1.2222",
        },
        "2019-04-10": {
            "1. label1": "1.0000",
            "2. label2": "1.1111",
            "3. label3": "1.2222",
        }

```

I tried many different things such as using json\_line codec, different multiline pattern, ranaming fields one by one, ...

Thanks for your help,  
Regards,  
Mark

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 12, 2019, 12:15pm UTC](https://discuss.elastic.co/t/another-logstash-json-filter/176586/2 "2019-04-12T12:15:34Z")

</div>

What have you tried, and what don't you like about the result?

---

<div class="post-metadata">

**Author:** ![Mrde](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@Mrde](https://discuss.elastic.co/u/Mrde)\
**Post date:** [April 12, 2019, 12:43pm UTC](https://discuss.elastic.co/t/another-logstash-json-filter/176586/3 "2019-04-12T12:43:21Z")

</div>

I have tried many different things I found in other topics but there must be something I do not understand.

Here is an example :

```auto
input {
	file {
		type => "json" #I also triend json_line
		path => "/home/user/data/test.json"
		codec => multiline {
           pattern => "^\{" #I tried different things here
           negate => "true"
           what => "previous"
		}
		start_position => "beginning"
		sincedb_path => "/dev/null"
	}
filter {
  json { source => "Time Series" }
  date {
    match => ["[Time Series][*]", "yyyy-MM-dd" ] #I tried many different things here but i don't really know what to put as there is no label
    target => "timestamp"
  }
}
output {
  elasticsearch {
    action => "index"
    hosts => "localhost:9200"
    index => "my_index"
  }
  stdout {
     codec => rubydebug
  }

#I have tried using more or less parameter, tried with Grok filter and tried with ruby (but i'm not an expert at all).

```

I do not get any result as it print big lines full of errors. (Ex : JSON parse error)  
Sometimes I am just not getting the right information in the index, ...

Thanks for your help,  
Regards,  
Mark

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 12, 2019, 1:13pm UTC](https://discuss.elastic.co/t/another-logstash-json-filter/176586/4 "2019-04-12T13:13:32Z")

</div>

Do you want to consume the entire file as a single event?

---

<div class="post-metadata">

**Author:** ![Mrde](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@Mrde](https://discuss.elastic.co/u/Mrde)\
**Post date:** [April 12, 2019, 3:11pm UTC](https://discuss.elastic.co/t/another-logstash-json-filter/176586/5 "2019-04-12T15:11:06Z")

</div>

I would like to get those informations labeled with the corresponding timestamp.

```auto
"2019-04-12": {
            "1. label1": "1.0000",
            "2. label2": "1.1111",
            "3. label3": "1.2222",
        },
        "2019-04-11": {
            "1. label1": "1.0000",
            "2. label2": "1.1111",
            "3. label3": "1.2222",
        },

```

@Timestamp = 2019-04-12 - Label1 = 1.0000  
@Timestamp = 2019-04-12 - Label2 = 1.1111  
@Timestamp = 2019-04-12 - Label3 = 1.2222

The goal of this is to chart the evolution of label1, label2 and label3 with 3 different plot 🙂

Thanks,  
Mark

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 12, 2019, 4:22pm UTC](https://discuss.elastic.co/t/another-logstash-json-filter/176586/6 "2019-04-12T16:22:04Z")

</div>

Do you want to consume the entire file as a single event, or are there multiple JSON objects in the file?

The JSON that you show is not valid JSON (missing trailing }, and you cannot have , immediately before a }. Is your JSON actually valid, or do you need to mutate it before parsing it?

---

<div class="post-metadata">

**Author:** ![Mrde](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@Mrde](https://discuss.elastic.co/u/Mrde)\
**Post date:** [April 15, 2019, 8:24am UTC](https://discuss.elastic.co/t/another-logstash-json-filter/176586/7 "2019-04-15T08:24:23Z")

</div>

Hello,

We can consume it as a single event, there is only one type of object in the file.

My json file is supposed to be correct :  
Here is a correct extract of my json file (I forgot to remove one "," and to close with } on the last extract)

```auto
"Time Series": {
        "2019-04-12": {
            "1. label1": "1.0000",
            "2. label2": "1.1111",
            "3. label3": "1.2222",
        },
        "2019-04-11": {
            "1. label1": "1.0000",
            "2. label2": "1.1111",
            "3. label3": "1.2222",
        }
}

```

Sorry for imprecisions of information.

Thanks for help,  
Mark

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 15, 2019, 12:59pm UTC](https://discuss.elastic.co/t/another-logstash-json-filter/176586/8 "2019-04-15T12:59:48Z")

</div>

To consume the entire file as a single event I would use a multiline codec with a pattern that never matches

```
 multiline { pattern => "^Spalanzani" what => "previous" negate => true auto_flush_interval => 1 }

```

The first issue we need to fix is that trailing comma on "3. label3". We can do that with a gsub filter. Once that is done we can use a json filter to parse.

```
    mutate { gsub => ["message", ",\s+}", "}"] }
    json { source => "message" target => "someField" remove_field => ["message"] }

```

It's hard to do much with a hash of hashes, but we can use ruby to convert it

```
    ruby { code => 'event.set("timeSeries", event.get("[someField][Time Series]").to_a)' }

```

gets us

```
"timeSeries" => [
    [0] [
        [0] "2019-04-10",
        [1] {
            "2. label2" => "1.1111",
            "3. label3" => "1.2222",
            "1. label1" => "1.0000"
        }
    ],

```

etc. We can use a split filter on that

```
 split { field => "timeSeries" }

```

which gets us three events instead of one. These look like this:

```
"timeSeries" => [
    [0] "2019-04-11",
    [1] {
        "2. label2" => "1.1111",
        "3. label3" => "1.2222",
        "1. label1" => "1.0000"
    }
],

```

Using

```
    mutate { rename => { "[someField][Meta Data]" => "metadata" } remove_field => "someField" }
    date { match => ["[timeSeries][0]", "YYYY-MM-dd" ] }
    ruby { code => 'event.get("[timeSeries][1]").each { |k, v| event.set(k,v) }' remove_field => timeSeries }

```

we can change this into

{  
"2. label2" =\> "1.1111",  
"@timestamp" =\> 2019-04-12T04:00:00.000Z,  
"tags" =\> [  
[0] "multiline"  
],  
"metadata" =\> {  
"6. Time Zone" =\> "GMT+8",  
"5. Last Refreshed" =\> "2019-04-12 16:45:00",  
"3. Parameter2" =\> "Value2",  
"2. Parameter1" =\> "Value1",  
"1. Information" =\> "Informations (label1, label2, label3, label4)"  
},  
"3. label3" =\> "1.2222",  
"1. label1" =\> "1.0000"  
}

which may not be exactly what you want, but hopefully gives you some ideas to try.

---

<div class="post-metadata">

**Author:** ![Mrde](https://avatars.discourse-cdn.com/v4/letter/m/3d9bf3/32.png) [@Mrde](https://discuss.elastic.co/u/Mrde)\
**Post date:** [April 16, 2019, 2:02pm UTC](https://discuss.elastic.co/t/another-logstash-json-filter/176586/9 "2019-04-16T14:02:54Z")

</div>

Thanks a lot for your answer. I should be able to understand with those inputs 🙂

I will keep you in touch and close the subject as soon as I succeed !

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [April 16, 2019, 2:39pm UTC](https://discuss.elastic.co/t/another-logstash-json-filter/176586/10 "2019-04-16T14:39:58Z")

</div>

To consume the whole file content as a single `message` field, you can also use `read mode` and set an impossible delimiter e.g. `øåø`. The delimiter is never found and so the whole file is read and, at EOF, the event is created from the buffered content. This way you don't need a multiline codec.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 14, 2019, 2:40pm UTC](https://discuss.elastic.co/t/another-logstash-json-filter/176586/11 "2019-05-14T14:40:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.