# JSON to fields in Logstash

**URL:** <https://discuss.elastic.co/t/json-to-fields-in-logstash/255016>\
**Category:** Logstash\
**Created:** [November 11, 2020, 7:49am UTC](https://discuss.elastic.co/t/json-to-fields-in-logstash/255016 "2020-11-11T07:49:03Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Stevek](https://avatars.discourse-cdn.com/v4/letter/s/839c29/32.png) [@Stevek](https://discuss.elastic.co/u/Stevek)\
**Post date:** [November 11, 2020, 7:49am UTC](https://discuss.elastic.co/t/json-to-fields-in-logstash/255016/1 "2020-11-11T07:49:03Z")

</div>

Hello,

I am struggling to obtain fields with values from simple JSON file using logstash.  
Currently putting the output to file so I see if it looks okay, but I always receive errors for each line.

JSON :

```auto
{
"cpuLoad": {
    "minute1": 1.7300000190734864,
    "minute5": 1.149999976158142,
    "minute15": 0.9800000190734864
  },
  "memory": {
    "usage": 48,
    "free": 3598,
    "total": 6980,
    "used": 3382
  },
  "uptime": 3608527,
  "time": "Wed Nov 11 08:32:36 2020",
  "model": "vfc4",
  "release": "GWAY-7.2.4-0186",
  "serialNumber": "1036380",
  "cpuCores": 2,
  "hostname": "Hostname1",
  "users": 0,
  "appliance": "VM",
  "hypervisor": "MSHV"
}

```

Logstash config:

```auto
    input {
      file {
        path => "/opt/logstash/fw-status.json"
        start_position => "beginning"
      }
    }

    filter {
      json {
        source => "message"
      }
    }

    output {
     file {
        path => "/tmp/parsed.log"
      }
    }

```

Errors from Logstash:

```auto
    [2020-11-11T08:42:46,244][WARN][logstash.filters.json][main][937092775ddd949f8f84f6a46203fd34db422fad527a0c49c2a8a0bc1a50d8d3] Error parsing json {:source=>"message", :raw=>" },", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])" },"; line: 1, column: 0])
     at [Source: (byte[])" },"; line: 1, column: 4]>}

```

```auto
    [2020-11-11T08:42:46,278][WARN][logstash.filters.json][main][937092775ddd949f8f84f6a46203fd34db422fad527a0c49c2a8a0bc1a50d8d3] Error parsing json {:source=>"message", :raw=>" \"appliance\": \"VM\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])" "appliance": "VM","; line: 1, column: 15]>}

```

I am receiving above error messages for every line of JSON. Tried also the most simple json with 2 lines, with same result.

Is there any mistake in the configuration file?  
Thank you in advance for helping.

Stefan

---

<div class="post-metadata">

**Author:** ![fadjar340](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fadjar340/32/43610_2.png) [@fadjar340](https://discuss.elastic.co/u/fadjar340)\
**Post date:** [November 11, 2020, 8:17am UTC](https://discuss.elastic.co/t/json-to-fields-in-logstash/255016/2 "2020-11-11T08:17:10Z")

</div>

Just try to remove the filter, so the filter just like this:

```auto
filter {
}

```

---

<div class="post-metadata">

**Author:** ![Stevek](https://avatars.discourse-cdn.com/v4/letter/s/839c29/32.png) [@Stevek](https://discuss.elastic.co/u/Stevek)\
**Post date:** [November 11, 2020, 8:43am UTC](https://discuss.elastic.co/t/json-to-fields-in-logstash/255016/3 "2020-11-11T08:43:02Z")

</div>

Hello,

I no longer receive errors, thank you!

However this is how data output looks like :

```auto
{"message":" \"usage\": 48,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.446Z"}
{"message":" \"release\": \"GWAY-7.2.4-0186\",","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.452Z"}
{"message":" \"minute15\": 0.949999988079071","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.444Z"}
{"message":" \"uptime\": 3612364,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.450Z"}
{"message":" \"appliance\": \"VM\",","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.456Z"}
{"message":"{","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.424Z"}
{"message":" \"free\": 3598,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.447Z"}
{"message":" \"serialNumber\": \"1036380\",","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.453Z"}
{"message":" \"minute1\": 1.159999966621399,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.443Z"}
{"message":" \"used\": 3382","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.449Z"}
{"message":" \"hostname\": \"hostname1\",","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.454Z"}
{"message":" },","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.445Z"}
{"message":" \"time\": \"Wed Nov 11 09:36:34 2020\",","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.451Z"}
{"message":" \"hypervisor\": \"MSHV\"","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.456Z"}
{"message":" \"minute5\": 1.0499999523162842,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.444Z"}
{"message":" },","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.449Z"}
{"message":" \"users\": 0,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.455Z"}
{"message":" \"memory\": {","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.446Z"}
{"message":" \"model\": \"vfc4\",","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.451Z"}
{"message":"}","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.457Z"}
{"message":" \"cpuLoad\": {","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.442Z"}
{"message":" \"total\": 6980,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.448Z"}
{"message":" \"cpuCores\": 2,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.454Z"}

```

Is it possible to obtain the fields from the JSON with its values (after ":" ) ?  
Now I only have field "message" with content from every line, which isn't really usefull.

Thank you

---

<div class="post-metadata">

**Author:** ![fadjar340](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fadjar340/32/43610_2.png) [@fadjar340](https://discuss.elastic.co/u/fadjar340)\
**Post date:** [November 11, 2020, 9:01am UTC](https://discuss.elastic.co/t/json-to-fields-in-logstash/255016/4 "2020-11-11T09:01:34Z")

</div>

In elasticsearch, it will store as JSON and it will create field as following your JSON struture.  
I think it will like

```auto
cpuLoad.minute1
cpuLoad.minute5
cpuLoad.minute15
cpuLoad.memory.usage
cpuLoad.memory.free
cpuLoad.memory.total
cpuLoad.memory.used
uptime
.
.
.
.

```

To see it, just put the output to rubydebug  
If you want to remove the message, just put

```auto
filter {
  mutate {
    remove => "message"
  }
}

```

And you just get the result only.

---

<div class="post-metadata">

**Author:** ![Stevek](https://avatars.discourse-cdn.com/v4/letter/s/839c29/32.png) [@Stevek](https://discuss.elastic.co/u/Stevek)\
**Post date:** [November 11, 2020, 9:47am UTC](https://discuss.elastic.co/t/json-to-fields-in-logstash/255016/5 "2020-11-11T09:47:43Z")

</div>

After using :

```auto
filter {
  mutate {
    remove => "message"
  }
}

```

Logstash wont start due :

```auto
[2020-11-11T10:23:48,973][ERROR][logstash.filters.mutate] Unknown setting 'remove' for mutate

```

I tried remove\_field option, but then the message with its content(which I need fields from) is not there:

```auto
 {
      "@version" => "1",
          "path" => "/opt/logstash/fw-status.json",
          "host" => "hostname1",
    "@timestamp" => 2020-11-11T09:45:45.360Z
}
{
      "@version" => "1",
          "path" => "/opt/logstash/fw-status.json",
          "host" => "hostname1",
    "@timestamp" => 2020-11-11T09:45:45.361Z
}
.
.
.

```

---

<div class="post-metadata">

**Author:** ![BBQigniter](https://avatars.discourse-cdn.com/v4/letter/b/3da27b/32.png) [@BBQigniter](https://discuss.elastic.co/u/BBQigniter)\
**Post date:** [November 11, 2020, 12:25pm UTC](https://discuss.elastic.co/t/json-to-fields-in-logstash/255016/6 "2020-11-11T12:25:43Z")

</div>

> [@Stevek](#):
>
> ```auto
> json {
> source => "message"
> }
> 
> ```

this is "overwriting" the message field. try to add a target where the jsonified stuff should land in like

```auto
     json {
        source => "message"
        target => "stats"
      }

```

I would not remove the message-field at all, because it's a kind of "default" field.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 11, 2020, 3:29pm UTC](https://discuss.elastic.co/t/json-to-fields-in-logstash/255016/7 "2020-11-11T15:29:53Z")

</div>

Answered [here](https://discuss.elastic.co/t/exception-logstash-unexpected-character-code-58-expected-a-valid-value-number-string-array-object-true-false-or-null/241198/2).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 9, 2020, 3:30pm UTC](https://discuss.elastic.co/t/json-to-fields-in-logstash/255016/8 "2020-12-09T15:30:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.