JSON to fields in Logstash

Hello,

I am struggling to obtain fields with values from simple JSON file using logstash.
Currently putting the output to file so I see if it looks okay, but I always receive errors for each line.

JSON :

{
"cpuLoad": {
    "minute1": 1.7300000190734864,
    "minute5": 1.149999976158142,
    "minute15": 0.9800000190734864
  },
  "memory": {
    "usage": 48,
    "free": 3598,
    "total": 6980,
    "used": 3382
  },
  "uptime": 3608527,
  "time": "Wed Nov 11 08:32:36 2020",
  "model": "vfc4",
  "release": "GWAY-7.2.4-0186",
  "serialNumber": "1036380",
  "cpuCores": 2,
  "hostname": "Hostname1",
  "users": 0,
  "appliance": "VM",
  "hypervisor": "MSHV"
}

Logstash config:

    input {
      file {
        path => "/opt/logstash/fw-status.json"
        start_position => "beginning"
      }
    }

    filter {
      json {
        source => "message"
      }
    }

    output {
     file {
        path => "/tmp/parsed.log"
      }
    }

Errors from Logstash:

    [2020-11-11T08:42:46,244][WARN ][logstash.filters.json    ][main][937092775ddd949f8f84f6a46203fd34db422fad527a0c49c2a8a0bc1a50d8d3] Error parsing json {:source=>"message", :raw=>"  },", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])"  },"; line: 1, column: 0])
     at [Source: (byte[])"  },"; line: 1, column: 4]>}
    [2020-11-11T08:42:46,278][WARN ][logstash.filters.json    ][main][937092775ddd949f8f84f6a46203fd34db422fad527a0c49c2a8a0bc1a50d8d3] Error parsing json {:source=>"message", :raw=>"  \"appliance\": \"VM\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])"  "appliance": "VM","; line: 1, column: 15]>}

I am receiving above error messages for every line of JSON. Tried also the most simple json with 2 lines, with same result.

Is there any mistake in the configuration file?
Thank you in advance for helping.

Stefan

Just try to remove the filter, so the filter just like this:

filter {
}

Hello,

I no longer receive errors, thank you!

However this is how data output looks like :

{"message":"    \"usage\": 48,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.446Z"}
{"message":"  \"release\": \"GWAY-7.2.4-0186\",","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.452Z"}
{"message":"    \"minute15\": 0.949999988079071","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.444Z"}
{"message":"  \"uptime\": 3612364,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.450Z"}
{"message":"  \"appliance\": \"VM\",","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.456Z"}
{"message":"{","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.424Z"}
{"message":"    \"free\": 3598,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.447Z"}
{"message":"  \"serialNumber\": \"1036380\",","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.453Z"}
{"message":"    \"minute1\": 1.159999966621399,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.443Z"}
{"message":"    \"used\": 3382","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.449Z"}
{"message":"  \"hostname\": \"hostname1\",","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.454Z"}
{"message":"  },","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.445Z"}
{"message":"  \"time\": \"Wed Nov 11 09:36:34 2020\",","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.451Z"}
{"message":"  \"hypervisor\": \"MSHV\"","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.456Z"}
{"message":"    \"minute5\": 1.0499999523162842,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.444Z"}
{"message":"  },","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.449Z"}
{"message":"  \"users\": 0,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.455Z"}
{"message":"  \"memory\": {","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.446Z"}
{"message":"  \"model\": \"vfc4\",","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.451Z"}
{"message":"}","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.457Z"}
{"message":"  \"cpuLoad\": {","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.442Z"}
{"message":"    \"total\": 6980,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.448Z"}
{"message":"  \"cpuCores\": 2,","@version":"1","path":"/opt/logstash/fw-status.json","host":"hostname1","@timestamp":"2020-11-11T08:36:52.454Z"}

Is it possible to obtain the fields from the JSON with its values (after ":" ) ?
Now I only have field "message" with content from every line, which isn't really usefull.

Thank you

In elasticsearch, it will store as JSON and it will create field as following your JSON struture.
I think it will like

cpuLoad.minute1
cpuLoad.minute5
cpuLoad.minute15
cpuLoad.memory.usage
cpuLoad.memory.free
cpuLoad.memory.total
cpuLoad.memory.used
uptime
.
.
.
.

To see it, just put the output to rubydebug
If you want to remove the message, just put

filter {
  mutate {
    remove => "message"
  }
}

And you just get the result only.

After using :

filter {
  mutate {
    remove => "message"
  }
}

Logstash wont start due :

[2020-11-11T10:23:48,973][ERROR][logstash.filters.mutate  ] Unknown setting 'remove' for mutate

I tried remove_field option, but then the message with its content(which I need fields from) is not there:

 {
      "@version" => "1",
          "path" => "/opt/logstash/fw-status.json",
          "host" => "hostname1",
    "@timestamp" => 2020-11-11T09:45:45.360Z
}
{
      "@version" => "1",
          "path" => "/opt/logstash/fw-status.json",
          "host" => "hostname1",
    "@timestamp" => 2020-11-11T09:45:45.361Z
}
.
.
.

this is "overwriting" the message field. try to add a target where the jsonified stuff should land in like

     json {
        source => "message"
        target => "stats"
      }

I would not remove the message-field at all, because it's a kind of "default" field.

Answered here.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.