Grok Parse Failure - Filebeat -> Logstash

[bdobsonbcm (Bryan Dobson)]

I apologize in that I may have missed the correct part in the documentation on this but have been getting nowhere so far this week.

I have a device which I believe is writing fairly simple JSON entries to a log.

Filebeat configuration:

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/gateway*

  json:
    message_key: message
    keys_under_root: true

And I have this on Logstash:

filter {
    json {
        source => "message"
        target => "parsedJson"
    }
}

The messages from the logs look something like:

Jun 11 10:48:56 device01 ssg: {"package":"com.bench.server.transport.http.HttpTransportModule","level":"INFO","log":{"message":"2401: Stopping HTTPS listener: Node HTTPS (2124) (#699e156f7fc4df052c4af880a521f730,v0) on port 2124"},"time":"2019-06-11T10:48:56.680-0700"}

However, I am only getting _jsonparsefailure in Kibana when viewing. I have a feeling I'm missing something simple here.

[bdobsonbcm (Bryan Dobson)]

I enabled debug and see the issue, I'm just not certain the standard/accepted way to work around this as I cannot change the JSON output itself:

[2019-06-11T11:04:58,523][WARN ][logstash.filters.json ] Error parsing json {:source=>"message", :raw=>"Jun 11 11:04:56 device01 ssg: {\"package\":\"com.bench.server\",\"level\":\"FINE\",\"log\":{\"message\":\"Server Closed\"},\"time\":\"2019-06-11T11:04:56.240-0700\"}", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'Jun': was expecting ('true', 'false' or 'null')

[Badger]

    dissect { mapping => { "message" => "%{[@metadata][ts]} %{+[@metadata][ts]} %{+[@metadata][ts]} %{devicename} %{something}: %{[@metadata][json]}" } }
    date { match => [ "[@metadata][ts]", "MMM dd HH:mm:ss" ] }
    json { source => "[@metadata][json]" }

[bdobsonbcm (Bryan Dobson)]

Works like a charm and I've learned something very valuable here.

Thank you very much.