Log with json value and timestamp value from syslog

[ramzey1981 (rame)]

we have a python application logging to syslog in json format but syslog allows appends a timestamp with the json message,

Jan 26 11:07:14 mkobit-ThinkPad {"@source_host": "mkobit-ThinkPad", "module": "pyscratch", "lineno": 63, "pathname": "pyscratch.py", "funcName": "", "message": "HEY there!", "levelname": "INFO", "@timestamp": "2018-01-26T17:07:14.743Z"}

can filebeat handle such a message with this type of format?

[pierhugues (Pier-Hugues Pellerin)]

Hello @ramzey1981,

Out of the box you won't be able to parse that kind of message, you will either need to send the message to an ingest pipeline or Logstash and the beats input for more event parsing using grok and the json decoder.

Thanks

[ramzey1981 (rame)]

thanks Pier,

Yes we thought that json decode would be able to do it but after reading the documentation seems like the first two strings has to be an actual json object which its really not because of the format.

If i send this to logstash i an grok the first field as timestamp and the second filed has hostname or source but then what would I do with the json message field?

[pierhugues (Pier-Hugues Pellerin)]

@ramzey1981 Use grok filter to extract the3 fields:

• date
• hostname
• json_source

After use the logstash-filer-json on the json_source field.

This scenario will work on either Logstash or the ingest pipeline, since they can both work with this kind of data.