Alter how filebeats send data to output

[kalapakim (Kawika A Moss)]

We are created custom logs in json format and appending as a string, such as:

{"host": "Kawika_DevServer","app_name": "Kafka Test Client","log_level": "Info","event_name": "MessageSend","extended_properties": {"kawika_log": {"logger_utility_testing": {"time": "' + time_now + '"}}}}

When filebeats consumes it and sends to the output, it wraps it in it's own json object, like so:

{
  "@timestamp": "2016-10-30T02:56:52.145Z",
  "beat": {
"hostname": "MTPCTSCID807",
"name": "MTPCTSCID807",
"version": "5.0.0"
  },
  "input_type": "log",
  "message": "{\"host\": \"Kawika_DevServer\",\"app_name\": \"Kafka Test Client\",\"log_level\": \"Info\",\"event_name\": \"MessageSend\",\"extended_properties\": {\"kawika_log\": {\"logger_utility_testing\": {\"time\": \"2016-10-30T02:56:43.030Z\"}}}}",
  "offset": 9630,
  "source": "D:\\Logs\\logger.log",
  "type": "logging_test"
}

But this is not how we want to send it, we just want the string. Is there a way to alter how filebeats send the logs it is consuming?

Thanks

[steffens (Steffen Siering)]

filebeat currently only sends JSON documents. Your event looks like you're forwarding the raw message. Consider using JSON parser in filebeat + put fields under root to get your original event + some beats meta-data. Use beats processors to remove fields you're not interested in.

[kalapakim (Kawika A Moss)]

Where can I find some documentation on how to make the JSON parser work. I'm not finding anything..

[ruflin]

Here you find all the details: https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html#config-json