How to remove some unnecessary parts if filebeat's output is file

use filebeat to send stderr or stdout to file.
but file has some unnecessary parts, such as "@timestamp", "beat", "fields" and so on.
All I need just is message part.
How can I remove unnecessary parts in file?

use filebeat 1.2
my filebeat.yml :

filebeat:
prospectors:
-
paths:
- "-"
input_type: stdin
close_eof: true
fields:
mesos_log_stream: $MESOS_LOG_STREAM
mesos_log_sandbox_directory: $MESOS_LOG_SANDBOX_DIRECTORY
$mesos_fields
output:
file:
path: "$MESOS_LOG_SANDBOX_DIRECTORY"
filename: $name
EOF

content in file as follows:

{"@timestamp":"2017-05-16T10:19:32.329Z","beat":{"hostname":"mesos2.","name":"mesos2."},"count":1,"fields":{"mesos_host":"mesos2.*","mesos_log_sandbox_directory":"/da1/mesos/slaves/fc86fdb5-7a6f-4ba4-96d3-d082fcfc1236-S5/frameworks/87e881bb-ce0e-404f-a3aa-200d82d94a68-0000/executors/test-logger.2712baee-3a21-11e7-b76a-0242da0a42cc/runs/a4b12e8c-1d7b-47ca-9f08-bb53e3a51e0d","mesos_log_stream":"STDERR","mesos_marathon_app_id":"/test-logger","mesos_marathon_app_resource_cpus":"1.0","mesos_marathon_app_resource_disk":"0.0","mesos_marathon_app_resource_gpus":"0","mesos_marathon_app_resource_mem":"128.0","mesos_marathon_app_version":"2017-05-16T10","mesos_port":"64466","mesos_port0":"64466","mesos_port_10004":"64466","mesos_port_default":"64466","mesos_ports":"64466","mesos_task_id":"test-logger.2712baee-3a21-11e7-b76a-0242da0a42cc"},"input_type":"stdin","message":"I0516 18:19:32.329854 22075 exec.cpp:162] Version: 1.2.1","offset":0,"source":"-","type":"log"}

You cannot do that sorry, we consider that part of the event as we process it.

1 Like

You can format output via codec.

e.g.:

output.logstash:
  codec.format.string: '%{[message]}'

Note, message is written as raw string. In case you want to build a json like document, the message fields will not be properly escaped.

Instead of codec, you can use the include_fields processor, to remove a number of fields from the event.

Doesn't %{[message]} contain things like the extra "meta" data beats add?

The docs say message just being the line read by filebeat.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.