How to remove some unnecessary parts if filebeat's output is file


(sunshine) #1

use filebeat to send stderr or stdout to file.
but file has some unnecessary parts, such as "@timestamp", "beat", "fields" and so on.
All I need just is message part.
How can I remove unnecessary parts in file?

use filebeat 1.2
my filebeat.yml :

filebeat:
prospectors:
-
paths:
- "-"
input_type: stdin
close_eof: true
fields:
mesos_log_stream: $MESOS_LOG_STREAM
mesos_log_sandbox_directory: $MESOS_LOG_SANDBOX_DIRECTORY
$mesos_fields
output:
file:
path: "$MESOS_LOG_SANDBOX_DIRECTORY"
filename: $name
EOF

content in file as follows:

{"@timestamp":"2017-05-16T10:19:32.329Z","beat":{"hostname":"mesos2.","name":"mesos2."},"count":1,"fields":{"mesos_host":"mesos2.*","mesos_log_sandbox_directory":"/da1/mesos/slaves/fc86fdb5-7a6f-4ba4-96d3-d082fcfc1236-S5/frameworks/87e881bb-ce0e-404f-a3aa-200d82d94a68-0000/executors/test-logger.2712baee-3a21-11e7-b76a-0242da0a42cc/runs/a4b12e8c-1d7b-47ca-9f08-bb53e3a51e0d","mesos_log_stream":"STDERR","mesos_marathon_app_id":"/test-logger","mesos_marathon_app_resource_cpus":"1.0","mesos_marathon_app_resource_disk":"0.0","mesos_marathon_app_resource_gpus":"0","mesos_marathon_app_resource_mem":"128.0","mesos_marathon_app_version":"2017-05-16T10","mesos_port":"64466","mesos_port0":"64466","mesos_port_10004":"64466","mesos_port_default":"64466","mesos_ports":"64466","mesos_task_id":"test-logger.2712baee-3a21-11e7-b76a-0242da0a42cc"},"input_type":"stdin","message":"I0516 18:19:32.329854 22075 exec.cpp:162] Version: 1.2.1","offset":0,"source":"-","type":"log"}


(Mark Walkom) #2

You cannot do that sorry, we consider that part of the event as we process it.


(Steffen Siering) #3

You can format output via codec.

e.g.:

output.logstash:
  codec.format.string: '%{[message]}'

Note, message is written as raw string. In case you want to build a json like document, the message fields will not be properly escaped.

Instead of codec, you can use the include_fields processor, to remove a number of fields from the event.


(Mark Walkom) #4

Doesn't %{[message]} contain things like the extra "meta" data beats add?


(Steffen Siering) #5

The docs say message just being the line read by filebeat.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.