# How to remove some unnecessary parts if filebeat's output is file

**URL:** <https://discuss.elastic.co/t/how-to-remove-some-unnecessary-parts-if-filebeats-output-is-file/85905>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 16, 2017, 10:39am UTC](https://discuss.elastic.co/t/how-to-remove-some-unnecessary-parts-if-filebeats-output-is-file/85905 "2017-05-16T10:39:33Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![tang](https://avatars.discourse-cdn.com/v4/letter/t/e8c25b/32.png) [@tang](https://discuss.elastic.co/u/tang)\
**Post date:** [May 16, 2017, 10:39am UTC](https://discuss.elastic.co/t/how-to-remove-some-unnecessary-parts-if-filebeats-output-is-file/85905/1 "2017-05-16T10:39:33Z")

</div>

use filebeat to send stderr or stdout to file.  
but file has some unnecessary parts, such as "@timestamp", "beat", "fields" and so on.  
All I need just is message part.  
How can I remove unnecessary parts in file?

use filebeat 1.2  
my filebeat.yml :

filebeat:  
prospectors:  
-  
paths:  
- "-"  
input\_type: stdin  
close\_eof: true  
fields:  
mesos\_log\_stream: $MESOS\_LOG\_STREAM  
mesos\_log\_sandbox\_directory: $MESOS\_LOG\_SANDBOX\_DIRECTORY  
$mesos\_fields  
output:  
file:  
path: "$MESOS\_LOG\_SANDBOX\_DIRECTORY"  
filename: $name  
EOF

content in file as follows:

{"@timestamp":"2017-05-16T10:19:32.329Z","beat":{"hostname":"mesos2._","name":"mesos2._"},"count":1,"fields":{"mesos\_host":"mesos2.\*","mesos\_log\_sandbox\_directory":"/da1/mesos/slaves/fc86fdb5-7a6f-4ba4-96d3-d082fcfc1236-S5/frameworks/87e881bb-ce0e-404f-a3aa-200d82d94a68-0000/executors/test-logger.2712baee-3a21-11e7-b76a-0242da0a42cc/runs/a4b12e8c-1d7b-47ca-9f08-bb53e3a51e0d","mesos\_log\_stream":"STDERR","mesos\_marathon\_app\_id":"/test-logger","mesos\_marathon\_app\_resource\_cpus":"1.0","mesos\_marathon\_app\_resource\_disk":"0.0","mesos\_marathon\_app\_resource\_gpus":"0","mesos\_marathon\_app\_resource\_mem":"128.0","mesos\_marathon\_app\_version":"2017-05-16T10","mesos\_port":"64466","mesos\_port0":"64466","mesos\_port\_10004":"64466","mesos\_port\_default":"64466","mesos\_ports":"64466","mesos\_task\_id":"test-logger.2712baee-3a21-11e7-b76a-0242da0a42cc"},"input\_type":"stdin", **"message"** :"I0516 18:19:32.329854 22075 exec.cpp:162] Version: 1.2.1","offset":0,"source":"-","type":"log"}

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 16, 2017, 10:53am UTC](https://discuss.elastic.co/t/how-to-remove-some-unnecessary-parts-if-filebeats-output-is-file/85905/2 "2017-05-16T10:53:33Z")

</div>

You cannot do that sorry, we consider that part of the event as we process it.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [May 16, 2017, 11:15am UTC](https://discuss.elastic.co/t/how-to-remove-some-unnecessary-parts-if-filebeats-output-is-file/85905/3 "2017-05-16T11:15:53Z")

</div>

You can format output via [codec](https://www.elastic.co/guide/en/beats/filebeat/current/file-output.html#_codec_3).

e.g.:

```auto
output.logstash:
  codec.format.string: '%{[message]}'

```

Note, message is written as raw string. In case you want to build a json like document, the `message` fields will not be properly escaped.

Instead of `codec`, you can use the [include\_fields](https://www.elastic.co/guide/en/beats/filebeat/current/include-fields.html) processor, to remove a number of fields from the event.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 16, 2017, 9:06pm UTC](https://discuss.elastic.co/t/how-to-remove-some-unnecessary-parts-if-filebeats-output-is-file/85905/4 "2017-05-16T21:06:45Z")

</div>

Doesn't `%{[message]}` contain things like the extra "meta" data beats add?

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [May 17, 2017, 12:32pm UTC](https://discuss.elastic.co/t/how-to-remove-some-unnecessary-parts-if-filebeats-output-is-file/85905/5 "2017-05-17T12:32:34Z")

</div>

[The docs](https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-log.html#_message) say message just being the line read by filebeat.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 14, 2017, 12:32pm UTC](https://discuss.elastic.co/t/how-to-remove-some-unnecessary-parts-if-filebeats-output-is-file/85905/6 "2017-06-14T12:32:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
