# How to modify output codec for logstash **URL:** \ **Category:** Logstash\ **Created:** [September 26, 2019, 6:51pm UTC](https://discuss.elastic.co/t/how-to-modify-output-codec-for-logstash/201296 "2019-09-26T18:51:55Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![shazbot](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@shazbot](https://discuss.elastic.co/u/shazbot)\ **Post date:** [September 26, 2019, 6:51pm UTC](https://discuss.elastic.co/t/how-to-modify-output-codec-for-logstash/201296/1 "2019-09-26T18:51:56Z")
I'm not expecting a step by step on this. But here is the backstory and what I hope to achieve. I have logstash decoding netflow and putting it out using `json_lines` This is working well, but it's very verbose and too large to store. How can I modify which fields are put out in this file? I'm hoping to pare it down to just a few fields. Secondarily Is it possible to convert it to plain text? When i use `plain` I get a timestamp and %message I appreciate any direction on this. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 26, 2019, 7:27pm UTC](https://discuss.elastic.co/t/how-to-modify-output-codec-for-logstash/201296/2 "2019-09-26T19:27:19Z")
You might find a prune filter helpful for removing top-level fields (you can either whitelist fields to keep or blacklist fields to remove). mutate can also remove fields, and in the worst case you can resort to ruby. What do you mean by plain text? A plain codec by default will emit the timestamp, hostname, and contents of [message]. You can tell it to use a different format. If you really only do want a handful of fields you could supply the list of fields in the format option of the codec and not bother pruning the rest. ``` codec => plain { format => "foo is %{foo}. bar is %{bar}" } } ``` ---
**Author:** ![shazbot](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@shazbot](https://discuss.elastic.co/u/shazbot)\ **Post date:** [September 27, 2019, 8:46pm UTC](https://discuss.elastic.co/t/how-to-modify-output-codec-for-logstash/201296/3 "2019-09-27T20:46:13Z")
Hey, Okay. So the prune filter definitely looks like what I want but I'm having a hard time getting what I'm looking for. my flow logs are as such `{"host":"1.1.1.1","@timestamp":"2019-09-27T20:09:48.000Z","@version":"1","netflow":{"ipv4_dst_addr":"2.2.2.2","src_as":16509,"ipv4_src_addr":"3.3.3.3","dst_as":0,"in_bytes":84,"first_switched":"2019-09-27T20:04:48.832Z","last_switched":"2019-09-27T20:04:48.832Z","input_snmp":95,"in_pkts":1,"flow_seq_num":206430370,"l4_dst_port":2048,"flowset_id":256,"version":9,"protocol":1,"l4_src_port":0,"ipv4_next_hop":"4.4.4.4"}}` And my config is this ``` input { udp { port => 9995 codec => netflow } } filter{ mutate { copy => { "[netflow][ipv4_dst_addr][ipv4_src_addr][low_seq_num]" => "what_i_want" } } prune { whitelist_names => ["what_i_want"] } } output { s3{ access_key_id => "X" secret_access_key => "X" bucket => "1logflowtest" codec => "json_lines" id => "NetflowV9" #encoding => "gzip" size_file => "1024000000" time_file => "60" } file { path => "/var/log/logstash/test.log" codec => "json_lines" } } ``` and the output is `{}` heh. I'm looking to cherry pick fields. I'd like to start with just Source and destination and go from there. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 27, 2019, 10:23pm UTC](https://discuss.elastic.co/t/how-to-modify-output-codec-for-logstash/201296/4 "2019-09-27T22:23:32Z")
Your filter should look more like ``` mutate { copy => { "[netflow][ipv4_dst_addr]" => "[ipv4_dst_addr]" "[netflow][ipv4_src_addr]" => "[ipv4_src_addr]" "[netflow][flow_seq_num]" => "[flow_seq_num]" } } prune { whitelist_names => ["flow_seq_num", "ipv4_dst_addr", "ipv4_src_addr", "@timestamp"] } ``` ---
**Author:** ![shazbot](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@shazbot](https://discuss.elastic.co/u/shazbot)\ **Post date:** [October 10, 2019, 3:48pm UTC](https://discuss.elastic.co/t/how-to-modify-output-codec-for-logstash/201296/5 "2019-10-10T15:48:55Z")
Thank you, @Badger I was able to build on what you posted to fine tune something that was relevant. One last question to avoid making a new thread. Is there any way to have the full output sent off to another destination or would that require a separate conf file ? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 10, 2019, 3:59pm UTC](https://discuss.elastic.co/t/how-to-modify-output-codec-for-logstash/201296/6 "2019-10-10T15:59:57Z")
Not sure I understand the question. You can have multiple outputs in a configuration. But you clearly know that, since you have two outputs in your existing configuration. ---
**Author:** ![shazbot](https://avatars.discourse-cdn.com/v4/letter/s/97f17d/32.png) [@shazbot](https://discuss.elastic.co/u/shazbot)\ **Post date:** [October 10, 2019, 5:29pm UTC](https://discuss.elastic.co/t/how-to-modify-output-codec-for-logstash/201296/7 "2019-10-10T17:29:51Z")
Sorry, let me clarify. I would like one output to be unaffected by the prune. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [October 10, 2019, 5:53pm UTC](https://discuss.elastic.co/t/how-to-modify-output-codec-for-logstash/201296/8 "2019-10-10T17:53:45Z")
You can do that using pipeline to pipeline communications with a [forked path](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#forked-path-pattern) pattern. If you are running on an old version you can do it by using a clone filter, then making the prune and output conditional upon the type set by the clone. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 7, 2019, 5:53pm UTC](https://discuss.elastic.co/t/how-to-modify-output-codec-for-logstash/201296/9 "2019-11-07T17:53:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.