# Can I filter the Output data separately?! **URL:** \ **Category:** Logstash\ **Created:** [December 1, 2015, 3:14pm UTC](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064 "2015-12-01T15:14:31Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![jjdepaul](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jjdepaul](https://discuss.elastic.co/u/jjdepaul)\ **Post date:** [December 1, 2015, 3:14pm UTC](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064/1 "2015-12-01T15:14:31Z")
We have the following env: LS 2.0.x; ES 2.0.x and K 4.2.x The input comes from the trace logs that an application produces, we filter the data and then wish to populate two separate (but related) indices: orders and orders\_alt. Each one of these has a separate mapping structure of data. We are having trouble controlling what fields go with what Index. We end up with the same fields in both Indices because ES creates them dynamically and we have to way to filter between the Outputs. How can we better control what goes into which index? Can we somehow remove fields in the Output section?! The MUTATE is part of filtering not output... ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 1, 2015, 6:04pm UTC](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064/2 "2015-12-01T18:04:51Z")
Currently Logstash has a single event pipeline so for a given message the same set of fields will be sent to all outputs. It sounds like you'd have to run multiple Logstash instances. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 1, 2015, 6:26pm UTC](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064/3 "2015-12-01T18:26:30Z")
You should be able to use the [clone filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-clone.html) to generate one event per output and format these differently. ---
**Author:** ![jjdepaul](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jjdepaul](https://discuss.elastic.co/u/jjdepaul)\ **Post date:** [December 1, 2015, 6:33pm UTC](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064/4 "2015-12-01T18:33:24Z")
Reading the documentation on that filter, sounds like it would create a COPY of the event to which we could then add fields as needed - that part sounds good, but how do I tell Logstash to send the original event to 'orders' index while sending a copy of the event + custom fields to the 'orders\_alt' index? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 1, 2015, 7:09pm UTC](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064/5 "2015-12-01T19:09:05Z")
> You should be able to use the clone filter3 to generate one event per output and format these differently. Ah, forgot about clone. Yes, that's the way to go. > [...] but how do I tell Logstash to send the original event to 'orders' index while sending a copy of the event + custom fields to the 'orders\_alt' index? The filter's `clones` option sets the `type` field of the clone(s), so just add conditionals to selectively apply filters and send them to different outputs. ```auto filter { clone { clones => ["cloned"] } if [type] == "cloned" { # this filter } else { # that filter } } ``` ---
**Author:** ![jjdepaul](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jjdepaul](https://discuss.elastic.co/u/jjdepaul)\ **Post date:** [December 1, 2015, 7:43pm UTC](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064/6 "2015-12-01T19:43:48Z")
I'm sorry, I still have trouble understanding how the CLONE filter affects the processing pipeline. Suppose we read in a line from the log, and we do a match and then find the stuff we want, then we call the CLONE filter on that event.... does that mean that the pipeline will then have 2 events - one original and then the Clone event (with a different type)!? Is my thinking correct on that, plz? Thx - ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [December 1, 2015, 8:28pm UTC](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064/7 "2015-12-01T20:28:38Z")
Yes, that's right. Why don't you just try it out? ```auto $ cat test.config input { stdin { type => "original" } } output { stdout { codec => "rubydebug" } } filter { clone { clones => ["clone"] } } $ echo hello | /opt/logstash/bin/logstash -f test.config Settings: Default filter workers: 1 Logstash startup completed { "message" => "hello", "@version" => "1", "@timestamp" => "2015-12-01T20:27:56.049Z", "type" => "original", "host" => "hallonet" } { "message" => "hello", "@version" => "1", "@timestamp" => "2015-12-01T20:27:56.049Z", "type" => "clone", "host" => "hallonet" } Logstash shutdown completed ``` ---
**Author:** ![jjdepaul](https://avatars.discourse-cdn.com/v4/letter/j/e0b2c6/32.png) [@jjdepaul](https://discuss.elastic.co/u/jjdepaul)\ **Post date:** [December 1, 2015, 8:38pm UTC](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064/8 "2015-12-01T20:38:37Z")
I tried it with my config locally and it sure works! Thank you so much - ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:20am UTC](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064/9 "2017-07-06T05:20:29Z")
---
**Author:** ![Alex\_Marquardt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_marquardt/32/42925_2.png) [@Alex\_Marquardt](https://discuss.elastic.co/u/Alex_Marquardt)\ **Post date:** [August 31, 2018, 8:11pm UTC](https://discuss.elastic.co/t/can-i-filter-the-output-data-separately/36064/10 "2018-08-31T20:11:33Z")
I have written a blog post that gives detailed steps on how to use Logstash to filter data in different ways, and then to drive that data to different outputs depending on which filters have been applied. See [https://alexmarquardt.com/2018/08/31/using-logstash-to-drive-filtered-data-from-a-single-source-into-multiple-output-destinations/](https://alexmarquardt.com/2018/08/31/using-logstash-to-drive-filtered-data-from-a-single-source-into-multiple-output-destinations/)